Questions & Answers: School Choice Demonstration Project Longitudinal Evaluation of the MPCP
Below is the text of a Q and A put out by the School Choice Demonstration Project (SCDP) on the progress of their longitudinal study of the Milwaukee Parental Program.
Questions & Answers
School Choice Demonstration Project Longitudinal Evaluation of the Milwaukee Parental Choice Program
Q. What is the status of the Milwaukee Parental Choice Program (MPCP) research project?
A. The School Choice Demonstration Project (SCDP) began a six-month planning and development phase in March 2006. Extensive data collection efforts occurred from October 2006 through August of 2007, the first year academic year of accountability testing for the schools and the baseline year for the five-year longitudinal evaluation.
Initial reports were issued in February 2008 .
Q. What is the SCDP?
A. It is a national research team dedicated to the objective and comprehensive analysis of the effects of school choice programs. The team includes scholars that have led or participated in most of the important peer-reviewed evaluations of school vouchers that have been conducted to date in the United States. The Principal Investigator of the SCDP is Patrick Wolf, Professor of Education Reform and 21st Century Chair in School Choice at the University of Arkansas. Co-Investigators are John Witte, Professor of Political Science and Public Affairs at the University of Wisconsin and Jay Greene, Head of the Department of Education Reform and Endowed Chair at the University of Arkansas. The lead data collection organization is Westat, the main evaluator of the DC Opportunity Scholarship Program and administrator of the National Assessment of Educational Progress (NAEP). The project’s advisory board includes senior scholars from a variety of academic, research, and policy organizations representing diverse perspectives about school choice and differing methodological approaches to school choice research. Research staff at the Milwaukee Public Schools (MPS) and Milwaukee’s Public Policy Forum have had input on the study design and methodology.
Q. Who is paying for the study?
A. The research is being financed by a diverse group of local and national foundations with interests in educational improvement. They include the Lynde and Harry Bradley, Annie E. Casey, Joyce, Robertson, and Walton Family foundations.
Q. What reports will be issued when?
A. Five annual reports will be released during the period of 2007-2011. Each annual report will have chapters that cover specific dimensions of this comprehensive evaluation. The
first reports released in February 2008 included baseline data collection including MPCP test scores for the accountability study, a descriptive report on the MPCP schools, school finance, and a description of who participates in choice-based programs in Milwaukee – both voucher and charter.
Q. What is the role of the Legislative Audit Bureau?
A. Under 2005 Wisconsin Act 125, the LAB “shall review and analyze the standardized test score data received” from the research team and report the results of its analysis to the Legislature (see s. 119.23 (7) (c) 2). The statutes provide for annual LAB reports from 2007 to 2011. Because LAB will not receive initial reports from the study team until December of 2007, the first LAB report will be issued in 2008.
Q. What tests must students in the choice program take?
A. Under 2005 Act 125, schools in the MPCP must administer annually a standardized test of their choosing in reading, math, and science to students in the fourth, eighth, and tenth grades. Act 125 requires the results of these and any other standardized tests administered by MPCP schools to be provided to the SCDP for analysis. Schools participating in the MPCP use a range of tests, including the Iowa Test of Basic Skills, Wisconsin Knowledge and Concepts Examinations, Stanford Achievement Test, and Terra Nova Achievement Test.
Q. What other tests will be administered in connection with the research study?
A. For a comparison with the performance of MPS students, a large, representative sample of students in the MPCP will take the same tests administered to MPS students (WKCE for students in grades 3-8 and 10 and the Milwaukee Benchmark Assessment for students in grade 9). These results also will be reported to the LAB.
Q. Will the study report raw test scores for individual named schools?
A. No. There are several reasons for this.
• Federal and state law prohibits research that would compromise the confidentiality of student data. These legal provisions are in addition to strict academic protocols for research projects like this one that involve human subjects. Since relatively small numbers of MPCP students are present in certain grades and schools, reporting test score data for named MPCP schools would risk violating the confidentiality and anonymity of study participants. That is why the original state sponsored MPCP evaluations (1991-1995) also did not report school level test scores.
• Raw scores on different tests would provide misleading, apples-to-oranges comparisons among schools. The study team will need to apply sophisticated techniques to convert raw scores from various assessments into a more uniform metric.
• Even scores from individual MPCP schools that are converted into a common scale would be somewhat misleading. Students enter the choice program and its various schools at many different levels of academic achievement and economic disadvantage. Schools with a disproportionate number of students below grade level or in extreme poverty will have lower test scores than other schools. Although some rigorous studies of student achievement have been able to factor such conditions into their calculations, we do not have access to the full set of background information about the MPCP students necessary to adjust for these differences across students for purposes of the MPCP Performance Report Card. That is why the longitudinal component of the study will measure individual achievement gains rather than comparing school-level performance and will serve as the main instrument for evaluating the effects of the program.
Q. Where can we get more information about this project?
A. For general information about the study or detailed information about the accountability or longitudinal panel testing, please contact Project Administrator Laura Jensen at 479-575-6345 or lijensen@uark.edu. For information about the scope and design of the overall research evaluation, please contact Principal Investigator Patrick Wolf at 479-575-2084 or pwolf@uark.edu.