A Good Opportunity (for Research)

The Department of Education recently released the second year impact evaluation of Washington, D.C.’s Opportunity Scholarship Program. This federally funded program–a fancy name for vouchers–provides randomly selected low-income students living in the District of Columbia a scholarship, or voucher, worth up to $7,500 to attend the private school of their choice. The report has generated response from national and local political figures on both sides of the voucher debate. D.C. Congressional Delegate Eleanor Holmes Norton believes the program should be cut because of school accountability issues, while Council Member Marion Barry supports the program because he believes it increases educational opportunities for D.C.’s children. The fact that the D.C. Opportunity Scholarships Program is up for Congressional reauthorization further intensifies these debates. Beyond the political rigmarole, we at Ed Money Watch believe that the program, and the resulting study, should continue through the full five years to further inform the debate surrounding voucher programs and their impact on academic progress.

The study found no statistically significant difference in math or reading achievement between students who did and did not receive scholarships. While the main finding is not encouraging, the study did find improvements in reading achievement for three subgroups–those who did not attend a low-performing school when they applied for the program, those who had relatively higher pre-program academic performance, and those who applied in the first year of program implementation–that combine to represent 88 percent of participating students. These improvements–roughly a two to four month lead in reading ability over students who did not receive a scholarship–are nothing to scoff at. Additionally, parents whose children participated in the program were more satisfied with the quality and safety of their schools.

These findings are admittedly not particularly compelling. Non-significant findings are rarely a blessing for an experimental program like Opportunity Scholarships. As a result, some politicians, including Norton, are citing the findings as an argument for eliminating the program altogether. It’s important to recognize, however, that these are impacts from only the second year of the program. Sea-changes in educational outcomes rarely, if ever, happen over a period of just two years. If all education reform efforts were held to such high standards, very few would make it past the first two years.

Additionally, there is something to be said for substantively significant differences versus statistically significant differences. Statistical significance means that researchers have 95% confidence that differences between outcomes for experimental and control groups are not due simply to chance. The aggregate impacts in this study don’t meet that threshold. But while the researchers found no statistically significant gains for scholarship recipients overall, trends do seem to be in the positive direction, and the achievement gains for the three student subgroups described above are meaningful differences in performance.

Despite the study’s inconclusive findings, the dispute over the Opportunity Scholarship Program is really about political and ideological controversy surrounding vouchers. Those who support the program believe that vouchers will improve education for all by increasing competition among schools. Those who oppose it take issue with the use of public funds for tuition at private institutions, particularly because private schools are not subject to the same public accountability requirements as public schools. Self interest also plays a role, particularly for the teachers’ unions who sense a threat to their hold on the job market.

However, there is another important angle here worth considering. The D.C. Opportunity Scholarship program is a pilot program connected with a valuable and important evaluation strategy–randomized control trial. This strategy is both rigorous and rare (a political powder keg for the Upward Bound pilot). So, regardless of one’s political or ideological stance on vouchers–or whatever you want to call them–stopping the Opportunity Scholarship Program in its tracks is not the answer. Ending the program will only slow the progress of rigorous, scientifically-based research in education.

Eliminating the Opportunity Scholarship program also poses pragmatic problems for D.C.’s public schools. Cutting the program would force an already highly mobile population to reshuffle back into District of Columbia public schools and further disrupt both the students’ academic progress and DCPS’ reform efforts. This would also be a major blow to families of scholarship recipients who are by and large happy with their chosen schools and believe their children are benefitting from the program.

People who are for vouchers should want to see this program continue to prove that “opportunity scholarships” work and are a valid option for D.C. residents and other districts across the nation. However, for people who are against vouchers, cutting this program short should be the last thing on their mind. What better evidence is there than a truly rigorous randomized control study to prove that this long contested idea is a no-go? In the ongoing debate over school choice and vouchers, it would be nice to have solid evidence to add to the pro or con pile based on a rigorous evaluation of a mature voucher program. As Congress considers these study results and the potential reauthorization of the Opportunity Scholarship Program, the need for solid evidence should prevail over political agendas.