PART Assessments of Department of Education Programs – The Results

During the presidential campaign, both Senator McCain and Senator Obama talked about eliminating ineffective federal programs. We commend their emphasis on funding programs that show results. But what evidence will President-elect Obama use to actually determine which programs are working and which aren’t?

One source of information the president could turn to for guidance is the Program Assessment Rating Tool (PART). But for many programs, particularly at the Department of Education, President-elect Obama might find only limited information to determine whether some programs are actually effective.

Results for Department of Education Programs

Since 2002, 93 Department of Education programs have gone through the PART review process. Of those, 44 were rated “results not demonstrated.” That means nearly half the programs did not have acceptable performance measures or lacked the performance data necessary to make a determination.

Of Department of Education programs that did have enough information for a rating, only six programs were found to be “effective” – Adult Education State Grants, Institute of Education Sciences Research, National Assessment for Educational Progress, National Center for Education Statistics, Reading First State Grants, and Transition to Teaching. Only four programs are currently rated “ineffective” – Even Start, Perkins Loan Programs, TRIO Upward Bound, and Vocational Education State Grants. The remaining 39 programs are rated “moderately effective” or “adequate.”

You can visit www.ExpectMore.Gov to see individual program results. The website provides information about ratings for programs at every agency and links to their individual PART assessments and improvement plans.

How Department of Education Results Compare with Other Agencies

Examining PART results for the Department of Education compared to program results in other departments provides important perspective on the efficacy of PART for education.

Generally, departments that administer social services, such as housing, labor, and education, have a lower percentage of programs rated “effective” and more programs with “results not demonstrated” than other agencies. Of all the cabinet level agencies, the Department of Education has the highest proportion of programs without demonstrated results. (The second highest is Veteran’s Affairs, which has 30 percent of the programs rated “results not demonstrated.”) There are a variety of reasons for this disparity.

Social service programs, particularly in education, can be hard to evaluate. Often, data is not available to provide evidence for a PART rating because many programs at the Department of Education have not undergone rigorous studies or evaluations. For those with available data, a scientifically controlled experiment may not be possible due to the nature of the services provided or because such a study would be time consuming and expensive. Additionally, many programs require longitudinal data and multiple cohorts to conduct rigorous research and such data systems, where they exist, are still too young to be relevant.

We find it problematic that so many education programs undergo PART assessments without evidence to either support or refute them. If PART is meant to be a litmus test for continued program funding, this dearth of research must be remedied. As we have talked about in past posts, this has significant implications for the Department of Education research budget. In fiscal year 2008, the Department of Education dedicated only one half of one percent of its budget to research and development.

The size and type of education programs examined under PART may also affect program results. The Department of Education has many small programs with limited staff and funding. Capacity often limits a program’s ability to collect or monitor annual and long term performance data.

Larger programs at the Department tend to be competitive or block grant programs. A report from OMB Watch suggests that competitive and block grant programs do not perform as well as other types of programs on the PART. Block grants distribute funding to local districts, making it difficult to monitor and evaluate results centrally. More than a third of Department of Education programs with PART ratings are block grant programs, and 84 of the 93 programs were either block grant or competitive grant programs – a higher percentage than in many other agencies.

PART is supposed to be an unbiased and accurate assessment of federal programs. But answering the PART questions requires some degree of subjectivity. As a result, the PART review process can be vulnerable to ideology or unreliability across reviewers or agencies. Similarly, PART’s one-size-fits-all approach to assessing federal programs may not accurately capture the nuances in different types of programs, disadvantaging Department of Education and other social service programs.

President-elect Obama talked about using a scalpel, not a hatchet, to eliminate specific programs that don’t work. But what will he use to guide his scalpel? How will he know which programs are working and which aren’t? In the end, President-elect Obama is likely to find that PART provides only limited guidance when determining which programs are and are not effective, especially when it comes to education.