Priming the Pump: Guiding Principles

Besides the state context, a set of guiding principles will help navigate the design of federal accountability and associated metrics. A review of the research and policy literature reveals a series of key decision points that must be addressed. This section seeks to help inform the choice process through a discussion and recommendation for each.

Report-Card or Consequential Accountability?

Deming and Figlio make the key distinction between accountability with and without consequences.¹ The latter, which they call report-card accountability, entails making performance information and data public and letting students and families “vote with their feet” without attaching any consequences (rewards or sanctions). The College Scorecard is an example of such a no-stakes approach in higher education, which essentially is just a form of robust transparency that seeks to improve how the market functions, with informed consumers making rational choices based on higher quality and more information. In contrast, consequential accountability involves attaching specific rewards and sanctions to certain performance thresholds, such as CDR in the HEA or the GE regulations.

While there is a long history of transparency initiatives, from the 1990 Student Right-to-Know Act to the College Navigator and then to the College Affordability and Transparency Center and, most recently, the College Scorecard, the track record clearly demonstrates that transparency alone has failed to drive performance improvements. At the same time, it is also evident that the federal government has a responsibility to address the problems of imperfect information and information asymmetries in the higher education marketplace and to give students, families, policymakers, stakeholders, and the public accurate, relevant, and timely information in a user-friendly manner. However, just as in federal accountability policy in elementary and secondary education, such transparency must be coupled with consequential accountability, in order to hold colleges and universities responsible for their performance, protect students and taxpayers, and encourage improvements. This paper argues for a combination of both transparency and accountability, with the former providing information on a wide range of data and the latter focusing on a more limited set of metrics that reflect the top policy priorities. Transparency is a prerequisite, not a synonym, for accountability.

Progress or No Accountability?

Undoubtedly, designing a consequential accountability system is a complex endeavor, and given the massive federal investments in student aid, through direct and tax expenditures across several departments, any attempt will be imperfect and the subject of intense scrutiny and criticism. In light of the many issues regarding the quality and scope of data collected by the federal government (in part due to the absence of a student-level data network, thanks to the statutory ban); the value-laden choice process of what to measure and how; ideological differences among policymakers about the proper role of the federal government; and historical industry opposition to accountability—especially among institutions likely to be negatively impacted, any outcome will be viewed as imperfect. However, this should not act as a barrier to moving forward. Instead, we must acknowledge from the outset that any accountability system and metrics will be imperfect and will need to be tweaked and refined in the future. Simply put, it is essential to recognize that there is no perfect accountability system that will get it absolutely right the first time around. But replacing the current status quo of no accountability with a better system, albeit one that requires evaluation and adjustment, will be a step in the right direction.

Carrots or Sticks?

An accountability system that only rewards high performers is more feasible politically, for obvious reasons. However, such an approach will fail to provide incentives for continuous improvement to the large majority of schools that are and will remain ineligible for rewards, and it will not protect students and taxpayers from low-performing schools and bottom-feeders. Absent penalties, the latter will have no problem proceeding with business as usual, as their access to student aid dollars remains unchanged. At the same time, the current system of failing to recognize and reward high performance undermines continuous improvement efforts by treating all schools the same, regardless of their outcomes. Therefore, any accountability system must include both carrots and sticks, rewards and penalties, to incentivize stronger performance and improved outcomes.

Adjust for Risk or Not?

A major sticking point in efforts to design accountability systems, such as President Obama’s college ratings effort, has been the debate on whether and how to adjust data for student and institutional characteristics, given the enormous diversity among institutions and the populations they serve. Under-resourced schools and/or schools serving larger shares of at-risk students (low-income students, students of color, students with inadequate academic preparation, non-traditional students, etc.) feel that their performance should be risk-adjusted. While there is certainly merit to such an argument, it is important to remember that the sole federal accountability outcome-based metric, CDR, is not risk-adjusted and that schools have a responsibility to educate the students they enroll and charge for tuition. Also, risk-adjustment effectively sets lower expectations for at-risk students,² directly contradicting the historical equity focus of the federal government’s role in higher education. Finally, an analysis of labor market outcomes found that the choice of metric and length of the follow-up data period—for instance, how many years after graduation earnings are measured—matter more than adjustments based on demographic and other factors.³

Therefore, instead of a lower bar, an alternative approach for recognizing different student profiles is to adjust indirectly and reward institutions by either weighing more heavily at-risk student outcomes, producing separate metrics for certain students (such as those eligible to receive Pell Grants), or a combination of both. Policymakers have had more success in implementing metrics such as these that balance the need to account for student characteristics with the need to be simple and understandable. This “bonus” approach, which is employed widely by states, balances the need to recognize student characteristics with the need for equity, and is also simpler, more straightforward, and easier to implement than the regression-adjusted metrics.⁴ A similar approach is already used in the CDR, which accounts for the borrowing rate to assess government exposure to risk from poor student loan performance, as opposed to school- or student-based characteristics, which is discussed more in the ensuing student loan accountability metrics section.

Institutional or Program-Level Accountability?

The GE regulations provided an enormous amount of outcome data at the program level and spurred calls for expanding the collection and release of program-level data across higher education, from both ends of the political spectrum. The simple fact that institution-level outcomes mask significant variation among individual programs by both field of study and credential level, illustrated by the shocking revelation that even Harvard’s graduate program in theater failed GE, is certainly a strong argument for the collection and dissemination of outcome data by program.⁵ Most recently, President Trump’s executive order to enhance the College Scorecard with program-level information,⁶ as well as the white paper released by Sen. Lamar Alexander (R-Tenn.), chair of the Senate education committee, that calls for program-level loan repayment,⁷ have further elevated the issue on the reauthorization agenda. However, there is growing concern about the limitations and “blind spots” of program-level data and their high risk of manipulation, as identified by industry associations,⁸ academics,⁹ and policy researchers.¹⁰ A balanced approach would use both program- and institutional-level outcome data, but for different accountability purposes: the former for report-card (low-stakes) accountability and the latter for consequential (high-stakes) accountability. An exception to this principle would be made for GE, in recognition of the fundamental differences between such programs and those originally supported by the HEA, as reflected in the law.

Universal or Targeted Accountability?

The choice between universal or targeted accountability includes questions of scope at the institutional, student, sector, and study levels. Should all accountability metrics apply to all institutions or only to some? Should accountability measures capture data from all students or only those receiving student aid? Should accountability apply just to the outcomes of undergraduate students or include graduate students as well?

In terms of which institutions should be subject to which accountability metrics, the question pertains primarily to the long-standing for-profit industry criticism of the HEA’s distinct treatment of its colleges through the statutory 90-10 rule and the GE regulations. In this paper, we will assume that this distinction will remain; in recognition of the fundamentally different incentives of for-profit entities, which have a legal, fiduciary responsibility to maximize profits for their owners or shareholders. However, we propose that the statutory 90-10 rule, which represents a market-based accountability test, should be restored to 85-15 and strengthened to include all other non-Title IV federal student aid revenues, but could also be waived as a reward for the highest performers within a new and strong accountability system that uses solid outcome metrics, as discussed in the following section, as well as under GE. The GE distinction—applying the requirement only to proprietary and postsecondary vocational programs—should be maintained, as it operates at the program level, and it also recognizes the distinct risk posed by such programs to those who finance their own job training with student loan debt. Simply put, the particular purpose of non-degree and career-college programs is different from the academic programs for which federal aid dollars were originally used.

In terms of the population whose outcomes should be included in an accountability system, some have argued that only students benefiting directly from Title IV student financial assistance should be included. However, all students should be included in principle, just as in the K–12 federal accountability system, where all states, districts, and schools receiving federal aid are held accountable for all of their students. Even students not receiving student loans and Pell Grants are likely benefiting from a variety of servicemember and veteran benefits, tax benefits, and/or research dollars through which taxpayers contribute to their education. At the same time, by definition, some metrics based on loans or grants would only capture a subset of students. This is both acceptable and desirable, given that the federal government does clearly have a particular interest in those students’ outcomes and (in the case of student loans) those students may be at risk if an institution’s quality is poor. Therefore, a balanced accountability system would include metrics capturing outcomes for all students, such as degree completion; metrics pertaining to student aid, such as loan-based metrics; and metrics addressing priority populations, such as Pell-based metrics. The inclusion of graduate students as a distinct category in federal accountability also seems necessary, given its outsized share of and impact on the federal student loan portfolio¹¹ and recent enrollment trends that suggest the programs often act as a “cash cow” for colleges.¹²

Overall, the answer to the question of universal versus targeted accountability is that both must be incorporated in the federal accountability system through a mix of provisions and metrics that account for all schools and students but also measure a subset based on risk and government subsidies.

Bright Lines or Graduated Accountability?

Another major decision point involves the criticism of the all-or-nothing feature of the CDR and GE that is used as an argument against using bright lines or thresholds in any new accountability system. For example, the House PROSPER Act retains such an approach in its loan repayment metric. While bright lines indeed fail to provide an incentive for improvement to schools that barely miss the threshold, as well as schools that do not come close to hitting a trigger, they still provide an important function in identifying a minimum level of performance considered acceptable for participation in student aid programs, thus weeding out schools with consistently horrific outcomes.

However, a system of graduated accountability, under which sanctions gradually increase as performance decreases, also provides several advantages in providing meaningful incentives, limiting gaming, and ensuring that penalties (and rewards) are commensurate with performance. Thus, a balanced approach would involve both bright lines for identifying the worst as well as a system of gradual incentives. Setting a bright line for identifying the best performers (similar to Blue Ribbon schools in K–12 education) would also be beneficial in recognizing and rewarding excellence, sending a strong signal to consumers, and providing an added incentive for institutional improvements, especially if funding incentives are insufficient across the performance spectrum. This approach, used in the Senate’s bipartisan Student Protection and Success Act, would combine the best elements of both approaches and ensure a minimum threshold of acceptable performance for participation in student aid (above 15 percent repayment rate) and meaningful incentives for continuous improvement (risk-sharing fee based on non-repayment rate and bonus for schools doing a good job for low- and moderate-income students).

Obviously, the hardest part, both technically and politically, is deciding where to set the floor, followed by setting the performance thresholds for gradual incentives and disincentives. One potential way to move past this barrier is to adopt and adapt the approaches used in other areas, such as K–12 education, where the bottom 5 percent of schools and those with graduation rates under 67 percent require intervention. While the specific thresholds can be adjusted, the point is that federal policymakers have agreed on bright lines when it comes to elementary and secondary education, so setting bright lines in the postsecondary arena, notwithstanding key differences between the two areas, should be doable. The impetus should be even greater, given the outsized federal investment in higher education. The two largest federal programs for K–12, Title I and IDEA, are roughly equal to Pell expenditures.

Accuracy or Simplicity?

Finally, the trade-off between accuracy and simplicity is paramount. If there is one thing everyone seems to agree on is that any accountability system must be simple and understandable, which necessarily comes at the expense of precision. An overly complex system with multiple metrics, various adjustments, and groupings of institutions might provide a more accurate and nuanced picture of performance, but will reduce political and practical feasibility, expand the scope for strategic responses and gaming, and fail to provide user-friendly and accessible information. At the same time, reliance on a single metric, such as the current CDR, might be the simplest way forward, but it also presents significant risk in undermining accountability efforts. For example, a sole loan repayment rate to assess performance and risk for the $100 billion annual federal student loan investment risks weakening the accountability effort by failing to capture the return on the $30 billion annual federal investment in Pell Grants.¹³ Instead, a balanced approach would use a small number of metrics sufficient to quantify institutional performance on outcomes that represent priority policy objectives and capture a broader set of indicators, while ensuring overall simplicity.