Discussion

These results suggest TAACCCT had a positive effect on both the education and employment outcomes of those who participated in its funded programs and strategies. The magnitude of TAACCCT’s positive effect appears greater for the educational outcomes of program completion and credential completion compared to the employment outcomes, measured as post-program employment and pre- to post-program wage change. These findings, resulting from a thorough review of more than 200 TAACCCT evaluation reports, make an important contribution to estimating the impact of the TAACCCT grant’s federal investment in community and technical colleges.

Although positive results emerged from the meta-analysis, it is important to heed the caution of Pigott (2012, p. 146), a noted expert in meta-analysis techniques, who observes that “even if a systematic review yields equivocal results, a well-conducted systematic review should help to illuminate the issues” related to the effects. This study is limited to the subset of QED studies that provided sufficient methodological and statistical information for inclusion in the analysis and therefore should not be assumed to necessarily represent all evaluation results. Thus, these results should not be expected to stand alone as a single source guiding decision-making about future policy and program design but complement the national evaluation efforts of the Urban Institute and ABT research teams who are continue to conduct independent evaluation studies to describe implementation and measure the impact of TAACCCT.

Complementing the quantitative results of the meta-analysis is a heat map, providing insight into the core elements implemented in the TAACCCT grants. In this analysis, the career pathways approach was implemented widely, including stackable and latticed credentials, prior learning assessment (PLA), comprehensive student supports, career advising and guidance, and employment supports. Online and technology-enabled learning and related online strategies were also implemented in a substantial number of the studies included in the meta-analysis. Partnerships between the community and technical colleges and employers and workforce agencies were also mentioned, and strategies emphasizing the importance of sustaining elements of the grant into the future was evident in the evaluation reports.

Evaluation was also mentioned as an important strategy associated with TAACCCT, with numerous grant reports offering recommendations to help grantees continue to track the educational and employment outcomes of grant-funded participants and add new participants into the tracking of these student outcomes in the future. Evaluators mentioned in their reports the importance of continuing to assess student outcomes, often recognizing complexities in carrying out rigorous impact evaluation.

Rigorous third-party evaluations using QED are challenging to conduct. As many as 70% of third-party evaluators made plans to implement a rigorous design for grants in Rounds 3 and 4, but many fewer were executed due to methodological, logistical, and other difficulties. Barriers that impeded the execution of rigorous evaluation designs are more fully explicated in Bragg (2019) in a companion brief that offers recommendations for future federal investments in community and technical colleges. These suggestions include a call for the federal government to establish realistic expectations and fund professional development and technical assistance that helps evaluators execute their studies usingdesigns that illuminate a program’s impact on targeted student populations, both in aggregate and broken down for sub-populations so that a determination can be made about whether outcomes are distributed equitably. More nuanced designs that delve into the impact of core elements, such as PLA and comprehensive student supports are also needed, and this idea is supported by the two briefs completed by members of the New America team (Palmer, Nguyen 2019; Love 2019).