Meta-Analysis Methods

Meta-analysis aggregates statistical findings across multiple studies. Used to inform such questions as “What does the research tell us about this intervention?” meta-analysis is commonly used to synthesize epidemiology and health research and increasingly found in the social sciences and education (Cooper, Hedges, & Valentine, 2009). A meta-analysis process distills hundreds of publications into a handful of studies that meet a set of criteria for inclusion. The preliminary data for this meta-analysis came from 216 TAACCCT evaluation reports (comprising 84% of all TAACCCT grants), from which 36 evaluation reports were selected for inclusion after a four-phase review process. The vast majority of final third-party evaluation reports were accessed from the SkillsCommons Repository (see:skillscommons.org), but a few reports were obtained directly from evaluators when a search of the SkillsCommons Repository did not yield a copy of the final report. The four-phase process for reviewing and making decisions on inclusion/exclusion is described below.

In Phase One, the research team reviewed each publicly available TAACCCT evaluation report. Two reviewers read each report identifying the interventions, looking for evidence of a grant theory of change, quantitatively scoring implementation and outcomes, and identifying the evalaution design including comparison groups.

In Phase Two, the reports that claimed to employ QED (n=143) were reviewed to determine whether they were plausibly QED studies. This phase required careful reading; many reports contained treatment and comparison groups because of DOL instructions that third-party evaluations should include comparison studies, but upon further inspection were determined by reviewers to not be QED. Recognizing that evaluators faced pressure to implement QEDs even in the face of data limitations and other methodological constraints, reviewers also detailed efforts to implement QEDs that were unsuccessful. An additional component of this phase was determining which outcomes might be sufficiently represented in the evaluation reports that a critical mass of data points were present. We determined the outcomes of program completion, credential completion, employment, and wage change to be the most meaningful and plentiful of measures used across the full spectrum of evaluation studies for TAACCCT, and we narrowed to these outcomes in the next phase of the review process.

In Phase Three, the studies where authors claimed to have used any form of experimental and quasi-experimental design with causal estimates linking TAACCCT program participation to the outcomes of program completion, credential completion, employment, and wage change were evaluated for inclusion in our meta-analysis (n=66). The Phase Three review determined whether the studies included evidence of a QED, and in all cases but two were revealed to use propensity score matching (PSM). Given that the TAACCCT evaluation reports had not been published in refereed journals and therefore were not subject to the rigorous methodological review process associated with publication of meta-analysis studies in the literature, we established selection criteria that took into account factors normally considered in peer-refereed review. These criteria included: study authors identified their evaluation study design as a QED and described aspects of the QED design in the text of their evaluation report in sufficient depth and detail to provide reasonable confidence that the study design was actually a QED. Studies reviewed in Phase Three where the evaluation report lacked sufficient statistical information consistent with the purported approach to QED were eliminated from review. However, a sub-set of QED studies that lacked a specific statistic needed for the meta-analysis (i.e., lacking a sample size or standard error statistic) but otherwise which appeared to be a bona fide QED design, were identified as a potential QED and the authors of these studies, totaling 15, were contacted via email and phone to request the missing statistic that would allow us to include the study. All but four authors provided the requested statistic so that these studies could be included in our meta-analysis. This step was important, as these added studies made up 31% of the total evaluations included here.

Phase Four entailed conducting the meta-analysis using the 36 studies that met our inclusion criteria. At this point, each report’s outcomes (relative to treatment and comparison groups) were recorded and converted to standardized effects. Studies that were not included in the final meta-analysis either lacked requisite statistics for inclusion or, after all evidence was examined, were determined to be descriptive in nature and actually not QED studies. Ultimately, 60 effects representing education and employment outcomes were drawn from the final set of 36 studies included in the meta-analysis.

Table 1 provides a summary of the TAACCCT grants included in the meta-analysis study by grant title, author(s), round, grant type (single institution, state consortium or multi-state consortium), industry sector, and outcome(s). No QED studies were included from Round 1, but numerous studies were included from each of the other three rounds, with 14 studies from Round 2, 9 studies from Round 3, and 13 studies from Round 4. The breakdown of grant type included 12 studies conducted of TAACCCT grants awarded to a single institution, 15 studies of single-state consortia, and 9 studies of multi-state consortia. As with the overall TAACCCT grants, manufacturing was the most prevalent industry sector included in the evaluations, with 8 studies focusing on healthcare, 5 on information technology (IT), and 4 multi-sector grants that also included manufacturing, healthcare, and IT. Other sectors represented in the grants included energy; transportation, distribution and logistics; business services, various trades (i.e., construction, electronics), and various other occupational areas.

Table 1. Summary of TAACCCT Grant Characteristics in Meta-Analysis Study

The relevant outcomes from qualified third-party evaluation reports were converted to a standardized effect in the form of an odds ratio and its associated standard error for the four outcomes: program completion, credential completion, post-program employment, and pre- to post-program wage change. Program completion and credential completion effects were pooled under the broader category of educational outcomes. Post-program employment and post-program wage change were likewise pooled and categorized as employment outcomes.

Using a random effects model, these standardized effects for education and employment outcomes were weighted and pooled, resulting in “forest plot” visualizations and an estimate of the overall effect for the two outcomes of interest. In addition, a measure of heterogeneity was calculated to gauge the consistency of the studies’ results across the education and employment outcomes analyzed in this brief. Appendix A provides additional technical narrative to supplement the description of our approach for this study. Even though more options for conducting meta-analysis are rapidly emerging as this form of research in education grows (including the emergence of various forms meta-regression), we chose a fairly straightforward approach to this initial meta-analysis and may advance to more advanced forms assuming the data support these analytical designs.