Appendix: Technical Notation

The preponderance of meta-analysis studies to synthesize research findings on education and workforce interventions have generally focused on student-level interventions, not public policy and its effects on student outcomes (Tight, 2018). The United States Department of Labor’s (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) grants to community colleges, plus the DOL’s evaluation requirement of TAACCCT grants, thus presents a unique opportunity to aggregate empirical findings across grantees to isolate the grant program’s broader effects on key policy-relevant outcomes.

For each effect included in the meta-analysis, a conventional log odds ratio was calculated as:

where for effect i from a given evaluation report, a and b are the participant counts of those in the treatment and comparison groups, respectively, achieving the outcome of interest (program completion, post-program employment, etc.); c and d are the participant counts for the treatment and comparison groups not achieving the outcome of interest. Using these counts for effect i, the standard error for each log odds ratio can also be calculated as (Pigott, 2012):

These odds ratios and standard errors were correspondingly analyzed using a random effects meta-analysis model (following DerSimonian & Laird, 1986). A random effects model was chosen, assuming that this analysis would be capturing a distribution of effects from TAACCCT, not one common or “true” effect (Borenstein et al., 2010).

To implement a random effects meta-analysis, a variance τ² for a distribution of observed treatment effects, assumed θᵢ~ N(θ, τ²), was considered relative to calculating a weight for each study’s effect size:

These weights, in turn, allow for an overall effect size and standard error given as:

These overall effect sizes and standard errors yield the estimates that are used to generalize about a potential effect of TAACCCT funding on education and employment outcomes of interest.

In addition to the meta-analysis overall effect, an I² test statistic was calculated to provide a "measure of inconsistency in the studies' results" and capture the percentage of total variation across studies that is due to heterogeneity rather than chance (Higgins & Thompson 2002). The meta-analysis data collected from TAACCCT evaluation reports contained varying sample sizes, study designs, and outcome measures; the I² statistic is well suited to gauge possible heterogeneity to arise from these factors since the measure was developed for these common characteristics of meta-analysis (Higgins et al. 2003). This heterogeneity measure is calculated as:

where Q is the conventional Cochran's heterogeneity statistic and df is degrees of freedom, i.e., number of effects in the given meta-analysis. Cochran’s heterogeneity statistic is calculated by taking the sum of the squared deviations of each study's estimate from the meta-analytic overall effect and weighting each study's contribution in a manner that mirrors the meta-analysis weights (Cochran, 1954). Negative values for I2 are set to zero such that the measure ranges from 0 to 100%.

Higgins et al. (2002) note that heterogeneity above 75% is considered high and that only a quarter of all published meta-analysis have I2 values above 50%. A low level of heterogeneity shows an overall effect, reflecting results from studies that are largely homogeneous in terms of their effects' direction, magnitude, and significance. A high level of heterogeneity, on the other hand, means that an overall effect may be positive and significant but results from highly inconsistent effects. Higgins et al. (2003) note that generalizing from meta-analysis largely depends on overall effects that are the result of consistent results.

This brief’s education and workforce meta-analyses exhibit highly heterogeneous overall effects. Such heterogeneity is intuitive, given the diverse effects portrayed in forest plots. The education outcome’s heterogeneity (98.0%, p<0.001), for instance, contains 20 effects that are positive and statistically significant compared to 18 effects that are statistically insignificant (7 negative and 11 positive). This yields an overall effect that is positive, statistically significant, and highly heterogeneous. The effect of TAACCCT on employment likewise has a similar I2 (92.4%) that reflects an overall effect that is positive and statistically significant (p<0.001).