Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?

References

• Bernstein, L., Rappaport, C. D., Olsho, L., Hunt, D., & Levin, M. (2009). Impact evaluation of the US Department of Education’s student mentoring program. Final report. NCEE 2009-4047. National Center for Education Evaluation and Regional Assistance.
   Google Scholar
• Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59. https://doi.org/10.3102/0162373707299550
   Web of Science ®Google Scholar
• Chipman, H. A., George, E. I., & McCulloch, R. E. (2010). BART: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1), 266–298. https://doi.org/10.1214/09-AOAS285
   Web of Science ®Google Scholar
• Clark, M. A., Chiang, H. S., Silva, T., McConnell, S., Sonnenfeld, K., Erbe, A., & Puma, M. (2013). The effectiveness of secondary math teachers from teach for America and the teaching fellows programs. NCEE 2013-4015. National Center for Education Evaluation and Regional Assistance.
   Google Scholar
• Donoho, D., & Stodden, V. (2006, July). Breakdown point of model selection when the number of variables exceeds the number of observations. In The 2006 IEEE International Joint Conference on Neural Network Proceedings (pp. 1916–1921). IEEE. https://doi.org/10.1109/IJCNN.2006.246934
   Google Scholar
• Dorie, V., Hill, J., Shalit, U., Scott, M., & Cervone, D. (2019). Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 34(1), 43–68. https://doi.org/10.1214/18-STS667
   Google Scholar
• DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest : A Journal of the American Psychological Society, 12(2), 57–91.
   PubMed Web of Science ®Google Scholar
• Eberlin, L. S., Tibshirani, R. J., Zhang, J., Longacre, T. A., Berry, G. J., Bingham, D. B., Norton, J. A., Zare, R. N., & Poultsides, G. A. (2014). Molecular assessment of surgical-resection margins of gastric cancer by mass-spectrometric imaging. Proceedings of the National Academy of Sciences of the United States of America, 111(7), 2436–2441.
   PubMed Web of Science ®Google Scholar
• Esbensen, F. A., Peterson, D., Taylor, T. J., & Osgood, D. W. (2012). Results from a multi-site evaluation of the GREAT program. Justice Quarterly, 29(1), 125–151. https://doi.org/10.1080/07418825.2011.585995
   Web of Science ®Google Scholar
• Finch, W. H., & Finch, M. E. H. (2016). Regularization methods for fitting linear models with small sample sizes: Fitting the Lasso estimator using R. Practical Assessment, Research, and Evaluation, 21(1), 7.
   Google Scholar
• Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22. https://doi.org/10.18637/jss.v033.i01
   PubMed Web of Science ®Google Scholar
• Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010). The evaluation of charter school impacts: Final report. NCEE 2010–4029. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Hill, J. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1), 217–240. https://doi.org/10.1198/jcgs.2010.08162
   Web of Science ®Google Scholar
• James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, p. 18). Springer.
   Google Scholar
• Jones, M. P. (1996). Indicator and stratification methods for missing explanatory variables in multiple linear regression. Journal of the American Statistical Association, 91(433), 222–230. https://doi.org/10.1080/01621459.1996.10476680
   Web of Science ®Google Scholar
• Kern, H. L., Stuart, E. A., Hill, J., & Green, D. P. (2016). Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness, 9(1), 103–127.
   PubMed Web of Science ®Google Scholar
• Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798
   Web of Science ®Google Scholar
• Lin, L. (2018). Bias caused by sampling error in meta-analysis with small sample sizes. PloS One, 13(9), e0204056.
   PubMed Web of Science ®Google Scholar
• Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S., & Busick, M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. National Center for Special Education Research.
   Google Scholar
• Olsen, R., Stuart, E. A., Orr, L., & Bell, S. (2020). How much can evidence from national studies improve local policy decisions that affect youth? - Analysis plan. The Open Science Framework. https://osf.io/vbs36/
   Google Scholar
• Orr, L. L., Olsen, R. B., Bell, S. H., Schmid, I., Shivji, A., & Stuart, E. A. (2019). Using the results from rigorous multisite evaluations to inform local policy decisions. Journal of Policy Analysis and Management, 38(4), 978–1003. https://doi.org/10.1002/pam.22154
   Web of Science ®Google Scholar
• Puma, M. J., Olsen, R. B., Bell, S. H., & Price, C. (2009). What to do when data are missing in group randomized controlled trials. NCEE 2009-0049. National Center for Education Evaluation and Regional Assistance.
   Google Scholar
• Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
   Web of Science ®Google Scholar
• Weiss, M. J., Bloom, H. S., & Brock, T. (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808. https://doi.org/10.1002/pam.21760
   Web of Science ®Google Scholar
• Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719
   Web of Science ®Google Scholar
• Wilkins, C., Gersten, R., Decker, L. E., Grunden, L., Brasiel, S., Brunnert, K., & Jayanthi, M. (2012). Does a summer reading program based on lexiles affect reading comprehension? Final report. NCEE 2012-4006. National Center for Education Evaluation and Regional Assistance.
   Google Scholar
• Wilson, D. B., Gottfredson, D. C., & Najaka, S. S. (2001). School-based prevention of problem behaviors: A meta-analysis. Journal of Quantitative Criminology, 17(3), 247–272. https://doi.org/10.1023/A:1011050217296
   Google Scholar