Propensity Score Matching with Cross-Classified Data Structures: A Comparison of Methods

References

• Arpino, B., & Cannas, M. (2016). Propensity Score matching with clustered data. An application to the estimation of the impact of caesarean section on the Apgar score. Statistics in Medicine, 35(12), 2074–2091.https://doi.org/10.1002/sim.6880
   PubMed Web of Science ®Google Scholar
• Arpino, B., & Mealli, F. (2011). The specification of the Propensity Score in multilevel observational studies. Computational Statistics & Data Analysis, 55(4), 1770–1780. https://doi.org/10.1016/j.csda.2010.11.008
   Web of Science ®Google Scholar
• Austin, P. C. (2011a). Comparing paired vs non-paired statistical methods of analyses when making inferences about absolute risk reductions in propensity-score matched samples. Statistics in Medicine, 30(11), 1292–1301.https://doi.org/10.1002/sim.4200
   PubMed Web of Science ®Google Scholar
• Austin, P. C. (2011b). An introduction to Propensity Score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424.https://doi.org/10.1080/00273171.2011.568786
   PubMed Web of Science ®Google Scholar
• Aydin, B., Leite, W. L., & Algina, J. (2016). The effects of including observed means or latent means as covariates in multilevel models for cluster randomized trials. Educational and Psychological Measurement, 76(5), 803–823.https://doi.org/10.1177/0013164415618705
   PubMed Web of Science ®Google Scholar
• Bandalos, D. L., & Leite, W. L. (2015). Use of Monte Carlo studies in structural equation modeling research. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 625–666). Information Age Publishing.
   Google Scholar
• Bates, D., Maechler, M., & Bolker, B. (2011). Lme4: Linear mixed-effects models using s4 classes. http://cran.cnr.berkeley.edu/web/packages/lme4/index.html
   Google Scholar
• Beretvas, S. N., & Murphy, D. L. (2013). An Evaluation of Information Criteria Use for Correct Cross-Classified Random Effects Model Selection. The Journal of Experimental Education, 81(4), 429–463. 10.1080/00220973.2012.745467
   Web of Science ®Google Scholar
• Carroll-Scott, A., Gilstad-Hayden, K., Rosenthal, L., Eldahan, A., McCaslin, C., Peters, S. M., & Ickovics, J. R. (2015). Associations of neighborhood and school socioeconomic and social contexts with body mass index among urban preadolescent students. American Journal of Public Health, 105(12), 2496–2502.https://doi.org/10.2105/AJPH.2015.302882
   PubMed Web of Science ®Google Scholar
• Cepeda, M. S., Boston, R., Farrar, J. T., & Strom, B. L. (2003). Optimal matching with a variable number of controls vs. a fixed number of controls for a cohort study: Trade-offs. Journal of Clinical Epidemiology, 56(3), 230–237. https://doi.org/10.1016/S0895-4356(02)00583-8
   PubMed Web of Science ®Google Scholar
• Collins, L. M., Graham, J. W., Long, J. D., & Hansen, W. B. (1994). Cross validation of latent class models of early substance use onset. Multivariate Behavioral Research, 29(2), 165–183. https://doi.org/10.1207/s15327906mbr2902_3
   PubMed Web of Science ®Google Scholar
• Grady, M. W., & Beretvas, S. N. (2010). Incorporating student mobility in achievement growth modeling: A cross-classified multiple membership growth curve model. Multivariate Behavioral Research, 45(3), 393–419.https://doi.org/10.1080/00273171.2010.483390
   PubMed Web of Science ®Google Scholar
• Gu, X. S., & Rosenbaum, P. R. (1993). Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphical Statistics, 2(4), 405–420. https://doi.org/10.1080/10618600.1993.10474623
   Google Scholar
• Hansen, B. B. (2007). Optmatch: Flexible, optimal matching for observational studies. R News, 7(2), 18–24.
   Google Scholar
• Hansen, B. B., & Klopfer, S. O. (2006). Optimal full matching and related designs via network flows. Journal of Computational and Graphical Statistics, 15(3), 609–627. https://doi.org/10.1198/106186006X137047
   Web of Science ®Google Scholar
• He, Z. (2018). Inverse conditional probability weighting with clustered data in causal inference.
   Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2007). Intraclass Correlation Values for Planning Group-Randomized Trials in Education. Educational Evaluation and Policy Analysis, 29(1), 60–87. 10.3102/0162373707299706
   Web of Science ®Google Scholar
• Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8), 1–28. https://doi.org/10.18637/jss.v042.i08
   Web of Science ®Google Scholar
• Ho, D. E., Imai, K., King, G., Stuart, E. A. (2014). How exactly are the weights created? http://r.iq.harvard.edu/docs/matchit/2.4-20/HowExactlyare.html
   Google Scholar
• Hong, G. (2010). Marginal mean weighting through stratification: Adjustment for selection bias in multilevel data. Journal of Educational and Behavioral Statistics, 35(5), 499–531. https://doi.org/10.3102/1076998609359785
   Web of Science ®Google Scholar
• Hong, G. (2012). Marginal mean weighting through stratification: A generalized method for evaluating multivalued and multiple treatments with nonexperimental data. Psychological Methods, 17(1), 44–60. https://doi.org/10.1037/a0024918
   PubMed Web of Science ®Google Scholar
• Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research, 26(3), 329–367. https://doi.org/10.1177/0049124198026003003
   Web of Science ®Google Scholar
• Imai, K., & Ratkovic, M. (2014). Covariate balancing propensity score. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(1), 243–263. https://doi.org/10.1111/rssb.12027
   Web of Science ®Google Scholar
• Imbens, G. W., & Wooldridge, J. M. (2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature, 47(1), 5–86. https://doi.org/10.1257/jel.47.1.5
   Web of Science ®Google Scholar
• Kim, J., Seltzer, M. (2007). Causal inference in multilevel settings in which selection process vary across schools [Working Paper 708]. Center for the Study of Evaluation (CSE), UCLA.
   Google Scholar
• Kim, J. S., & Steiner, P. M. (2015). Multilevel PS methods for estimating causal effects: A latent class modeling strategy. In L. van der Ark, D. Bolt, W. C. Wang, J. Douglas, & S. M. Chow (Eds.), Quantitative psychology research. Springer Proceedings in Mathematics & Statistics (Vol. 140). Springer.
   Google Scholar
• Kim, S., Jeong, Y., & Hong, S. (2021). The impact of ignoring a crossed factor in cross-classified multilevel modeling. Frontiers in Psychology, 12(637645), 637645.https://doi.org/10.3389/fpsyg.2021.637645
   PubMed Web of Science ®Google Scholar
• Konstantopoulos, S., Miller, S. R., & van der Ploeg, A. (2013). The impact of Indiana’s system of interim assessments on mathematics and reading achievement. Educational Evaluation and Policy Analysis, 35(4), 481–499. https://doi.org/10.3102/0162373713498930
   Web of Science ®Google Scholar
• Lai, M. H. C. (2019). Correcting fixed effect standard errors when a crossed random effect was ignored for balanced and unbalanced designs. Journal of Educational and Behavioral Statistics, 44(4), 448–472. https://doi.org/10.3102/1076998619843168
   Web of Science ®Google Scholar
• Lanza, S. T., & Rhoades, B. L. (2013). Latent class analysis: An alternative perspective on subgroup analysis in prevention and treatment. Prevention Science, 14(2), 157–168. https://doi.org/10.1007/s11121-011-0201-1
   PubMed Web of Science ®Google Scholar
• Lee, Y., Nguyen, T. Q., & Stuart, E. A. (2021). Partially pooled propensity score models for average treatment effect estimation with multilevel data. Journal of the Royal Statistical Society: Series A (Statistics in Society), 184(4), 1578–1598. https://doi.org/10.1111/rssa.12741
   Web of Science ®Google Scholar
• Leite, W. L. (2017). Practical propensity score methods using R. Sage Publishing.
   Google Scholar
• Leite, W. L., & Collier, Z. K. (2016, April 8–12). Quasi-experimental program evaluation by combining propensity score matching with cross-classified random effects models [Poster presentation]. American Educational Research Association.
   Google Scholar
• Leite, W. L., Jimenez, F., Kaya, Y., Stapleton, L. M., MacInnes, J. W., & Sandbach, R. (2015). An evaluation of weighting methods based on propensity scores to reduce selection bias in multilevel observational studies. Multivariate Behavioral Research, 50(3), 265–284.https://doi.org/10.1080/00273171.2014.991018
   PubMed Web of Science ®Google Scholar
• Leroux, A. J., & Beretvas, S. N. (2022). Cross-classified random effects models. In A. A. O’Connell, D. B. McCoach, & B. A. Bell (Eds.), Multilevel modeling methods with introductory and advanced applications (pp. 165–208). Information Age Publishing. https://www.infoagepub.com/products/Multilevel-Modeling-Methods-with-Introductory-and-Advanced-Applications
   Google Scholar
• Li, F., Zaslavsky, A. M., & Landrum, M. B. (2013). Propensity Score weighting with multilevel data. Statistics in Medicine, 32(19), 3373–3387.https://doi.org/10.1002/sim.5786
   PubMed Web of Science ®Google Scholar
• Li, Y., Lee, Y., Port, F. K., & Robinson, B. M. (2020). The impact of unmeasured within-and between cluster confounding on the bias of effect estimators of a continuous exposure. Statistical Methods in Medical Research, 29(8), 2119–2139. https://doi.org/10.1177/0962280219883323
   PubMed Web of Science ®Google Scholar
• Li, Y., & Li, L. (2021). Propensity score analysis methods with balancing constraints: A Monte Carlo study. Statistical Methods in Medical Research, 30(4), 1119–1142. https://doi.org/10.1177/0962280220983512
   PubMed Web of Science ®Google Scholar
• Lumley, T. (2004). Analysis of complex survey samples. Journal of Statistical Software, 9(8), 1–19. https://doi.org/10.18637/jss.v009.i08
   Google Scholar
• Luo, W., & Kwok, O-m. (2009). The Impacts of Ignoring a Crossed Factor in Analyzing Cross-Classified Data. Multivariate Behavioral Research, 44(2), 182–212. 10.1080/00273170902794214
   PubMed Web of Science ®Google Scholar
• McKelvey, R. D., & Zavoina, W. (1975). A statistical model for the analysis of ordinal level dependent variables. The Journal of Mathematical Sociology, 4(1), 103–120. https://doi.org/10.1080/0022250X.1975.9989847
   Web of Science ®Google Scholar
• Meyers, J. L., & Beretvas, S. N. (2006). The Impact of Inappropriate Modeling of Cross-Classified Data Structures. Multivariate Behavioral Research, 41(4), 473–497. 10.1207/s15327906mbr4104_3
   PubMed Web of Science ®Google Scholar
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage Publications.
   Google Scholar
• R Core Team. (2019). R: A Language and Environment for Statistical Computing. Vienna, Austria. Retrieved from https://www.R-project.org/
   Google Scholar
• Rickles, J. H., & Seltzer, M. (2014). A two-stage Propensity Score matching strategy for the treatment effect estimation in a multisite observational study. Journal of Educational and Behavioral Statistics, 39(6), 612–636. https://doi.org/10.3102/1076998614559748
   Web of Science ®Google Scholar
• Ridgeway, G., McCaffrey, D., Morral, A., Burgette, L., Griffin, B. A. (2013). Toolkit for weighting and analysis of nonequivalent groups: A tutorial for the twang package. http://cran.r-project.org/web/packages/twang/vignettes/twang.pdf
   Google Scholar
• Rights, J. D., & Sterba, S. K. (2018). Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures. Psychological Methods, 23(3), 434–457. https://doi.org/10.1037/met0000184
   PubMed Web of Science ®Google Scholar
• Rosenbaum, P. R. (1989). Optimal matching for observational studies. Journal of the American Statistical Association, 84(408), 1024–1032. https://doi.org/10.1080/01621459.1989.10478868
   Web of Science ®Google Scholar
• Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the Propensity Score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41
   Web of Science ®Google Scholar
• Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control group using multivariate matched sampling models that incorporate the propensity score. American Statistician, 39(1), 33.
   Web of Science ®Google Scholar
• Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. 10.1037/h0037350
   Web of Science ®Google Scholar
• Schuler, M. S., Chu, W., & Coffman, D. (2016). Propensity score weighting for a continuous exposure with multilevel data. Health Services & Outcomes Research Methodology, 16(4), 271–292.
   PubMedGoogle Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-experimental designs for generalized causal inference. Houghton Mifflin Company.
   Google Scholar
• Shechtman, N., Roschelle, J., Feng, M., & Singleton, C. (2019). An efficacy study of a digital core curriculum for grade 5 mathematics. AERA Open, 5(2), 233285841985048. https://doi.org/10.1177/2332858419850482
   Web of Science ®Google Scholar
• Shi, Y., Leite, W. L., & Algina, J. (2010). The impact of omitting the interaction between crossed-factors in cross-classified random effects modeling. The British Journal of Mathematical and Statistical Psychology, 63(Pt 1), 1–15. https://doi.org/10.1348/000711008X398968
   PubMed Web of Science ®Google Scholar
• Snijders, T. A., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Sage Publishing.
   Google Scholar
• Spybrook, J., & Raudenbush, S. W. (2009). An Examination of the Precision and Technical Accuracy of the First Wave of Group-Randomized Trials Funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31(3), 298–318. 10.3102/0162373709339524
   Web of Science ®Google Scholar
• Steiner, P. M., Kim, J., & Thoemmes, F. (2012). Matching strategies for observational multilevel data. In JSM proceeding (pp. 5020–5032). American Statistical Association.
   Google Scholar
• Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1–21. https://doi.org/10.1214/09-STS313
   PubMed Web of Science ®Google Scholar
• Thoemmes, F. J., & Kim, E. S. (2011). A systematic review of Propensity Score methods in the social sciences. Multivariate Behavioral Research, 46(1), 90–118. https://doi.org/10.1080/00273171.2011.540475
   PubMed Web of Science ®Google Scholar
• Thoemmes, F. J., & West, S. G. (2011). The use of Propensity Scores for nonrandomized designs with clustered data. Multivariate Behavioral Research, 46(3), 514–543.https://doi.org/10.1080/00273171.2011.569395
   PubMed Web of Science ®Google Scholar
• What Works Clearinghouse. (2020). What Works Clearinghouse standards handbook (version 4.1). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWCStandards-Handbook-v4-1-508.pdf
   Google Scholar
• Winpenny, E. M., Corder, K. L., Jones, A., Ambrosini, G. L., White, M., & van Sluijs, E. (2017). Changes in diet from age 10 to 14 years and prospective associations with school lunch choice. Appetite, 116, 259–267. https://doi.org/10.1016/j.appet.2017.05.012
   PubMed Web of Science ®Google Scholar
• Yang, S. (2018). Propensity Score Weighting for Causal Inference with Clustered Data. Journal of Causal Inference, 6(2)10.1515/jci-2017-0027
   PubMed Web of Science ®Google Scholar
• Ye, F., & Daniel, L. (2017). The Impact of Inappropriate Modeling of Cross-Classified Data Structures on Random-Slope Models. Journal of Modern Applied Statistical Methods, 16(2), 458–484. 10.22237/jmasm/1509495900
   Google Scholar
• Zetterqvist, J., Vansteelandt, S., Pawitan, Y., & Sjolander, A. (2016). Doubly robust methods for handling confounding by cluster. Biostatistics, 17(2), 264–276. https://doi.org/10.1093/biostatistics/kxv041
   PubMed Web of Science ®Google Scholar
• Zhao, Q. (2019). Covariate balancing propensity score by tailored loss functions. The Annals of Statistics, 47(2), 965–993. https://doi.org/10.1214/18-AOS1698
   Web of Science ®Google Scholar
• Zubizarreta, J. R., & Keele, L. (2017). Optimal multilevel matching in clustered observational studies: A case study of the effectiveness of private schools under a large-scale voucher system. Journal of the American Statistical Association, 112(518), 547–560. https://doi.org/10.1080/01621459.2016.1240683
   Web of Science ®Google Scholar