Overlap Violations in Clustered Observational Studies of Educational Interventions

Abstract

In education settings, treatments are often non-randomly assigned to clusters, such as schools or classrooms, while outcomes are measured for students. This research design is called the clustered observational study (COS). We examine the consequences of common support violations in the COS context. Common support violations occur when the covariate distributions of treated and control units have substantial areas of non-overlap. Such violations are likely to occur in a COS, especially with a small number of treated clusters. We provide a comprehensive review of methods for overlap violations in the context of COS designs. We provide an overview of diagnostic tests and trimming methods to ensure overlap holds for the distributions of treated and control covariates. We then outline how trimming changes the estimand and how profiling can be used to understand the causal quantity for which overlap holds. Finally, we demonstrate how steps to achieve adequate overlap can result in very narrowly defined causal effects that may have little policy relevance. We use data on Catholic schools to illustrate concepts throughout.

Keywords:

• Causal inference
• clustered observational studies
• hierarchical/multilevel data
• overlap

Open Scholarship

This article has earned the Center for Open Science badges for Open Data through Open Practices Disclosure. The data are available through the R package matchMulti. For more information, see https://cran.rstudio.com/web/packages/matchMulti/index.html.

Open Research Statements

Study and Analysis Plan Registration

There is no study and analysis plan registration associated with this manuscript.

Data, Code, and Materials Transparency

The data that support the findings of this study are an openly available subsample of the 1982 High School & Beyond survey. Members of our team have incorporated the data into the R package matchMulti as “catholic_schools”: https://cran.rstudio.com/web/packages/matchMulti/index.html. The code for producing analyses in this article is available on Harvard Dataverse: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/6ITHBH.

Design and Analysis Reporting Guidelines

This manuscript was not required to disclose the use of reporting guidelines, as it was initially submitted before JREE mandating open research statements in April 2022.

Transparency Declaration

The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Replication Statement

This manuscript reports an original study.

Additional information

Funding

We gratefully acknowledge funding support for this work from the Spencer Foundation. The opinions expressed here do not necessarily reflect those of the Spencer Foundation. All errors are our own.