A checklist to guide sensitivity analyses and replications of impact evaluations

ABSTRACT

The transparency, reproducibility, and ethics movement has helped promote efforts to improve the rigour and validity of social science research. However, replication research, where the original data is used to check the validity and robustness of the estimations and recommendations in the original paper, has not been widely used as a tool in this field. This article outlines a checklist and standardised process that can guide replication researchers in creating and justifying their replication plans. To create the checklist, we reviewed 31 resources that were identified through unstructured keyword searches. We also reviewed and coded replication studies from 3ie’s Replication Program which includes more than 20 3ie-funded replication papers and guidance on our replication process. There were five categories of potential validation tests and robustness checks that researchers could include. The checklist attributes range from testing specific assumptions for select impact evaluation methods to data transformations or dealing with heterogeneity. We hope that this checklist helps stimulate demand for replication research and promotes the use of replication research as a tool within transparency, reproducibility and ethics.

KEYWORDS:

• Replication
• robustness checks
• sensitivity analysis
• impact evaluation

1. Introduction

The transparency, reproducibility, and ethics (TRE) movement has grown such that TRE has now become a widely accepted and innate part of social science research. Multiple institutions, both research and funders alike, have enacted TRE policies to guide their own research. Within the reproducibility component, there have been substantial efforts to promote computational reproducibility (ability to reproduce statistical code and results used in a study), such as the creation of the Social Science Reproduction Platform. While computational reproducibility can help identify foundational coding discrepancies, it cannot provide insight into the internal validity, model specifications, or analytical choices used to produce the results in the original paper. Replications are an underutilised tool within TRE that can help to support transparency and reproducibility by providing additional verification and insight into the results from an original paper.

The term ‘replication’ can frequently be used as a catchall term and can be used to describe multiple different scenarios (Clemens 2015; Vilhuber 2020). Here, we use it to describe when data from the original study is used to check the validity and robustness of the estimations and recommendations (International Initiative for Impact Evaluation (3ie), n.d.). Replication studies are beneficial to the research world as they can be used to increase a study’s credibility (Brown and Wood 2018). The benefits that replication can offer to decision-making were the motivations behind the International Initiative for Impact Evaluation’s (3ie) replication program. Through this programme, researchers were awarded funding to replicate influential impact evaluations across multiple sectors.

However, replication research is not always positively viewed within the sector. The relationship between replication researchers and original researchers is often affected by perceptions that replication researchers are trying to overturn results rather than to promote science (Gertler, Galiani, and Romero 2018). Part of this perception of replications is due to the lack of standardisation within replication research. For the most part, replication researchers select the analyses that they will focus on guided by their own experiences and knowledge. This ad hoc process of identifying analyses does not support co-creation of the replication with the original authors and does not mitigate negative perceptions of replication research.

With this paper, we aim to provide a resource for impact evaluation researchers to use, either for replication research or for sensitivity analyses of their own work. Building on the taxonomy created by Brown and Wood (2018), we provide a checklist that provides guidance on specific attributes that should be checked based on the impact evaluation method used along with recommended robustness and sensitivity tests. This checklist is not meant to be exhaustive but serves as a starting point that researchers can use when writing their replication plans. We provide guidance on how to integrate this checklist into a transparent process that documents how and why analytical choices are made. By creating a standardised process, we hope that this checklist will encourage conversation between replication and original researchers and ensure that it becomes a key tool within the TRE movement.

2. Creation of checklist

The initial checklist was informed by the framework provided in Brown and Wood’s (2018) article, which provides a broad taxonomy of tips for replication organised into four groups (validity of assumptions, data transformations, estimation methods, and heterogeneous impacts). We expand on their work by including separating assumptions by the identification strategy used as well as by including a broader set of checks to perform based on a review of existing replication. We focused on six common impact evaluation designs to include in the checklist: randomised controlled trials, difference-in-differences, instrumental variable, interrupted time series, matching, and regression discontinuity design. To identify additional attributes and tests to include in the checklist, we followed a two-step approach to identify resources. We first generated a list of foundational resources for the common impact evaluation methodologies that were included in the checklist. We then used an unstructured keyword search within Google Scholar to identify additional resources. In addition, we also used backwards citation tracking to assess if any of the resources cited within an article would also be useful to inform the checklist. Thirty-one resources were identified and reviewed by the study team with the full list provided in the references section. Within the checklist, we included citations to specific resources if they provided guidance on the test or process to check that appropriate attribute.

To further refine the checklist framework, we used 3ie’s database of replication studies. These replications were funded by 3ie between 2014 and 2022 as part of the organisation’s efforts to improve replication within the social sciences. Initially, 3ie’s replications were performed in a somewhat ad hoc manner, with general replication guidelines and expectations, but no specific instructions outlining what to check in various papers. 3ie’s guidelines for replication researchers that they fund, which were originally published in 2012, have been adjusted over time to continually improve transparency in the replication process. The current guidelines for 3ie funded replications discuss what type of replication is to be performed (those focused on internal replication not external validity), which papers would be replicated (impact evaluations within low and/or middle income countries), and requires a replication plan that ensures researchers stick to pre-specified robustness checks, as well as take part in an extensive review process. However, which robustness checks should be included in a replication is not specified, and the pre-specified checks can be deviated from as necessary, provided an explanation is given. Since the replication analyses were primarily determined based on the replication researcher’s experience and intuition, we decided to categorise the replication analyses by Brown and Wood’s (2018) framework. We extracted data from each study and categorised it according to our initial checklist criteria. For analyses that could not be easily categorised, we created additional categories and/or criteria to capture those elements. The summary of the replication study data extraction is presented in the section below.

3. Summary of 3ie’s replication studies

We reviewed 24 3ie funded replication studies, which were published between 2014 and 2022. Most of the original papers that were replicated were Randomised Control Trials (RCTs), though some also employed other identification strategies, sometimes in addition to the RCT: sixteen RCTs, six difference-in-differences, two instrumental variable, one propensity score matching, and three other impact evaluation methods (coarsened exact matching with difference-in-differences, fixed effects with matching, and natural experiment) (Figure 1).

Figure 1. Original paper identification strategy within replication studies (source: 3ie). Studies categorised as ‘other’ used coarsened exact matching with difference-in-difference, fixed effects with matching, and natural experiment.

We then categorised the replication researchers’ methods into the replication categories discussed by Brown and Wood (2018) (Figure 2). The replications employed many of the checklist categories, though in an ad hoc way and not always mentioning the checks performed that did not result in any new information or issues.

Figure 2. Replication analyses by checklist categories (source: 3ie).

The most common checks included adjustments to the estimation method and checks for heterogeneous outcomes. The validity of assumptions and conditions, and standard checks have not been as common in 3ie replication studies.

For instance, in the case of the RCTs, a common estimation method check was to account for clustering with the generalised linear mixed model (GLMM) (four of the RCT replications performed this check). For other identification strategies, replication researchers often included propensity score matching or added in regressors to check the estimation methods. Heterogeneous outcomes were often checked to see if there are different outcomes and additional information to be gleaned from sub-groups. A relatively common validity check for RCTs is the Cox Proportional Hazard (three of the 3ie replications performed this check) or confirming that the pre-treatment groups are balanced. There are many other assumptions by identification strategy that should be checked if possible, such as parallel trends for difference-in-differences, which only two of the six difference-in-differences replications mention checking. Based on our review of the 3ie replications, it is evident that there has not been consistency in the types of checks being performed, and often common checks that are to be expected for certain identification strategies are not mentioned in the replication.

Furthermore, even within the same category, there is variation in how the check is performed. With regard to data transformations, many papers change the way the variables are constructed, and how outliers and missing data are dealt with (for instance, impute instead of drop) since this can significantly influence the results. However, these data transformations also only occur in some of the replications when researchers identified a potential issue, rather than including this as a standard check to the original paper methods. About one-third of the replication studies looked at variable construction and outliers, which can also affect the results. We have included this as a standard check in the checklist that researchers can check this every time.

4. Checklist

There are five main categories in the checklist: 1) Validation of Assumptions/Conditions; 2) Data Transformations; 3) Estimation Methods; 4) Heterogenous Outcomes; 5) Standard Checks. Each section follows the same structure. The first column provides the key attribute/condition that may not have been validated or tested in the original study. The second column provides the recommended test to check if the attribute has not been met. Replication authors are expected to fill in the third column with comments on whether or not that specific attribute needs to be tested. If the authors determine that the attribute should be tested, they should fill out the action column with details on the specific analysis that they will be using. Table 1 provides an illustrative example of how authors could fill out the checklist table. The full complete checklist, including a guidance to authors section, is included in the Online Appendix.

Table 3. Key attributes and recommended tests for the data transformations section of checklist

Key Attribute	Recommended Test/Check*	Comment	Action	Resources
If imputation methods are used	Re-run analyses using complete case analysis where observations with missing data are dropped If random seeds are not used in imputation methods, redo imputation using random seed If random seeds are used, redo imputation using average values Use alternative imputation methods			Lall (2016); Sterne et al. (2009)
If differential attrition is present or suspected	Use inverse probability weighting to account for attrition			Weuve et al. (2012)
If there is non-compliance where participants in the treatment group did not take up treatment	Use per-protocol analysis where non-compliant participants are dropped from sample Use as-treated analysis where participants are analysed according to treatment they took in study			Souza et al. (2016)
If data is suspected to be missing at random (no association between missingness and treatment status as well as outcome) or missing not at random (associated with other missing data)	Conduct complete case analysis by dropping those who have missing data Impute missing data and re-run analyses Use appropriate maximum likelihood models to control for these covariates			Ibrahim, Chu, and Chen (2012); Bell et al. (2013); Souza et al. (2016)

*If tests are numbered, all tests listed must be performed and should be performed in sequential order listed. Bulleted tests indicate that there are multiple tests that could be used and it is up to the replication authors to decide which test to run. Bolded bullet points are recommended tests based on what is recommended in the literature.

Table 4. Key attributes and recommended tests for the estimation methods section of checklist

Key Attribute	Recommended Test/Check*	Comment	Action	Resources
If estimation strategy is suspected to not be appropriate for the study set-up	1) Conduct robustness or goodness-of-fit tests on original estimation strategy 2) Re-run alternative estimation strategy			Romano and Wolf (2005)
If alternative estimation strategies could be used (e.g. use multi-level modelling techniques, parametric vs nonparametric, etc.)	Re-run analyses using alternative estimation strategies			de Souza et al. (2016)
If robustness tests for key parameters or specification choices have not been explicitly tested	Run relevant robustness tests for key parameters or specification choices			N/A
If alternative specifications for (e.g. using likelihood-based variance estimators instead of bootstrap)	Re-run analyses using alternative specifications			Mantobaye, Rea, and Robert Reed (2018)

*If tests are numbered, all tests listed must be performed and should be performed in sequential order listed. Bulleted tests indicate that there are multiple tests that could be used and it is up to the replication authors to decide which test to run. Bolded bullet points are recommended tests based on what is recommended in the literature.

Table 5. Key attributes and recommended tests for the heterogenous outcomes section of checklist

Key Attribute	Recommended Test/Check*	Comment	Action	Resources
If there are demographic subgroups that should have been analysed (e.g. age, sex, socioeconomic status, race, ethnicity, etc.) Subgroups could be created based on demographic variables, geographic communities, clinically-relevant distinctions, or known prior experience (e.g. previous exposure to intervention, prior market participation, prior incarceration, etc.).	Re-run primary impact analyses subsetting dataset to specific subgroup Run treatment effect heterogeneity analyses to identify variables that had the greatest change in average treatment effect			Imai and Ratkovic (2013)
If there are geographic subgroups that should have been analysed (e.g. community, region, district, province, etc.)	Re-run primary impact analyses subsetting dataset to specific subgroup Run treatment effect heterogeneity analyses to identify variables that had the greatest change in average treatment effect			Imai and Ratkovic (2013)
If there are clinically-relevant subgroups that should have been analysed (e.g. prior history of disease X, prior treatment with Y, existence of other conditions, etc.)	Re-run primary impact analyses subsetting dataset to specific subgroup Run treatment effect heterogeneity analyses to identify variables that had the greatest change in average treatment effect			Imai and Ratkovic (2013)
If there are any other subgroups that should have been analysed (e.g. policy change, previous exposure to the intervention, prior market participation, prior incarceration, etc.)	Re-run primary impact analyses subsetting dataset to specific subgroup Run treatment effect heterogeneity analyses to identify variables that had the greatest change in average treatment effect			Imai and Ratkovic (2013)

*If tests are numbered, all tests listed must be performed and should be performed in sequential order listed. Bulleted tests indicate that there are multiple tests that could be used and it is up to the replication authors to decide which test to run. Bolded bullet points are recommended tests based on what is recommended in the literature.

Table 6. Key attributes and recommended tests for the standard checks section of checklist

Key Attribute	Recommended Test/Check*	Comment	Action	Resources
Concordance with pre-analysis plan	Perform primary outcome analyses detailed in pre-analysis plan but not conducted in original study			N/A
If covariate balance tests are not provided or covariate imbalance is suspected	1) Check covariate balance across study arms, for all sub-group analyses and across all time points in the study 2) Use inverse probability weighting to account for possible attrition			Weuve et al. (2012)
If outliers were not excluded	Drop outliers and re-run primary analysis Run robust regression			de Souza et al. (2016)
If there are alternate variable transformations	Re-run primary analyses using alternate variable transformations (i.e. using ordinal or continuous variable instead of primary)			N/A
If standard errors were not adjusted for clustering in clustered study designs	Re-run analyses adjusting standard errors for clustering			N/A
If ex-post power calculations were not performed	Use standard error-effect size method to assess power as described in Tianet al. (2022).			Tian et al. (2022)

*If tests are numbered, all tests listed must be performed and should be performed in sequential order listed. Bulleted tests indicate that there are multiple tests that could be used and it is up to the replication authors to decide which test to run. Bolded bullet points are recommended tests based on what is recommended in the literature.

Table 1. Example of filled out table

Key Attribute	Recommended Test/Check	Comment	Action	Resources
If the relevance condition has not been met	Check association between instrument and exposure	Study did not explicitly test this condition even though IV was used	Run t-test to assess association between instrument Z and exposure X
Concordance with pre-analysis plan	Perform primary outcome analyses detailed in pre-analysis plan of original study but not conducted	Original study authors conducted all analyses detailed in their pre-analysis plan	No additional action is needed	N/A

4.1 Validity of assumptions and conditions

This section presents key attributes of identification strategies that should be validated (Table 2). General causal inference attributes were identified from foundational impact evaluation texts detailed in the methods section. These attributes cover key assumptions such as the ‘Stable Unit Treatment Values Assumption’ (SUTVA), consistency, and ignorability. These general assumptions are applicable to any impact evaluation study and authors should comment on whether or not the assumptions were met or adequately considered in the study to be replicated.

Table 2. Key attributes and recommended tests for the validity of assumptions and conditions section of checklist

Key Attribute	Recommended Test/Check*	Comment	Action	Resources
Identification Strategy: General/Universal
If the ‘no interference’ component of the Stable Unit Treatment Values Assumption (SUTVA) is suspected to be violated (e.g. geographic proximity of intervention and control groups could lead to contact)	Generally, SUTVA is assumed to be true 1) Check with study authors for potential violation of this attribute and describe justification in replication paper 2) If potential geographic spillover, use ‘fried egg model’ or comparable approach to draw new analytical boundaries around treatment and control clusters to remove any contact			Hayes and Moulton (2017)
If the ‘one treatment’ component of SUTVA is suspected to be violated	Generally, SUTVA is assumed to be true Check with study authors for potential violation of this attribute and describe justification in replication paper			N/A
If observed outcomes are suspected to be different than the potential outcomes (i.e. consistency assumption is violated)	Generally, consistency is assumed to be true Check with study authors for potential violation of this attribute and describe justification in replication paper			N/A
If omitted variable bias is suspected to have affected the analysis and the ignorability assumption is violated (i.e. given the pre-treatment covariates accounted for in the analysis, the treatment assignment is independent of the potential outcomes)	1) Identify covariates that were omitted from original analysis and may be associated with outcome 2) Re-run regression analyses with and without added covariates 3) Compare sensitivity statistics such as ‘robustness value’ and ‘partial R^2’ between analyses			Cinelli and Hazlett (2020)
Identification Strategy: Randomised Controlled Trials/Clustered Randomised Trials
If there is non-compliance where participants in the treatment group did not take up treatment	Use per-protocol analysis where non-compliant participants are dropped from sample Use as-treated analysis where participants are analysed according to treatment they took in study			de Souza et al. (2016)
If covariates are imbalanced at baseline and authors did not adjust for imbalance or explain why imbalance was not adjusted for	Re-run analyses adjusting for imbalanced baseline covariates Stratify analysis by imbalanced covariates Use matching to control for imbalanced covariates			de Souza et al. (2016)
If contamination is suspected	Use instrumental variable methods to estimate the local average treatment effect with random assignment as the instrumental variable Depending on the nature of the contamination bias, potentially revise the type of programme impact being measured by assessing the lower bound of the programme impact (e.g. if control group members participate in other similar programmes, compare impact of RCT programme relative to other similar programmes)			Glewwe and Todd (2022)
Identification Strategy: Instrumental Variable
If the relevance condition has not been explicitly tested (i.e. instrument is not associated with treatment)	Check association between instrument and treatment			Jeremy and Swanson (2018)
If the exclusion restriction (i.e. effect of instrument on outcome is solely through treatment) is assumed to be violated	Generally, the exclusion restriction is assumed to be true 1) Check with study authors for potential violation of this attribute and describe justification in replication paper 2) Use omitted variable bias framework if exclusion restriction is assumed to be violated due to omitted confounders			Cinelli and Hazlett (2022); Jeremy and Swanson (2018)
If the exogeneity assumption (i.e. instrument is not confounded with outcome) is assumed to be violated	Use Anderson-Rubin test to assess effect size of the instrument on the exposure			Wang et al. (2018)
If the homogeneity assumption (i.e. instrument is not an effect modifier of treatment and outcome) is not met	Estimate effect estimate among the ‘always-takers’, the ‘compliers’, and the ‘never-takers’ Check for differences in instrument strength across covariates			Jeremy and Swanson (2018)
If overidentification (i.e. more than one proposed instrument is available) is present	Conduct overidentification restriction tests, such as the Hausman test, and compare p-value Use alternative causal effect estimation strategy (i.e. pool individual estimates, use all instruments in same model, or combine instruments into a summary risk score) Conduct sensitivity analysis as described in Small (2007)			Jeremy and Swanson (2018); Small (2007)
If the monotonicity assumption (i.e. instrument affects exposure in only one direction among entire study population) is not met	For continuous treatments, can assess violation by visually assessing the difference in treatment level for each level of the instrument Can also check imbalances between binary treatment, binary instrument, and binary outcome			Jeremy and Swanson (2018)
Identification Strategy: Regression Discontinuity Design
If it is suspected that the running variable was manipulated	Assess continuity by assessing density of the observations			Smith et al. (2016)
If the discontinuity assumption (i.e. there is a discontinuous change in the treatment at the cut-off point) is not met	Visually inspect the running variable and outcome variable			Smith et al. (2016)
If the continuity assumption (continuous change in potential outcome at the cut-off) is not met	Visually inspect the outcome trends with covariates Use placebo cut-offs to assess discontinuity			Bärnighausen et al. (2017)
If the analysis is based on ‘local randomisation’ around the threshold (the running variable is viewed as random and the window around the cut-off is such that treatment assignment is as if it was randomised)	Compare distribution of covariates for those right above and below the threshold (akin to ‘balance tables’)			Bärnighausen et al. (2017); Branson and Mealli (2019)
If the exchangeability assumption (i.e. observations around the cut-off are similar to each other across observable covariates) was not demonstrated	Plot distribution of baseline characteristics by the running variable Compare distribution of covariates for those right above and below the threshold			Smith et al. (2016)
If it suspected that the functional form has not been properly specified	Drop outermost points (outermost 1%, 5%, and 10%) in sample to see if estimated impacts remain constant Use F-tests to eliminate overly restrictive model specifications			Jacob et al. (2012)
If it suspected that the optimal bandwidth has not been selected	Use the cross-validation procedure as described by Jacob et al. (2012) Use the plug-in procedure as described by Jacob et al. (2012) Rerun analyses with a reduced sample and then with an expanded sample to assess sensitivity of impact estimates			Jacob et al. (2012)
Identification Strategy: Difference-in-Differences
If parallel trends assumption (i.e. treatment group without the treatment would have followed same trend as control group) is suspected to be violated	Graph the outcome trends for the treatment and control group to compare pre-intervention trends using event-study plot accompanied by pre-test diagnostics using non-inferiority approaches as described in Bilinski and Hatfield (2020) or Dette and Schumann (2020) For studies with more than two time points, assess group-specific linear trends using a regression that interacts group effects with time Use methods that condition models on covariates as described in Roth et al. (2023) Incorporate in time-varying treatment effects if treatment effect could vary with time Use alternative bounding methods such as bounds using pre-trends or bracketing as described in Roth et al. (2023)			Bilinski and Hatfield (2020); Wing, Simon, and Bello-Gomez (2018); Dette and Schumann 2020; Roth et al. (2023)
If additional data is available beyond treatment ending	Re-run analyses with extended data comparing treatment effects after treatment has ended to determine if effect size is maintained			Bärnighausen et al. (2017)
If additional data is available on separate control group	Re-run analyses with a different control group to assess if findings remain similar			Bärnighausen et al. (2017)
If difference-in-difference methods used in original analysis is suspected to not be credible	Re-run analyses with outcomes that theoretically should not be affected by treatment to see if any effects on alternate outcomes are found			Bärnighausen et al. (2017)
If there are multiple treatment periods	Use heterogeneity-robust estimator for staggered treatment timing			Roth et al. (2023)
If study population clusters are sampled from a super-population	Use cluster-robust methods if large number of treated and untreated clusters Use alternative methods that condition on cluster-level shocks and treat as violations of parallel trends or use permutation-based methods if small number of treated clusters Use design-based justification where units in data are fixed and treatment assignment is stochastic for inference if super-population can’t be imagined			Roth et al. (2023)
Identification Strategy: Propensity Score Matching
If the common support assumption (i.e. there is sufficient overlap between the treatment and control units to find a match) is not explicitly tested	Conduct visual inspection of the propensity score distributions for treatment and comparison Drop observations that are outside the region of common support			Heinrich, Maffioli, and Vázquez (2010)
If the conditional independence assumption (i.e. that after controlling for a set of observable covariates that the potential outcomes are independent of treatment status) is not explicitly tested	Check balance in treatment and control groups after matching Use alternative variable specifications to improve covariate balance			Heinrich, Maffioli, and Vázquez (2010)
If the robustness of the matching technique has not been explicitly tested	Re-run analyses using other matching techniques (e.g. nearest neighbour, caliper, kernel, local-linear, etc.)			Heinrich, Maffioli, and Vázquez (2010)
If it is suspected that there is covariate measurement error or unobserved confounding in propensity score matching	Correct for classical or systematic differential measurement errors using VanderWeele and Arah’s sensitivity analysis Correct for classical, systematic differential or heteroscedastic differential measurement errors using Rosenbaum’s sensitivity analysis or propensity score calibration			Rudolph and Stuart (2018)
Identification Strategy: Interrupted Time Series
If time-varying confounders have not been included in the original analysis	Re-run analysis including time-varying confounders as controls			Bernal, Cummins, and Gasparrini (2017)
If seasonality has not been controlled for but could impact the outcomes of interest	Conduct time-stratified models Use periodic functions Use flexible spline functions			Bernal, Cummins, and Gasparrini (2017); Bhaskaran et al. (2013)
If overdispersion is present (variance in outcome is greater than the expected count)	Incorporate in scaling adjustments into the model to account for overdispersion			Bernal, Cummins, and Gasparrini (2017); Bhaskaran et al. (2013)
If autocorrelation (consecutive observations are more similar than those further apart) has not been explicitly tested	Examine the residuals plot and the partial autocorrelation function For normally distributed data, conduct tests such as Breusch-Godfrey test Adjust for autocorrelation using Prais regressions or autoregressive integrated moving average			Bernal, Cummins, and Gasparrini (2017)
If the continuity assumption (i.e. that no other interventions or covariates affect the outcomes so that the potential outcomes are continuous at the threshold as individual moves from treatment to control)	Visually inspect the outcome trends Control for potential covariates that may affect the treatment continuity			Bärnighausen et al. (2017)

*If tests are numbered, all tests listed must be performed and should be performed in sequential order listed. Bulleted tests indicate that there are multiple tests that could be used and it is up to the replication authors to decide which test to run. Bolded bullet points are recommended tests based on what is recommended in the literature.

We then present key assumptions and considerations by the six identification strategies included in this research. For study designs that combine multiple identification strategies, the replication researcher should comment on the assumptions for each included identification strategy to ensure they were met.

In some cases (e.g. SUTVA or instrumental variable’s exclusion restriction), the assumption is assumed to be met and may not be able to be tested. In those cases, we recommend that the replication researcher consider reaching out to the original study authors to comment on whether there were any suspected violations and include that explanation in the checklist.

4.2 Data transformations

This section provides various data transformations that should be checked and/or implemented if not included in original analysis (Table 3). The data transformations were informed by Brown and Wood (2018) as well as through a keyword search in the literature. Key attributes for this section primarily focus on handling missing data, such as attrition, missing data patterns, and non-compliance. Additional data transformations are included in the standard checks section as they are recommended to be tested in every replication study.

4.3 Estimation methods

The following considerations are to test the appropriateness of the estimation strategy and determine alternatives (Table 4). This section primarily focuses on the estimation strategy chosen by the original researcher. It includes testing alternative estimation strategies (e.g. using survival modelling to assess incidence ratios instead of general estimating equations) or testing alternative estimation specifications (e.g. using likelihood-based variance estimators instead of bootstrapped estimators). We have not included specific guidelines for each estimation strategy type as there are numerous regression models that could be chosen. Instead, we recommend that the replication researcher provides justification and references to explain why they have chosen that estimation strategy and how this analysis will complement the original results. It is important to note that the aim in choosing a different estimation strategy than the original research should not be to invalidate the original results but to provide insight. For example, if the original research aggregated individual data to the cluster level before conducting the regression, the replication researcher may choose to use individual-level data in the regression as a measure of controlling for within cluster variation.

4.4 Heterogeneous outcomes

The following considerations are to assess for potential heterogeneity (Table 5). A new entry should be used for each separate heterogeneity analysis. The heterogenous outcomes selected should be justified in the checklist below based on the original paper as well as sector-specific references. Replication researchers should be cautious to note that there may not be sufficient power to fully test these sub-group analyses. These considerations along with potential issues related to multiple hypothesis testing and imbalance between sub-groups (especially for randomised studies where randomisation may have been implemented at a higher unit of analysis) should be commented upon in the checklist and in the replication paper when interpreting results.

4.5 Standard checks

The following considerations are recommended to be checked in every replication study (Table 6). These considerations include concordance with the pre-analysis plan, assessing covariate balance, various data transformations (such as treatment of outliers, variable transformations, and clustered standard errors), and ex-post power calculations that may be relevant in each replication study.

5. Conclusion

In this paper, we outlined a checklist that can be used to guide both replication research as well as sensitivity analyses within an original study. We recognise that this checklist is not exhaustive and that there are additional assumptions or robustness tests that could be included. Rather than creating an exhaustive list, we aimed to create this guidance document that can serve as the starting point for a replication researcher.

This work complements the broader TRE movement as it continues to support transparency and reproducibility within the social science field. It also complements other resources, such as the Guide for Accelerating Reproducibility in the Social Sciences (Berkeley Initiative for Transparency in the Social Sciences 2022). Along with these other resources, our checklist can be used to help identify which analyses should be included and provides a clear justification for why it should be included.

This checklist is a value-add into the impact evaluation world as it includes specific methods-based assumptions along with guidance on estimation methods, how to treat heterogeneity, and data transformations into one resource. Each of these attributes are found in other resources but as of yet, have not been compiled into one guidance document. The checklist and the associated process that comes with completing the checklist provides a transparent way for replication researchers to guide their analyses. It also builds in communication with the original authors to ensure that the replication research is co-created. Application of this checklist can help mitigate the risk that replication researchers deliberately seek to overturn the original paper results or are perceived to be doing so. We recommend that replication researchers include the completed checklist in their pre-analysis plan as well as an appendix within their completed study to further promote transparency on their replication. We hope that this checklist stimulates demand for replication research and that researchers continue to revise and add to the checklist as a public good.

Acknowledgements

We would like to acknowledge Marie-Eve Augier and Katherine Quant for their support in managing this project. We also would like to thank Sayak Khatua and Carolyn Huang for reviewing the initial drafts of the replication checklist.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/19439342.2024.2318695.

Additional information

Funding

This work was supported by the Bill and Melinda Gates Foundation under grant number OPP1034373.

Notes on contributors

Sridevi K. Prasad

Sridevi K Prasad is a Senior Research Associate and Leader of Trainings at the International Initiative for Impact Evaluation (3ie). At 3ie, Sridevi currently leads 3ie’s Virtual Trainings Initiative, which seeks to develop capacity building resources in impact evaluation methodology for researchers and policymakers. Sridevi also led evidence mapping research on the associations between water, sanitation, and hygiene achievements and higher-level outcomes related to prosperity, stability and resilience for USAID’s Center for Water Security, Sanitation, and Hygiene. She also led and conducted two replication studies on HIV combination prevention interventions. She is currently providing technical support to an impact evaluation within the financial sector for the European Bank of Reconstruction and Development. She has a Master’s in Public Health, specialising in Monitoring & Evaluation as well as Epidemiology and Biostatistics, from Boston University and a Bachelor of Arts in Molecular and Cell Biology from the University of California, Berkeley.

Fiona Kastel

Fiona Kastel is a Research Associate at the International Initiative for Impact Evaluation (3ie). At 3ie, Fiona provides research, programme management, and data analytics support for multiple programmes and initiatives on agriculture, education, finance, health, and policy and institutional reform. She leads 3ie’s Data Innovations Group and provides technical support on impact evaluations, evaluability assessments, evidence synthesis products, and virtual trainings. Her primary projects include a geospatial impact evaluation of an agricultural intensification programme in Niger, an impact evaluation of the European Bank for Reconstruction and Development’s COVID-19 Solidarity Package, and the strengthening evidence and economic modelling partnership with Millennium Challenge Corporation. Fiona holds a Master of Public Affairs from Brown University and a Bachelor of Science in Quantitative Analysis of Markets and Organization from the University of Utah.

Suvarna Pande

Suvarna Pande is a Research Associate at the International Initiative for Impact Evaluation (3ie). At 3ie, Suvarna contributes to the production of systematic reviews, replication studies, and evidence gap maps of research on the effectiveness of economic and social interventions in low- and middle-income countries. She provides research support at all stages of the review process, from literature synthesis to data and meta-analysis. She is particularly interested in systematic, critical assessment of behavioural interventions in experimental and quasi-experimental designs. Suvarna is finishing her PhD in Behavioural Development Economics from University of East Anglia, UK on the Psychology of poverty, its effect on cognitive systems and risk and social preferences. She holds an MPhil and Masters in Economics from Jawaharlal Nehru University and a Bachelors in Economics University of Delhi.

Chenxiao Zhang

Chenxiao Zhang is a Data Scientist at the World Bank. He interned at the International Initiative for Impact Evaluation (3ie) to identify new robustness checks and approaches to causal inference techniques as well as to conduct research on how remote sensing techniques can be implemented in impact evaluations. Chenxiao holds a Masters of Science in Data Science for Public Policy from Georgetown University and a Bachelors of Social Science in Sociology and Political Science from Hong Kong Baptist University.

Douglas M. Glandon

Douglas Glandon is Chief of Strategic Initiatives and Leader of Methods Development at the International Initiative for Impact Evaluation (3ie). At 3ie, Douglas designs, leads, and quality assures projects in evidence generation, synthesis, and knowledge translation, including portfolios of work with USAID and the Millennium Challenge Corporation. As Leader of Methods Development, he also guides the organisation in updating and enhancing our methods and approaches for generating timely, policy-responsive, rigorous evidence to inform international development policy, programming, and investments. Douglas is particularly interested in the intersection of data science and econometrics and measuring the costs and effectiveness of complex and multi-sectoral programmes. Douglas has a PhD in Health Systems from the Johns Hopkins Bloomberg School of Public Health, a Master’s in Public Health from the Tufts University School of Medicine, and a Bachelor of Arts in International Relations and Community Health from Tufts University.