![Sample our Education journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5932/TOC-Subject-Sample-2021-Education.jpg"})
Abstract
Replication studies have been promoted as a means to investigate the fragility or robustness of findings from prior studies. However, less well known is that replication studies can be done with nonexperimental or secondary datasets and are not just for experimental studies. We present a framework of different types of replication studies with nonexperimental or secondary data and provide examples in the context of school psychology. We show that replication studies can be used as robustness checks, as a means of testing the generalizability of existing theories, and as a way of extending findings of prior studies. We advocate for the thoughtful use of replication studies and highlight some of the benefits of replications.
IMPACT STATEMENT
The replication rate for articles in school psychology journals (2011–2020) was 0.86%.
Although replication efforts largely pertain to experimental studies, we discuss replication studies using secondary or nonexperimental datasets.
We show a typology of four types of replication studies and provide examples in the context of school psychology research.
ASSOCIATE EDITOR:
DISCLOSURE
No potential conflict of interest was reported by the author(s).
Notes
1 The TED talk of one of the co-authors on the topic has, as of January 31, 2024, over 70 million views (see https://www.ted.com/talks/amy_cuddy_your_body_language_may_shape_who_you_are?language=en).
3 The journal has dropped Quarterly from its name and is now known as School Psychology and is published bimonthly.
4 As school psychology is a multidisciplinary field (with articles published in several journals which do not have the words “school” and “psychology” in them), this is likely a lower bound estimate. In another count, but limited to a two year period, 6% of intervention studies were classified as replication studies in a review of 189 studies from nine school psychology journals (McGill et al., Citation2018). Replications are important as intervention studies help form the basis of understanding and recommending evidence-based practices in school psychology (Shaw et al., Citation2019).
7 See here https://authorservices.taylorandfrancis.com/data-sharing/share-your-data/data-availability-statements/.
8 Approximately only a quarter of authors who provided a ‘data upon reasonable request’ statement actually provided data (Hussey, Citation2023) when contacted with a data request though this is likely to be field dependent as well (Tedersoo et al., Citation2021). A reviewer had commented that, based on experience as an editor who has requested data in the past, this is likely an understatement.
9 As pointed out by a reviewer, the use of open-source software such as R, Jamovi, or JASP (as opposed to commercial software) can also help with the reproducibility of results.
10 See for example: https://data.mendeley.com/and https://zenodo.org/.
11 As an example, Silberzahn et al. (2018) provided a single dataset to 29 research teams and gave each team a common research question: Were soccer players with a dark skin tone more likely than those with light skin tone to receive red cards from referees? The analytic teams could analyze the data in the manner they deemed fit and results ranged (in odds ratios; ORs) from 0.89 to 2.93 (Mdn = 1.31) with 31% indicating nonstatistically significant findings and the majority indicating statistically significant results. For other examples of the “many analysts, one dataset” approach in different disciplines, see Breznau et al. (Citation2022), Huntington-Klein et al. (Citation2021), and Gould et al. (Citation2023).
13 For details on cross cohort comparisons, see https://nces.ed.gov/pubs2008/2008347_2.pdf
14 Which are in essence Type III replications.
15 Available at https://nces.ed.gov/datalab/onlinecodebook.
16 The public-use documentation and data files are available at: https://nces.ed.gov/ecls/kinderdatainformation.asp and the replication syntax for the analysis is available at: https://francish.net/post/prior-problem-behavior-and-suspensions-a-replication/.
17 The study was first initiated with elementary school students, though due to delays in the publication process, the middle and high school studies appear to have come out earlier.
19 The entire TIMSS datasets for different years and participating countries can be downloaded from https://timssandpirls.bc.edu/timss-landing.html.
20 See https://psycnet.apa.org/PsycARTICLES/journal/spq/mostfrequentkeywords as of February 28, 2024.
21 Another possibility would be to provide preprints of their manuscript.
Additional information
Notes on contributors
Francis L. Huang
Francis L. Huang, PhD is a professor in the Statistics, Measurement, and Evaluation in Education program in the College of Education and Human Development at the University of Missouri and the methodology co-director of the Missouri Prevention Science Institute, 16 Hill Hall, Columbia, MO 65211; [email protected]. His research focuses on both methodological (e.g., analysis of clustered data) and substantive (e.g., school climate, bullying, disparities in disciplinary sanctions) areas of interest.
Anna B. Huang
Anna B. Huang is the director of the Early Childhood Program at the Moberly Area Community College. She is a doctoral candidate in the Educational Leadership and Policy program at the University of Missouri.