Accelerating Literacy for Adolescents (ALFA): Evaluating ALFA Lab Using a Regression Discontinuity Study

Abstract

This article investigates the impact of assignment to the Accelerating Literacy for Adolescents (ALFA) Lab, an additional semester-long class taken during an elective period, on ninth-grade student reading achievement, motivation, and frequency. We conducted a regression discontinuity (RD) study where 1,378 students from diverse high schools in four states over 3 years were assigned either to the treatment (ALFA, n = 540) if their score on a pre-assignment reading measure fell below a threshold cutoff or to business as usual (BAU, n = 838) if they scored at or above the cutoff. The study found positive but nonsignificant effects on reading achievement, motivation, and frequency. Bayesian sensitivity analyses suggest that positive effects are more likely than not. We discuss several issues that may have contributed to the failure to detect significant impacts: Student absences, student attrition, and limited instructional time due to the COVID-19 pandemic.

Keywords:

• Adolescent literacy
• reading motivation
• regression discontinuity

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Open Research Statements

Study and Analysis Plan Registration

There is no registration associated with this study.

Data, Code, and Materials Transparency

The data underlying the results reported in this manuscript are openly available on openICPSR: https://www.openicpsr.org/openicpsr/project/190621/version/V1/view. The intervention curriculum materials (teachers manual, student pages, and additional materials) used in this study are available at https://alfa.every1graduates.org.

Design and Analysis Reporting Guidelines

Not applicable.

Transparency Declaration

The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Replication Statement

This manuscript reports an original study.

Open Scholarship

This article has earned the Center for Open Science badges for Open Data. The data are openly accessible at https://openicpsr.org/openicpsr/project/190621/version/V1/view.

Notes

1 Whole-school approaches that did not include organizational elements such a teacher teams or schoolwide instructional strategies had a zero effect size.

2 Although use of a single, uniform assessment score as the assignment variable would have been ideal, schools were reluctant to agree to participate if the administration of an additional assessment was required. To encourage participation, the research team agreed to work with each site to make use of existing information for assignment purposes.

3 STAR scales scores range from 0 to 1,400. Given that our sample of students included those most in need of academic assistance, we calculated a conditional standard error of measurement (CSEM) as $\mathit{\text{CSEM}}={\sigma }_{\mathit{\text{STAR}}}\sqrt{1-r}$, using the standard deviation of STAR scores for our sample and the reported reliability of 0.90. Sample scale scores ranged from 8 to 1,345, with a standard deviation of 278.4, yielding an CSEM of 88.04.  Renaissance Learning (2023)  reports an overall CSEM of 54 and a CSEM of 70 for ninth grade. Though our sample-based CSEM of 88 suggests less reliability for the STAR, the computer-adaptive administration nature of the Star test may ensure a more consistent SEM across a range of scores. Thanks to a reviewer for pointing this out.

Additional information

Funding

This study was funded by the U.S. Department of Education’s Institute of Education Sciences (R305A180154).