![Sample our Education journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5932/TOC-Subject-Sample-2021-Education.jpg"})
Abstract
This study assesses the impact of a 2-year virtual coaching program on teacher and student outcomes. The program—Feedback on Alignment and Support for Teachers (FAST)—was intended to deepen teachers’ understanding of their state’s standards and improve the alignment of their instruction with the standards. Relying on a multisite school-level randomized controlled trial, we assessed the impact of the FAST program in fourth-grade math and fifth-grade English language arts (ELA). We found that the program had a positive impact on the alignment of teachers’ instruction with state standards for both math and ELA teachers (effect sizes = 0.70 and 0.40, respectively), although the impact was statistically significant only for math teachers. In contrast to a growing body of evidence on the benefits of instructional coaching, the program’s impact on student achievement, was in the negative direction for both fourth-grade mathematics and fifth-grade ELA (effect sizes = −0.07 and −0.10, respectively), although the impact on achievement was statistically significant only for ELA. These results suggest that improving instructional alignment alone may not be sufficient for improving student achievement.
Notes
1 Five states have since repealed the CCSS they originally adopted and replaced them with their own CCR standards (Common Core State Standards Initiative, Citation2021).
2 The project director of this study participated in the development of the PD program and oversaw the implementation of the program during the impact evaluation. No other members of the program development team were involved in the evaluation.
3 Later meta-analyses of rigorous studies of teacher PD programs found that teacher coaching programs, on average, had positive impacts on both teachers’ instruction (Garrett et al., Citation2019; Kraft et al., Citation2018) and student achievement (Kraft et al., Citation2018).
4 Details about the development of the program are available upon request.
5 Sources for the videos and resources used in the CAST meetings included state department of education websites and professional organizations, such as Student Achievement Partners.
6 See Polikoff et al. (Citation2020) for a discussion of the process by which the SEC content languages for math and ELA were refined for the FAST program.
7 Teachers used the following rules to identify the target students: the median student was the student who fell in the middle of the prior year’s state test score distribution within the class, and the target EL and SWD students were those who were most representative of the students in those subgroups in the class.
8 For the FAST program, experts independently coded each standard intended for fourth-grade math or fifth-grade ELA in a school year using up to six topic and cognitive demand pairs per standard. The coders then met to resolve discrepancies in their coding, creating a master list of codes for the collection of standards associated with each focal grade and subject.
9 For the instructional alignment impact analyses, the school-level overall and differential attrition rates were 14.5% and 0.5%, respectively, for Grade 4 math, and they were 19.2% and 12.5% for Grade 5 ELA. The teacher-level overall and differential attrition rates were 13.0% and 4.6%, respectively, for Grade 4 math, and 21.1% and 2.9% for Grade 5 ELA. These teacher-level attrition rates were calculated using teachers teaching the target grades and subjects in study schools in spring Year 2 as the reference samples, consistent with the WWC (Citation2022) guidance. For student achievement impact analyses, the school-level overall and differential attrition rate were both zero for Grade 4 math and 1.9% and 4.2%, respectively, for Grade 5 ELA. Since our student achievement analyses were based on administrative data, it is reasonable to assume student-level attrition was low, per the latest guidance provided by the WWC (Citation2022).
10 Perfect alignment occurs when for every topic–cognitive demand pair, equals
A complete lack of alignment occurs when for every topic–cognitive demand pair, either
or
(but not both) takes a value of 0.
11 In a conventional CACE analysis for RCTs based on the seminal work by Angrist et al. (Citation1996), the definition of compliers and the measurement of treatment participation/take-up are at the level of treatment assignment—typically the individual level. For this school-level RCT, however, treatment assignment occurred at the school level, but we defined compliers and measured treatment participation at the teacher level, following the approach to CACE estimation for two-level clustered RCTs described by Schochet and Chiang (Citation2009).
12 While path coefficient a is a causal estimate based on an ITT impact analysis, path coefficient b is correlational. Therefore, unlike the ITT impact estimates, mediation effect estimates do not have a causal interpretation.
13 We also conducted analyses with listwise deletion, and the results are largely similar.
14 The alignment index values presented in Table 3 are consistent with those found in prior research on the alignment of teachers’ instruction with state standards, which are generally less than 0.50, with means ranging from 0.20 to 0.30 depending on the subject and alignment target (Polikoff & Porter, Citation2014).
15 For the CACE analyses of instructional alignment, the first-stage estimates indicate that the differences in teachers’ FAST participation rate between the treatment and control schools were, on average, 61% points for fourth-grade math teachers and 66% points for fifth-grade ELA teachers. Both were statistically significant at the 0.001 level, indicating that random assignment was a sufficiently strong instrument for the CACE analyses.
16 Other recent studies of teacher PD have found a significant positive impact on instruction but not achievement. See Garet et al. (2016) for a discussion; The lack of statistical significance for some of the effect estimates from this study needs to be interpreted with caution, as this study was somewhat underpowered for detecting effects on most of the outcomes (realized minimum detectable effect sizes = 0.57 and 0.76 for instructional alignment and 0.15 and 0.12 for student achievement in math and ELA, respectively).
17 For Grade 4 math teachers, the correlation between Year 1 and Year 2 alignment indices was 0.840 for control group teachers and 0.646 for FAST teachers. For Grade 5 ELA teachers, the correlation was 0.935 for control group teachers and 0.740 for FAST teachers.