Improving the Precision of Classroom Observation Scores Using a Multi-Rater and Multi-Timepoint Item Response Theory Model

ABSTRACT

Classroom observational protocols, in which raters observe and score the quality of teachers’ instructional practices, are often used to evaluate teachers for consequential purposes despite evidence that scores from such protocols are frequently driven by factors, such as rater and temporal effects, that have little to do with teacher quality. In this study, we present an item response theory model that improves precision of observation scores by accounting for multiple sources of measurement error. Unlike the sum-scoring approach typically used in practice, the model presented here identifies raters and protocol items that are more reliable and should therefore be weighted more in scores. We then show that teachers identified as substandard using our model versus the sum-scoring approach are oftentimes very different. Our results suggest that accountability systems could improve observation scores by increasing the scoring weights of items and raters that provide greater information.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/10627197.2024.2350966

Notes

¹ In some cases, observations occurred via video such that observations and lessons did not occur simultaneously.

² Throughout the paper, we refer to construct-relevant variance as variance attributable to the teacher. We adopt this terminology in order to distinguish (a) true time-invariant trait-level variability between teachers from (b) variability due to idiosyncratic sources, such as raters, occasions, and lessons. However, we acknowledge that some sources of error variance may be inextricably linked to differences within teachers. For example, variance due to lessons implies that teachers’ instructional quality can vary from lesson to lesson. While such variability is indeed attributable to the teacher, it does not reflect stable trait-level variance relevant to the construct of interest.