An Examination of the Linking Error Currently Used in PISA

ABSTRACT

Educational large-scale assessment (LSA) studies like the program for international student assessment (PISA) provide important information about trends in the performance of educational indicators in cognitive domains. The change in the country means in a cognitive domain like reading between two successive assessments is an example of a trend estimate. The uncertainty of trend estimates includes sampling and linking errors, which are regularly reported in the PISA study. This article focuses on the linking error that assesses the variability in trend estimation regarding the choice of items. Since PISA 2015, the linking error estimation method has changed. This article compares the statistical behavior and the concept of the new PISA linking error and the PISA linking error utilized until PISA 2012. It turns out that the newly proposed linking error provides no generalization of the old PISA linking error but rather reflects different aspects of uncertainties and model error.

KEYWORDS:

• Large-scale assessment
• trend estimation
• linking
• differential item functioning
• item parameter drift
• linking error
• PISA

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. If all items were used for computing the recalibrated country mean, linking errors would become very small if the number of link items is much smaller than the number of unique items at T1. The reason is that the same item parameters were used for unique items. Hence, only link items can have different item parameters in the recalibrated country mean, which, in turn, can only impact the estimated linking error.

2. Using ${\hat{\sigma }}_{\mathit{ct}}/\sqrt{\mathit{N}}$ instead of the log-likelihood based standard error results in a slight underestimate of the true standard error regarding person sampling. However, this article mainly addresses the assessment of linking error and the degree of biased standard errors vanishes in a large sample sizes N of persons per country.

3. We do not present the old PISA linking error in this simulation.