Prevalence of random responders as a function of scale position and questionnaire length in the TIMSS 2015 eighth-grade student questionnaire

Figures & data

Table 1. Overview of scales in the TIMSS 2015 student questionnaire.

			Response	Position
Scale		Items	Options	ISP	SSP
Domain: School climate
Students’ sense of school belonging		7	1 (agree a lot) − 4 (disagree a lot)	0	0
Domain: School safety
Student bullying		9	1 (at least once a week) − 4 (never)	1	1
Domain: Student engagement and attitudes
Students like learning mathematics		9	1 (agree a lot) − 4 (disagree a lot)	2	2
Students’ views on engaging teaching in mathematics lessons		10	1 (agree a lot) − 4 (disagree a lot)	3	3
Students confident in mathematics		9	1 (agree a lot) − 4 (disagree a lot)	4	4
Students value mathematics		9	1 (agree a lot) − 4 (disagree a lot)	5	5
Students like learning science${}^{*}$		9	1 (agree a lot) − 4 (disagree a lot)	6	{6, 9, 12, 15}
Students’ views on engaging teaching in science lessons${}^{*}$		10	1 (agree a lot) − 4 (disagree a lot)	7	{7, 10, 13, 16}
Students confident in science${}^{*}$		8	1 (agree a lot) − 4 (disagree a lot)	8	{8, 11, 14, 17}
Students value science		9	1 (agree a lot) − 4 (disagree a lot)	9	18

Note. ${}^{*}$For this scale, there is a distinction between countries with an integrated or a separated science program, respectively referring to one general scale related to science as a single subject or to four separate scales related to each of the specific science subjects. In the questionnaire for countries with a separated science program, these scales are grouped per domain and appear in the following order: biology, earth science, chemistry, and physics. In the corresponding statements “science” is then replaced by the specific subject name. ISP = integrated science program; SSP = separated science program. Note that the first scale at position 0 represents the first substantive scale after 14 more general background questions.

Figure 1. Mixture IRT model Framework to Define and Operationalize Random Responders in terms of Independence and Uniformity of Item Responses.Note. Symbols follow standard path diagram conventions, with squares representing observed variables (i.e., item responses); circles, latent variables (i.e., a trait to be measured by the scale of items); arrows indicating dependence relations; vertical lines, response category thresholds. Reprinted under the terms of CC-BY-NC from “Random responders in the TIMSS 2015 student questionnaire: A threat to validity?” by S. van Laar & Braeken, 2022, Journal of Educational Measurement.

Figure 2. Differences in Prevalence of Random Responders across Countries.Note. The vertical gray line represents the prevalence of random responders for an average country on an average scale under the baseline model ${\mathcal{M}}_0$ (${\widehat{\beta }}_0=8.9\%)$. The black horizontal lines are 95% confidence intervals of the country-specific deviations in prevalence (ΔPREV(RR)) to that average.

Table 2. Overview of the number of excluded scales: Data collection procedures and/or lack of scale quality.

	Integrated Science								Separated Science
	scales			excluded					scales			excluded
ISO	tot	$N_{\text{eff}}$	%	E	α^z	$(E,{\alpha }^z)$	NA	ISO	tot	$N_{\text{eff}}$	%	E	α^z	$(E,{\alpha }^z)$	NA
BHR	10	8	80%	1	1			ARM	19	18	95%	1
BWA	10	7	70%	3				GEO	19	18	95%		1
EGY	10	8	80%		1	1		LBN	19	9	47%	1	6		3
ISR	10	9	90%				1	MAR	19	12	63%	1	6
JOR	10	8	80%		2			SWE	19	16	84%				3
KWT	10	9	90%	1
MYS	10	9	90%		1
OMN	10	8	80%		2
SAU	10	8	80%			2
ZAF	10	7	70%	2	1
THA	10	9	90%		1

Note. tot = total number of scales expected for a country within the corresponding science program; $N_{\text{eff}}$ = the number of included scales for a country; % = percentage of included scales for a country. Quality criteria for the measurement model for the regular responders in the mixture: E = number of scales excluded due to the classification entropy being below .70; α^z = number of scales excluded due to two or more standardized item discrimination parameters being below .40; $(E,{\alpha }^z)$ = number of scales excluded due to the quality criteria for standardized item discrimination parameters and classification entropy not being met; NA = number of scales excluded due to the scale not being administered or the data not being available for public use. For the integrated science program, all scales were administered as normal and the quality criteria were met for the following 18 countries: AUS, CAN, CHL, TWN, ENG, HGK, IRN, IRL, ITA, JPN, KOR, NZL, NOR, QAT, SGP, TUR, ARE, and USA. For the separated science program, this was the case for the following 6 countries: HUN, KAZ, LTU, MLT, RUS, and SVN.

Table 3. Cross-classified mixed models of the prevalence of random responders as a function of scale position and questionnaire length.

	${\mathcal{M}}_0$		${\mathcal{M}}_1$		${\mathcal{M}}_2$		${\mathcal{M}}_3$		${\mathcal{M}}_4$
	Parameter Estimates
predictor	$\widehat{\beta }$	SE	$\widehat{\beta }$	SE	$\widehat{\beta }$	SE	$\widehat{\beta }$	SE	$\widehat{\beta }$	SE
β0: intercept	8.90	1.14	7.92	1.39	8.88	1.18	7.93	1.39	6.54	2.00
β1: position			0.12	0.11			0.13	0.11	0.39	0.30
β2: length					0.06	0.81	−0.12	0.82	0.33	0.96
β3: position × length									−0.17	0.19
	Variance Components
${\sigma }_{\mathit{\text{country}}}^2$	3.8		3.9		4.0		4.0		4.0
${\sigma }_{\mathit{\text{scale}}}^2$	25.5		22.9		25.5		22.9		21.2
${\sigma }_{\mathit{\text{residual}}}^2$	8.5		8.5		8.5		8.5		8.6
${\mathrm{R}}_{\mathit{\text{scale}}}^2$			10.1%		0.0%		10.2%		16.9%
npar	4		5		5		6		7
−2LL	2428.2		2426.9		2428.2		2426.9		2426.1

Note. npar = number of parameters; -2LL = deviance; ${\mathrm{R}}_{\mathit{\text{scale}}}^2$ = percentage of systematic variation in the prevalence of random responders across scales under ${\mathcal{M}}_0$ that can be attributed to differences in the predictor(s) in the corresponding model (${\mathcal{M}}_1$ - ${\mathcal{M}}_4$). Located at position zero is the first substantive scale that followed after 14 more general background questions. Length is a binary variable differentiating between the shorter student questionnaire with 10 survey scales (i.e., length = 0) and the longer student questionnaire with 19 survey scales (i.e., length = 1).

Figure 3. Observed Prevalence of Random Responders per Country across Scales by Science Program .Note. The solid black line represents the observed prevalence of random responders across scales. The dashed grey lines represent the average prevalence on the first and the last scale across countries in the corresponding questionnaire. The spikes in prevalence are related to the confidence scales; within the integrated science questionnaire located at position 4 and 8 and within the separated science questionnaire located at position 4, 8, 11, 14, and 17 (i.e., indicated by the grey vertical bars).

Table 4. Revisited with confidence in mind: Cross-classified mixed models of the prevalence of random responders as a function of scale position and questionnaire length.

	${\mathcal{M}}_{0c}$		${\mathcal{M}}_{1c}$		${\mathcal{M}}_{2c}$		${\mathcal{M}}_{3c}$		${\mathcal{M}}_{4c}$
	Parameter estimates
Predictor	$\widehat{\beta }$	SE	$\widehat{\beta }$	SE	$\widehat{\beta }$	SE	$\widehat{\beta }$	SE	$\widehat{\beta }$	SE
β0: intercept	6.14	0.54	4.57	0.51	6.22	0.58	4.70	0.52	3.53	0.61
${\beta }_{\mathit{\text{confidence}}}$	9.89${}^{\ast }$	1.39	9.23${}^{\ast }$	1.20	9.88${}^{\ast }$	1.39	9.17${}^{\ast }$	1.19	8.79${}^{\ast }$	1.17
β1: position			0.22${}^{\ast }$	0.05			0.23${}^{\ast }$	0.05	0.50${}^{\ast }$	0.10
β2: length					−0.25	0.57	−0.67	0.57	−0.04	0.62
β3: position × length									−0.22${}^{\ast }$	0.08
	Variance components
${\sigma }_{\mathit{\text{country}}}^2$	2.1		2.1		2.1		2.1		2.1
${\sigma }_{\mathit{\text{confidence}}}^2$	30.9		31.3		30.9		31.3		31.7
ρ	0.26		0.24		0.27		0.27		0.27
${\sigma }_{\mathit{\text{scale}}}^2$	3.6		1.3		3.6		1.3		0.9
${\sigma }_{\mathit{\text{residual}}}^2$	3.1		3.1		3.1		3.1		3.1
$\Delta {\mathrm{R}}_{\mathit{\text{scale}}}^2$			8.7%		0.0%		9.0%		10.3%
npar	7		8		8		9		10
−2LL	2046.7		2035.4		2046.5		2034.0		2025.9

Note. npar = number of parameters; -2LL = deviance; ρ = correlation between country-varying coefficient for confidence and the country-varying intercept; $\Delta {\mathrm{R}}_{\mathit{\text{scale}}}^2$ = percentage of reduction in ${\sigma }_{\mathit{\text{scale}}}^2$ under ${\mathcal{M}}_0$ uniquely attributed to the difference in the predictor(s) in the model beyond confidence. Calculated as the difference between the reduction attributed to the combined effect of confidence and the predictor(s) in the corresponding model (${\mathcal{M}}_{1c}$ - ${\mathcal{M}}_{4c}$) and the reduction attributed to confidence on its own in ${\mathcal{M}}_{0c}$; ${}^{\ast }=p<.05$. Confidence is a binary variable differentiating between non-confidence (i.e., confidence = 0) and confidence scales (i.e., confidence = 1). Located at position zero is the first substantive scale that followed after 14 more general background questions. Length is a binary variable differentiating between the shorter student questionnaire with 10 survey scales (i.e., length = 0) and the longer student questionnaire with 19 survey scales (i.e., length = 1).

Figure 4. Prevalence of random responders as a function of scale position and questionnaire length in TIMSS 2015 under the cross-classified mixed model ${\mathcal{M}}_{4c}$.Note. The TIMSS 2015 student questionnaire consisted of 10 survey scales for countries with an integrated science program, whereas it consisted of 19 survey scales for countries with a separated science program. For the model parameters of model ${\mathcal{M}}_{4c}$, see Table 4.

van Laar, S., & Braeken, J. (2022). Random responders in the TIMSS 2015 student questionnaire: A threat to validity? Journal of Educational Measurement, 59(4), 470–501. https://doi.org/10.1111/jedm.12317

 Web of Science ®Google Scholar