Explaining performance decline over the course of taking comprehensive proficiency tests: the roles of effort and omission propensity

References

• Adams, R. J., & Wu, M. L. (2007). The Mixed-coefficients multinomial logit model: A generalized form of the Rasch Model. In M. von Davier & C. H. Carstensen (Eds.), Multivariate and mixture distribution rasch models (pp. 57–76). Springer.
   Google Scholar
• Baumert, J., & Demmrich, A. (2001). Test motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology of Education, 16(3), 441–462. https://doi.org/10.1007/BF03173192
   Web of Science ®Google Scholar
• Becker, B., Weirich, S., Mahler, N., & Sachse, K. A. (2019). Testdesign und Auswertung des IQB-Bildungstrends 2018: Technische Grundlagen. In P. Stanat, S. Schipolowski, N. Mahler, S. Weirich, & S. Henschel (Eds.), IQB-Bildungstrend 2018. Mathematische und naturwissenschaftliche Kompetenzen am Ende der Sekundarstufe I im zweiten Ländervergleich (pp. 411–425) Waxmann.
   Google Scholar
• Borgonovi, F., & Biecek, P. (2016). An international comparison of students’ ability to endure fatigue and maintain motivation during a low-stakes test. Learning and Individual Differences, 49, 128–137. https://doi.org/10.1016/j.lindif.2016.06.001
   Web of Science ®Google Scholar
• Chang, Y.-W., Hsu, N.-J., & Tsai, R.-C. (2021). An item response tree model with not-all-distinct end nodes for non-response modelling. The British Journal of Mathematical and Statistical Psychology, 74(3), 487–512. https://doi.org/10.1111/bmsp.12236
   PubMed Web of Science ®Google Scholar
• Cintron, D. W. (2021). Methods for measuring speededness: Chronology, classification, and ensuing research and development. ETS Research Report Series, 2021(1), 1–36. https://doi.org/10.1002/ets2.12337
   Google Scholar
• Cosgrove, J., & Cartwright, F. (2014). Changes in achievement on PISA: The case of Ireland and implications for international assessment practice. Large-Scale Assessments in Education, 2(1), 2. https://doi.org/10.1186/2196-0739-2-2
   Google Scholar
• Debeer, D., Buchholz, J., Hartig, J., & Janssen, R. (2014). Student, school, and country differences in sustained test-taking effort in the 2009 PISA reading assessment. Journal of Educational and Behavioral Statistics, 39(6), 502–523. https://doi.org/10.3102/1076998614558485
   Web of Science ®Google Scholar
• Debeer, D., & Janssen, R. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164–185. https://doi.org/10.1111/jedm.12009
   Web of Science ®Google Scholar
• Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53(1), 109–132. https://doi.org/10.1146/annurev.psych.53.100901.135153
   PubMed Web of Science ®Google Scholar
• Eklöf, H. (2010). Student Motivation and Effort in the Swedish TIMSS Advanced Field Study. Paper presented at the IEA International Research Conference.
   Google Scholar
• Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Erlbaum.
   Google Scholar
• Ganzeboom, H., Graaf, P. d., Treiman, D., & Leeuw, J. d (1992). A standard international socio-economic index of occupational status. Social Science Research, 21(1), 1–56. https://doi.org/10.1016/0049-089X(92)90017-B
   Web of Science ®Google Scholar
• Gawronski, B., Morrison, M., Phills, C. E., & Galdi, S. (2017). Temporal stability of implicit and explicit measures: A longitudinal analysis. Personality & Social Psychology Bulletin, 43(3), 300–312. https://doi.org/10.1177/0146167216684131
   PubMed Web of Science ®Google Scholar
• Goldhammer, F., Martens, T., & Lüdtke, O. (2017). Conditioning factors of test-taking engagement in PIAAC: An exploratory IRT modelling approach considering person and item characteristics. Large-Scale Assessments in Education, 5(1), 18. https://doi.org/10.1186/s40536-017-0051-9
   Google Scholar
• Gonzalez, E., & Rutkowski, L. (2010). Principles of multiple matrix booklet designs and parameter recovery in large-scale assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 3, 125–156.
   Google Scholar
• Hartig, J., & Buchholz, J. (2012). A multilevel item response model for item position effects and individual persistence. Psychological Test and Assessment Modeling, 54(4), 418–431.
   Google Scholar
• Hitt, C., Trivitt, J., & Cheng, A. (2016). When you say nothing at all: The predictive power of student effort on surveys. Economics of Education Review, 52, 105–119. https://doi.org/10.1016/j.econedurev.2016.02.001
   Web of Science ®Google Scholar
• Holman, R., & Glas, C. A. W. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1–17. https://doi.org/10.1111/j.2044-8317.2005.tb00312.x
   PubMed Web of Science ®Google Scholar
• Klopack, E. T., & Wickrama, K. (2020). Modeling latent change score analysis and extensions in Mplus: A practical guide for researchers. Structural Equation Modeling, 27(1), 97–110. https://doi.org/10.1080/10705511.2018.1562929
   PubMed Web of Science ®Google Scholar
• Knott, M., Albanese, M. T., & Galbraith, J. (1991). Scoring attitudes to abortion. Journal of the Royal Statistical Society, 40(2), 217–223. https://doi.org/10.2307/2348494
   Google Scholar
• Köhler, C., Pohl, S., & Carstensen, C. H. (2015). Investigating mechanisms for missing responses in competence tests. Psychological Test and Assessment Modeling, 57(4), 499–522.
   Google Scholar
• Köhler, C., Pohl, S., & Carstensen, C. H. (2017). Dealing with item nonresponse in large-scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54(4), 397–419. https://doi.org/10.1111/jedm.12154
   Web of Science ®Google Scholar
• Leary, L. F., & Dorans, N. J. (1985). Implications for altering the context in which test items appear: A historical perspective on an immediate concern. Review of Educational Research, 55(3), 387–413. https://doi.org/10.3102/00346543055003387
   Web of Science ®Google Scholar
• Lee, Y. H., & Chen, H. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53(3), 359.
   Google Scholar
• Lord, F. M. (1983). Maximum likelihood estimation of item response parameters when some responses are omitted. Psychometrika, 48(3), 477–482. https://doi.org/10.1007/BF02293689
   Web of Science ®Google Scholar
• Martin, M. O., Mullis, I. V. S., & Hooper, M. (Eds.). (2017). Methods and Procedures in PIRLS 2016. International Association for the Evaluation of Educational Achievement. https://eric.ed.gov/?id=ED580352.
   Google Scholar
• McArdle, J. J., & Nesselroade, J. R. (2014). Longitudinal data analysis using structural equation models (S. xi, 426). American Psychological Association. https://doi.org/10.1037/14440-000
   Google Scholar
• Messick, S. (1984). The psychology of educational measurement. ETS Research Report Series, 1984(1), i–55. https://doi.org/10.1002/j.2330-8516.1984.tb00046.x
   Google Scholar
• Mislevy, R. J., Beaton, A. E., Kaplan, B., & Sheehan, K. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29(2), 133–161. https://doi.org/10.1111/j.1745-3984.1992.tb00371.x
   Web of Science ®Google Scholar
• Mullis, I. V. S., & Martin, M. O. (Eds.). (2013). TIMSS 2015 Assessment Frameworks. TIMSS & PIRLS International Study Center. http://timssandpirls.bc.edu/timss2015/frameworks.html.
   Google Scholar
• Muthén, L. K., & Muthén, B. O. (2017). Mplus (Version 8). [Computer software].
   Google Scholar
• Nagy, G., Lüdtke, O., & Köller, O. (2016). Modeling test context effects in longitudinal achievement data: Examining position effects in the longitudinal German PISA 2012 assessment. Psychological Test and Assessment Modeling, 58(4), 641–670.
   Google Scholar
• Nagy, G., Nagengast, B., Frey, A., Becker, M., & Rose, N. (2019). A multilevel study of position effects in PISA achievement tests: Student- and school-level predictors in the German tracked school system. Assessment in Education, 26(4), 422–443. https://doi.org/10.1080/0969594X.2018.1449100
   Google Scholar
• Nagy, G., Ulitzsch, E., & Lindner, M. A. (2023). The role of rapid guessing and test-taking persistence in modelling test-taking engagement. Journal of Computer Assisted Learning, 39(3), 751–766. https://doi.org/10.1111/jcal.12719
   Web of Science ®Google Scholar
• O’Muircheartaigh, C., & Moustaki, I. (1999). Symmetric pattern models: A latent variable approach to item non-response in attitude scales. Journal of the Royal Statistical Society Series A, 162(2), 177–194. https://doi.org/10.1111/1467-985X.00129
   Web of Science ®Google Scholar
• OECD. (2017). PISA 2015 Technical Report. OECD Publishing. http://www.oecd.org/pisa/data/2015-technical-report.
   Google Scholar
• Pant, H. A., Stanat, P., Hecht, M., Heitmann, P., Jansen, M., Lenski, A. E., Penk, C., Pöhlmann, C., Roppelt, A., Schroeders, U., & Siegle, T. (2015). IQB-Ländervergleich Mathematik und Naturwissenschaften 2012 (IQB-LV 2012) (Version 4) [Datensatz]. IQB – Institut zur Qualitätsentwicklung im Bildungswesen. https://doi.org/10.5159/IQB_LV_2012_v4
   Google Scholar
• Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review, 18(4), 315–341. https://doi.org/10.1007/s10648-006-9029-9
   Web of Science ®Google Scholar
• Penk, C., Pöhlmann, C., & Roppelt, A. (2014). The role of test-taking motivation for students’ performance in low-stakes assessments: An investigation of school-track-specific differences. Large-Scale Assessments in Education, 2(1), 2–17. https://doi.org/10.1186/s40536-014-0005-4
   Google Scholar
• Penk, C., & Richter, D. (2017). Change in test-taking motivation and its relationship to test performance in low-stakes assessments. Educational Assessment, Evaluation and Accountability, 29(1), 55–79. https://doi.org/10.1007/s11092-016-9248-7
   Web of Science ®Google Scholar
• Pokropek, A. (2011). Missing by design: Planned missing-data designs in social science. ASK. Research & Methods, 20, 81–105.
   Google Scholar
• Qian, J. (2014). An investigation of position effects in large-scale writing assessments. Applied Psychological Measurement, 38(7), 518–534. https://doi.org/10.1177/0146621614534312
   Web of Science ®Google Scholar
• R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/.
   Google Scholar
• Rios, J. (2021). Improving test-taking effort in low-stakes group-based educational testing: A meta-analysis of interventions. Applied Measurement in Education, 34(2), 85–106. https://doi.org/10.1080/08957347.2021.1890741
   Web of Science ®Google Scholar
• Robitzsch, A., & Rupp, A. A. (2009). Impact of missing data on the detection of differential item functioning: The case of Mantel-Haenszel and logistic regression analysis. Educational and Psychological Measurement, 69(1), 18–34. https://doi.org/10.1177/0013164408318756
   Web of Science ®Google Scholar
• Sachse, K. A., Mahler, N., & Pohl, S. (2019). When nonresponse mechanisms change: Effects on trends and group comparisons in large-scale assessments. Educational and Psychological Measurement, 79(4), 699–726. https://doi.org/10.1177/0013164419829196
   PubMed Web of Science ®Google Scholar
• Sachse, K. A., Weirich, S., Becker, B., Hafiz, N. J., Schneider, R., & Schipolowski, S. (2022). Methodische Grundlagen: Anlage, Durchführung und Auswertung des IQB-Bildungstrends 2021. In P. Stanat, S. Schipolowski, R. Schneider, K. A. Sachse, S. Weirich, & S. Henschel (Eds.), IQB-Bildungstrend 2021. Kompetenzen in den Fächern Deutsch und Mathematik am Ende der 4. Jahrgangsstufe im dritten Ländervergleich (pp. 233–258) Waxmann.
   Google Scholar
• Sundre, D. L., & Kitsantas, A. (2004). An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance? Contemporary Educational Psychology, 29(1), 6–26. https://doi.org/10.1016/S0361-476X(02)00063-2
   Web of Science ®Google Scholar
• Ulitzsch, E., Penk, C., Davier, M. v., & Pohl, S. (2021). Model meets reality: Validating a new behavioral measure for test-taking effort. Educational Assessment, 26(2), 104–124. https://doi.org/10.1080/10627197.2020.1858786
   Web of Science ®Google Scholar
• van Buuren, S. (2018). Flexible imputation of missing data (Second edition). Chapman & Hall/CRC.
   Google Scholar
• van Buuren, S., & Groothuis-Oudshoorn, K. (2011). MICE: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1–68. https://doi.org/10.18637/jss.v045.i03
   Web of Science ®Google Scholar
• von Davier, M., Gonzalez, E., & Mislevy, R. (2009). What are plausible values and why are they useful. IERI Monograph Series, 2, 9–36.
   Google Scholar
• Weirich, S., Hecht, M., & Böhme, K. (2014). Modeling item position effects using generalized linear mixed models. Applied Psychological Measurement, 38(7), 535–548. https://doi.org/10.1177/0146621614534955
   Web of Science ®Google Scholar
• Weirich, S., Hecht, M., Penk, C., Roppelt, A., & Böhme, K. (2017). Item position effects are moderated by changes in test-taking effort. Applied Psychological Measurement, 41(2), 115–129. https://doi.org/10.1177/0146621616676791
   PubMed Web of Science ®Google Scholar
• Wise, S. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19(2), 95–114. https://doi.org/10.1207/s15324818ame1902_2
   Web of Science ®Google Scholar
• Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38. https://doi.org/10.1111/j.1745-3984.2006.00002.x
   Web of Science ®Google Scholar
• Wise, S. L., Soland, J., & Bo, Y. (2020). The (Non)impact of differential test taker engagement on aggregated scores. International Journal of Testing, 20(1), 57–77. https://doi.org/10.1080/15305058.2019.1605999
   Web of Science ®Google Scholar
• Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2
   Web of Science ®Google Scholar
• Wise, S. L., & Kuhfeld, M. R. (2021). Using retest data to evaluate and improve effort-moderated scoring. Journal of Educational Measurement, 58(1), 130–149. https://doi.org/10.1111/jedm.12275
   Web of Science ®Google Scholar
• Wise, S. L., & Smith, L. F. (2011). A model of examinee test-taking effort. In High-stakes testing in education: Science and practice in K–12 settings (pp. 139–153). American Psychological Association. https://doi.org/10.1037/12330-009
   Google Scholar
• Wise, S. L., & Smith, L. F. (2016). The validity of assessment when students don’t give good effort. In Handbook of human and social conditions in assessment. Routledge.
   Google Scholar
• Wu, M. L., Adams, R. J., Wilson, M. R., & Haldane, S. A. (2007). ACER ConQuest Version 2: Generalised item response modelling software [computer program]. Australian Council for Educational Research.
   Google Scholar