A Model-Based Approach to the Disentanglement and Differential Treatment of Engaged and Disengaged Item Omissions

Abstract

Item omissions in large-scale assessments may occur for various reasons, ranging from disengagement to not being capable of solving the item and giving up. Current response-time-based classification approaches allow researchers to implement different treatments of item omissions presumably going back to different mechanisms. These approaches, however, are limited in that they require a clear-cut decision on the underlying missingness mechanism and do not allow to take the uncertainty in classification into account. We present a response-time-based model-based mixture modeling approach that overcomes this limitation. The approach (a) facilitates disentangling item omissions stemming from disengagement from those going back to solution behavior, (b) considers the uncertainty in omission classification, (c) allows for omission mechanisms to vary on the item-by-examinee level, (d) supports investigating person and item characteristics associated with different types of omission behavior, and (e) gives researchers flexibility in deciding on how to handle different types of omissions. The approach exhibits good parameter recovery under realistic research conditions. We illustrate the approach on data from the Programme for the International Assessment of Adult Competencies 2012 and compare it against previous classification approaches for item omissions.

Keywords:

• Response times
• item omissions
• mixture modeling
• test-taking engagement
• item response theory

Article information

Conflict of Interest Disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical Principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was partly supported by Grant 288472689 from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation).

Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The ideas and opinions expressed herein are those of the authors alone, and endorsement by the author’s institutions or the funding agency is not intended and should not be inferred.

Open Scholarship

This article has earned the Center for Open Science badges for Open Data. The data are openly accessible at https://osf.io/bvh4c/.

Notes

1 Researchers may incorporate an additional positive constant for modeling response times associated with item omissions, i.e. ${\beta }_j={\mu }_C+{\beta }_j^{*}+d_{\mathit{\text{ij}}}{\beta }_O,$ where ${\beta }_j^{*}\ge 0$ and ${\beta }_O\ge 0,$ allowing for omissions under solution behavior to potentially take longer than generating responses. In our analyses of empirical data, however, we found this constant to be commonly very small and, for the sake of simplicity, decided to not include it.

2 When items are not reached, examinees failed to attempt a sequence of items presented at the end of a timed test. Since examinees did not have the opportunity to engage with not-reached items, no response times are available for these items (Ulitzsch et al., 2019a).

3 Note that high variability of solution behavior tendency/engagement haven oftentimes been reported in applications of the model presented in Ulitzsch et al. (2020) and further developments thereof (Nagy & Ulitzsch, 2021; Ulitzsch, Penk, et al., 2021; Ulitzsch, Pohl, et al., 2021; Ulitzsch et al., 2022).