Detecting Rater Bias in Mixed-Format Assessments

ABSTRACT

Mixed-format assessments made up of multiple-choice (MC) items and constructed response (CR) items that are scored using rater judgments include unique psychometric considerations. When these item types are combined to estimate examinee achievement, information about the psychometric quality of each component can depend on that of the other. For example, the presence of differential item functioning (DIF) in MC items could compromise the sensitivity of rater effect indicators, such as differential rater functioning (DRF) indices. Likewise, the presence of DRF could compromise the sensitivity of DIF indices. We used real data and a simulation study to consider the impact of DIF on DRF indices, and the impact of DRF on DIF indices. Results indicate some interaction between DIF and DRF that varies across data collection designs. We consider the implications of these results for research and practice.

KEYWORDS:

• mixed – format assessment
• rater bias
• differential item functioning

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. For data security purposes, we rounded the sample sizes to the nearest 10.

Additional information

Funding

The work was supported by the Spencer Foundation [202000164]