Moving beyond Likert and Traditional Forced-Choice Scales: A Comprehensive Investigation of the Graded Forced-Choice Format

Abstract

The graded forced-choice (FC) format has recently emerged as an alternative that may preserve the advantages and overcome the issues of the dichotomous FC measures. The current study presented the first large-scale evaluation of the performance of three types of FC measures (FC2, FC4 and FC5 with 2, 4 and 5 response options, respectively) and compared their performance to their Likert (LK) counterparts (LK2, LK4, and LK5) on (1) psychometric properties, (2) respondent reactions, and (3) susceptibility to response styles. Results showed that, compared to LK measures with the same number of response options, the three FC scales provided better support for the hypothesized factor structure, were perceived as more faking-resistant and cognitive demanding, and were less susceptible to response styles. FC4/5 and LK4/5 demonstrated similarly good reliability, while LK2 provided more reliable scores than FC2. When compared across the three FC measures, FC4 and FC5 displayed comparable psychometric performance and respondent reactions. FC4 exhibited a moderate presence of extreme response style, while FC5 had a weak presence of both extreme and middle response styles. Based on these findings, the study recommends the use of graded FC over dichotomous FC and LK, particularly FC5 when extreme response style is a concern.

Keywords:

• Graded forced-choice scale
• Validity
• Respondent reactions
• Response styles

Acknowledgments

The authors would like to thank Professor Fritz Drasgow for his comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors’ institutions or the National Key R&D Program of China is not intended and should not be inferred.

Notes

1 Note that the length of a FC measure is less clear-cut than that of a LK measure. Some researchers count the number of blocks as the length of a FC measure while others use the number of pairs as scale length. In the present study, we used the number of pairs as scale length because pairs (pseudo items) are the indicators of latent variables in the Thurstonian Item Response Theory (TIRT; Brown & Maydue-Olivares, 2011) model. Accordingly, the length of a FC measure with 20 triplets has a length of 60 instead of 20. As reliability is positively related to scale length, it is also important to keep scale length constant when comparing the reliability of FC and LK scales.

2 By decomposing each triplet in an existing FC measure into three pairs, we ensured that the FC and LK measures administered in the current study have equal length and identical statement content. However, it is not necessary to construct pairs by decomposing bigger blocks. Instead, an equal number of pairs can be formed directly by allowing the same statements to appear in two different pairs to achieve equal length and identical statements across the FC and LK formats.

3 We fitted the same model using Mplus 8.5 and estimates from Mplus were the same as estimates from lavan.