Contributions to Constructing Forced-Choice Questionnaires Using the Thurstonian IRT Model

References

• Anguiano-Carrasco, C., MacCann, C., Geiger, M., Seybert, J. M., & Roberts, R. D. (2015). Development of a forced-choice measure of typical-performance emotional intelligence. Journal of Psychoeducational Assessment, 33(1), 83–97. https://doi.org/10.1177/0734282914550387
   Web of Science ®Google Scholar
• Bäckström, M. (2007). Higher-order factors in a five-factor personality inventory and its relation to social desirability. European Journal of Psychological Assessment, 23(2), 63–70. https://doi.org/10.1027/1015-5759.23.2.63
   Web of Science ®Google Scholar
• Bäckström, M., Björklund, F., & Larsson, M. R. (2009). Five-factor inventories have a major general factor related to social desirability which can be reduced by framing items neutrally. Journal of Research in Personality, 43(3), 335–344. https://doi.org/10.1016/j.jrp.2008.12.013
   Web of Science ®Google Scholar
• Bartram, D. (2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15(3), 263–272. https://doi.org/10.1111/j.1468-2389.2007.00386.x
   Web of Science ®Google Scholar
• Billiet, J. B., & McClendon, M. J. (2000). Modeling acquiescence in measurement models for two balanced sets of items. Structural Equation Modeling: A Multidisciplinary Journal, 7(4), 608–628. https://doi.org/10.1207/S15328007SEM0704_5
   Web of Science ®Google Scholar
• Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. T., & Smith, M. A. (2006). A meta-analytic investigation of job applicant faking on personality measures. International Journal of Selection and Assessment, 14(4), 317–335. https://doi.org/10.1111/j.1468-2389.2006.00354.x
   Web of Science ®Google Scholar
• Böckenholt, U. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665–678. https://doi.org/10.1037/a0028111
   PubMed Web of Science ®Google Scholar
• Böckenholt, U. (2014). Modeling motivated misreports to sensitive survey questions. Psychometrika, 79(3), 515–537. https://doi.org/10.1007/s11336-013-9390-9
   PubMed Web of Science ®Google Scholar
• Bolt, D. M., & Johnson, T. R. (2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33(5), 335–352. https://doi.org/10.1177/0146621608329891
   Web of Science ®Google Scholar
• Bolt, D. M., & Newton, J. R. (2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71(5), 814–833. https://doi.org/10.1177/0013164410388411
   Web of Science ®Google Scholar
• Brown, A. (2016a). Item response models for forced-choice questionnaires: A common framework. Psychometrika, 81(1), 135–160. https://doi.org/10.1007/s11336-014-9434-9
   PubMed Web of Science ®Google Scholar
• Brown, A. (2016b). Thurstonian scaling of compositional questionnaire data. Multivariate Behavioral Research, 51(2–3), 345–356. https://doi.org/10.1080/00273171.2016.1150152
   PubMed Web of Science ®Google Scholar
• Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112
   Web of Science ®Google Scholar
• Brown, A., & Maydeu-Olivares, A. (2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44(4), 1135–1147. https://doi.org/10.3758/s13428-012-0217-x
   PubMed Web of Science ®Google Scholar
• Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36–52. https://doi.org/10.1037/a0030641
   PubMed Web of Science ®Google Scholar
• Brown, A., & Maydeu-Olivares, A. (2018). Modelling forced-choice response formats. In P. Irwing, T. Booth, & D. Hughes (Eds.), The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development (pp. 523–569). Wiley. https://doi.org/10.1002/9781118489772.ch18
   Google Scholar
• Bürkner, P. C. (2019). thurstonianIRT: Thurstonian IRT models in R. Journal of Open Source Software, 4(42), 1662. https://doi.org/10.21105/joss.01662
   Google Scholar
• Bürkner, P. C. (2022). On the information obtainable from comparative judgments. Psychometrika, 87(4), 1439–1472. https://doi.org/10.1007/s11336-022-09843-z
   PubMed Web of Science ®Google Scholar
• Bürkner, P. C., Schulte, N., & Holling, H. (2019). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement, 79(5), 827–854. https://doi.org/10.1177/0013164419832063
   PubMed Web of Science ®Google Scholar
• Cao, M., & Drasgow, F. (2019). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11), 1347–1368. https://doi.org/10.1037/apl0000414
   PubMed Web of Science ®Google Scholar
• Cheung, M. W. L., & Chan, W. (2002). Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 9(1), 55–77. https://doi.org/10.1207/S15328007SEM0901_4
   Web of Science ®Google Scholar
• Christiansen, N., Burns, G., & Montgomery, G. (2005). Reconsidering the use of forced-choice formats for applicant personality assessment. Human Performance, 18(3), 267–307. https://doi.org/10.1207/s15327043hup1803_4
   Web of Science ®Google Scholar
• Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24(4), 349–354. https://doi.org/10.1037/h0047358
   PubMedGoogle Scholar
• Dai, X., Yao, S., & Cai, T. (2004). Reliability and validity of the NEO-PI-R in Mainland China. Chinese Mental Health Journal, 18(3), 171–174.
   Google Scholar
• Edwards, A. L. (1957). The social desirability variable in personality assessment and research. Dryden Press.
   Google Scholar
• Edwards, L. K., & Edwards, A. L. (1991). A principal-components analysis of the Minnesota multiphasic personality inventory factor scales. Journal of Personality and Social Psychology, 60(5), 766–772. https://doi.org/10.1037/0022-3514.60.5.766
   Web of Science ®Google Scholar
• Ferrando, P. J. (2005). Factor analytic procedures for assessing social desirability in binary items. Multivariate Behavioral Research, 40(3), 331–349. https://doi.org/10.1207/s15327906mbr4003_3
   PubMed Web of Science ®Google Scholar
• Fisher, P. A., Robie, C., Christiansen, N. D., Speer, A. B., & Schneider, L. (2019). Criterion-related validity of forced-choice personality measures: A cautionary note regarding Thurstonian IRT versus classical test theory scoring. Personnel Assessment and Decisions, 5(1), 49–61. https://doi.org/10.25035/pad.2019.01.003
   Google Scholar
• Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625–641. https://doi.org/10.1080/10705510903203573
   Web of Science ®Google Scholar
• Goffin, R. D., & Christiansen, N. D. (2003). Correcting personality tests for faking: A review of popular personality tests and an initial survey of researchers. International Journal of Selection and Assessment, 11(4), 340–344. https://doi.org/10.1111/j.0965-075X.2003.00256.x
   Web of Science ®Google Scholar
• Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality Psychology in Europe (Vol. 7, pp. 7–28). Tilburg University Press.
   Google Scholar
• Griffith, R. L., Chmielowski, T., & Yoshita, Y. (2007). Do applicants fake? An examination of the frequency of applicant faking behavior. Personnel Review, 36(3), 341–355. https://doi.org/10.1108/00483480710731310
   Web of Science ®Google Scholar
• Griffith, R. L., & Converse, P. D. (2011). The rules of evidence and the prevalence of applicant faking. In M. Ziegler, C. MacCann, & R. Roberts (Eds.), New perspectives on faking in personality assessment (Vol. 1, pp. 34–52). Oxford University Press. https://doi.org/10.1093/acprof:Oso/9780195387476.003.0018
   Google Scholar
• Guenole, N., Brown, A. A., & Cooper, A. J. (2018). Forced-choice assessment of work-related maladaptive personality traits: Preliminary evidence from an application of Thurstonian item response modeling. Assessment, 25(4), 513–526. https://doi.org/10.1177/1073191116641181
   PubMed Web of Science ®Google Scholar
• Heggestad, E. D., Morrison, M., Reeve, C. L., & McCloy, R. A. (2006). Forced-choice assessments of personality for selection: Evaluating issues of normative assessment and faking resistance. Journal of Applied Psychology, 91(1), 9–24. https://doi.org/10.1037/0021-9010.91.1.9
   PubMed Web of Science ®Google Scholar
• Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
   Web of Science ®Google Scholar
• Jackson, D., Wroblewski, V., & Ashton, M. (2000). The impact of faking on employment tests: Does forced choice offer a solution? Human Performance, 13(4), 371–388. https://doi.org/10.1207/S15327043HUP1304_3
   Web of Science ®Google Scholar
• Joubert, T., Inceoglu, I., Bartram, D., Dowdeswell, K., & Lin, Y. (2015). A comparison of the psychometric properties of the forced choice and Likert scale versions of a personality instrument. International Journal of Selection and Assessment, 23(1), 92–97. https://doi.org/10.1111/ijsa.12098
   Web of Science ®Google Scholar
• Kreitchmann, R. S., Abad, F. J., & Sorrel, M. A. (2022). A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires. Behavior Research Methods, 54(3), 1476–1492. https://doi.org/10.3758/s13428-021-01677-4
   PubMed Web of Science ®Google Scholar
• Lee, P., Lee, S., & Stark, S. (2018). Examining validity evidence for multidimensional forced choice measures with different scoring approaches. Personality and Individual Differences, 123(1), 229–235. https://doi.org/10.1016/j.paid.2017.11.031
   Google Scholar
• Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 14, 1–55.
   Google Scholar
• Lin, Y. (2022). Reliability estimates for IRT-based forced-choice assessment scores. Organizational Research Methods, 25(3), 575–590. https://doi.org/10.1177/1094428121999086
   Web of Science ®Google Scholar
• MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149. https://doi.org/10.1037/1082-989X.1.2.130
   Web of Science ®Google Scholar
• Martin, B. A., Bowen, C. C., & Hunt, S. T. (2002). How effective are people at faking on personality questionnaires? Personality and Individual Differences, 32(2), 247–256. https://doi.org/10.1016/S0191-8869(01)00021-6
   Web of Science ®Google Scholar
• Maydeu-Olivares, A., & Brown, A. (2010). Item response modeling of paired comparison and ranking data. Multivariate Behavioral Research, 45(6), 935–974. https://doi.org/10.1080/00273171.2010.531231
   PubMed Web of Science ®Google Scholar
• Maydeu-Olivares, A., & Coffman, D. L. (2006). Random intercept item factor analysis. Psychological Methods, 11(4), 344–362. https://doi.org/10.1037/1082-989X.11.4.344
   PubMed Web of Science ®Google Scholar
• McCrae, R. R., & Costa, P. T. (1992). Discriminant validity of NEO-PIR facet scales. Educational and Psychological Measurement, 52(1), 229–237. https://doi.org/10.1177/001316449205200128
   Web of Science ®Google Scholar
• Meade, A. (2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77(4), 531–551. https://doi.org/10.1348/0963179042596504
   Web of Science ®Google Scholar
• Merk, J., Schlotz, W., & Falter, T. (2017). The motivational value systems questionnaire (MVSQ): Psychometric analysis using a forced choice Thurstonian IRT model. Frontiers in Psychology, 8, 1626. https://doi.org/10.3389/fpsyg.2017.01626
   PubMed Web of Science ®Google Scholar
• Moors, G. (2010). Ranking the ratings: A latent-class regression model to control for overall agreement in opinion research. International Journal of Public Opinion Research, 22(1), 93–119. https://doi.org/10.1093/ijpor/edp036
   Web of Science ®Google Scholar
• Muthén, L. K., & Muthén, B. O. (1998–2019). Mplus 8.3 [computer software].
   Google Scholar
• Neill, J. A., & Jackson, D. N. (1970). An evaluation of item selection strategies in personality scale construction. Educational and Psychological Measurement, 30(3), 647–661. https://doi.org/10.1177/001316447003000312
   Web of Science ®Google Scholar
• Ng, V., Lee, P., Ho, M. H. R., Kuykendall, L., Stark, S., & Tay, L. (2021). The development and validation of a multidimensional forced-choice format character measure: Testing the Thurstonian IRT approach. Journal of Personality Assessment, 103(2), 224–237. https://doi.org/10.1080/00223891.2020.1739056
   PubMed Web of Science ®Google Scholar
• Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46(3), 598–609. https://doi.org/10.1037/0022-3514.46.3.598
   Web of Science ®Google Scholar
• Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver & L. S. Wrightsman (Eds.), Measures of personality and social psychology attitudes (pp. 17–59). Academic Press. https://doi.org/10.1016/B978-0-12-590241-0.50006-X
   Google Scholar
• Paulhus, D. L. (2002). Socially desirable responding: The evolution of a construct. In H. I. Braun, D. N. Jackson & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (pp. 49–69). Erlbaum. https://doi.org/10.4324/9781410607454-10
   Google Scholar
• Paulhus, D. L., & Vazire, S. (2007). The self-report method. In R. W. Robins, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 224–239). Guilford.
   Google Scholar
• Pavlov, G., Maydeu-Olivares, A., & Fairchild, A. J. (2019). Effects of applicant faking on forced-choice and Likert scores. Organizational Research Methods, 22(3), 710–739. https://doi.org/10.1177/1094428117753683
   Web of Science ®Google Scholar
• Pavlov, G., Shi, D., Maydeu-Olivares, A., & Fairchild, A. (2021). Item desirability matching in forced-choice test construction. Personality and Individual Differences, 183, 111114. https://doi.org/10.1016/j.paid.2021.111114
   Web of Science ®Google Scholar
• SHL. (2013). OPQ32r technical manual version 1.0. SHL Group.
   Google Scholar
• Sun, L. (2022). Full results of the simulation studies. Figshare. Dataset. https://doi.org/10.6084/m9.figshare.19222407.v1
   Google Scholar
• Van Vaerenbergh, Y., & Thomas, T. D. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25(2), 195–217. https://doi.org/10.1093/ijpor/eds021
   Web of Science ®Google Scholar
• Vasilopoulos, N. L., Cucina, J. M., Dyomina, N. V., Morewitz, C. L., & Reilly, R. R. (2006). Forced-choice personality tests: A measure of personality and cognitive ability? Human Performance, 19(3), 175–199. https://doi.org/10.1207/s15327043hup1903_1
   Web of Science ®Google Scholar
• Viswesvaran, C., Deller, J., & Ones, D. S. (2007). Personality measures in personnel selection: Some new contributions. International Journal of Selection and Assessment, 15(3), 354–358. https://doi.org/10.1111/j.1468-2389.2007.00394.x
   Web of Science ®Google Scholar
• Walton, K. E., Cherkasova, L., & Roberts, R. D. (2020). On the validity of forced choice scores derived from the Thurstonian item response theory model. Assessment, 27(4), 706–718. https://doi.org/10.1177/1073191119843585
   PubMed Web of Science ®Google Scholar
• Wang, W. C., Qiu, X. L., Chen, C. W., Ro, S., & Jin, K. Y. (2017). Item response theory models for ipsative tests with multidimensional pairwise comparison items. Applied Psychological Measurement, 41(8), 600–613. https://doi.org/10.1177/0146621617703183
   PubMed Web of Science ®Google Scholar
• Wetzel, E., Böhnke, J. R., Brown, A. (2016). Response biases. In Leong, F. T. L., Bartram, D., Cheung, F., Geisinger, K. F., & Iliescu, D. (Eds.), ITC international handbook of testing and assessment (pp. 349–363) Oxford University Press. https://doi.org/10.1093/med:Psych/9780199356942.003.0024
   Google Scholar
• Wetzel, E., & Frick, S. (2020). Comparing the validity of trait estimates from the multidimensional forced-choice format and the rating scale format. Psychological Assessment, 32(3), 239–253. https://doi.org/10.1037/pas0000781
   PubMed Web of Science ®Google Scholar