Examining the Question-Behavior Effect in Low-Stakes Testing Contexts: A Cheap Strategy to Increase Examinee Effort

References

• Abdelfattah, F. (2010). The relationship between motivation and achievement in low-stakes examinations. Social Behavior and Personality, 38(2), 159–167. doi:10.2224/sbp.2010.38.2.159
   Web of Science ®Google Scholar
• Barry, C., & Finney, S. J. (2016). Modeling change in effort across a low-stakes testing session: A latent growth curve modeling approach. Applied Measurement in Education, 29(1), 46–64. doi:10.1080/08957347.2015.1102914
   Web of Science ®Google Scholar
• Braun, H., Kirsch, I., & Yamamoto, K. (2011). An experimental study of the effects of monetary incentives on performance on the 12th-Grade NAEP reading assessment. Teachers College Record, 113(11), 2309–2344. doi:10.1177/016146811111301101
   Web of Science ®Google Scholar
• Brown, A. R., & Finney, S. J. (2011). Low-stakes testing and psychological reactance: Using the Hong psychological reactance scale to better understand compliant and non-compliant examinees. International Journal of Testing, 11(3), 248–270. doi:10.1080/15305058.2011.570884
   Google Scholar
• Bryan, C. J., Walton, G. M., Rogers, T., & Dweck, C. S. (2011). Motivating voter turnout by invoking the self. Proceedings of the National Academy of Sciences, 108(31), 12653–12656. doi:10.1073/pnas.1103343108
   PubMed Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (Second ed.). Hillsdale, New Jersey: Lawrence Erlbaum.
   Google Scholar
• Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, New Jersey: Erlbaum.
   Google Scholar
• Cole, J. S., & Osterlind, S. J. (2008). Investigating differences between low- and high-stakes test performance on a general education exam. The Journal of General Education, 57(2), 119–130. doi:10.2307/27798099
   Google Scholar
• Council for Higher Education Accreditation (2022, March 3). Accreditation & recognition. https://www.chea.org/about-accreditation
   Google Scholar
• Debeer, D., Buchholz, J., Hartig, J., & Janssen, R. (2014). Student, school, and country differences in sustained test-taking effort in the 2009 PISA reading assessment. Journal of Educational and Behavioral Statistics, 39(6), 502–523. doi:10.3102/1076998614558485
   Web of Science ®Google Scholar
• DeMars, C. E., Baskkov, B. M., & Socha, A. B. (2013). The role of gender in test-taking motivation under low-stakes conditions. Research & Practice in Assessment, 8, 69–82. https://www.rpajournal.com/dev/wp-content/uploads/2013/11/A4.pdf
   Google Scholar
• Eklöf, H., Pavešič, B. J., & Grønmo, L. S. (2014). A cross-national comparison of reported effort and mathematics performance in TIMSS advanced. Applied Measurement in Education, 27(1), 31–45. doi:10.1080/08957347.2013.853070
   Web of Science ®Google Scholar
• Finney, S. J., Mathers, C. E., & Myers, A. J. (2016). Investigating the dimensionality of examinee motivation across instruction conditions in low-stakes testing contexts. Research & Practice in Assessment, 11, 5–17. https://files.eric.ed.gov/fulltext/EJ1137932.pdf
   Google Scholar
• Finney, S. J., Myers, A. J., & Mathers, C. E. (2018). Test instructions do not moderate the indirect effect of perceived test importance on test performance in low-stakes testing contexts. International Journal of Testing, 18(4), 297–322. doi:10.1080/15305058.2017.1396466
   Web of Science ®Google Scholar
• Finney, S. J., Pastor, D. A., & Silver, S. (2023). Students’ understanding of assessment for institutional accountability and improvement: Relation with test-taking effort and remote test administration. Research & Practice in Assessment, 18(1), 5–18. https://www.rpajournal.com/dev/wp-content/uploads/2023/03/Test-Taking-Effort-and-Remote-Test-Administration-RPA.pdf
   Google Scholar
• Finney, S. J., Satkus, P., & Perkins, B. A. (2020). The effect of perceived test importance and examinee emotions on expended effort during a low-stakes test: A longitudinal panel model. Educational Assessment, 25(2), 159–177. doi:10.1080/10627197.2020.1756254
   Web of Science ®Google Scholar
• Finney, S. J., Sundre, D. L., Swain, M. S., & Williams, L. M. (2016). The validity of value-added estimates from low-stakes testing contexts: The impact of change in test-taking motivation and test consequences. Educational Assessment, 21(1), 60–87. doi:10.1080/10627197.2015.1127753
   Web of Science ®Google Scholar
• Hawthorne, K. A., Bol, L., Pribesh, S., & Suh, Y. (2015). Effects of motivational prompts on motivation, effort, and performance on a low-stakes standardized test. Research & Practice in Assessment, 10, 30–39. https://www.rpajournal.com/dev/wp-content/uploads/2015/06/A3.pdf
   Google Scholar
• Hopfenbeck, T. N., & Kjaernsli, M. (2016). Students’ test motivation in PISA: The case of norway. The Curriculum Journal, 27(3), 406–422. doi:10.1080/09585176.2016.1156004
   Web of Science ®Google Scholar
• Knekta, E. (2016). Are all pupils equally motivated to do their best on all tests? Differences in reported test-taking motivation within and between tests with different stakes. Scandinavian Journal of Educational Research, 61(1), 95–111. doi:10.1080/00313831.2015.1119723
   Web of Science ®Google Scholar
• Kopp, J. P., & Finney, S. J. (2013). Linking academic entitlement and student incivility using latent means modeling. The Journal of Experimental Education, 81(3), 322–336. doi:10.1080/00220973.2012.727887
   Web of Science ®Google Scholar
• Levav, J., & Fitzsimons, G. J. (2006). When questions change behavior: The role of ease of representation. Psychological Science, 17(3), 207–213. doi:10.1111/j.1467-9280.2006.01687.x
   PubMed Web of Science ®Google Scholar
• Liu, O. L. (2017). Ten Years after the spellings commission: From accountability to internal improvement. Educational Measurement: Issues & Practice, 36(2), 34–41. doi:10.1111/emip.12139
   Google Scholar
• Liu, O. L., Bridgeman, B., & Adler, R. (2012). Measuring learning outcomes in higher education: Motivation matters. Educational Researcher, 41(9), 352–362. doi:10.3102/0013189X12459679
   Web of Science ®Google Scholar
• Liu, O. L., Rios, J. A., & Borden, V. (2015). The effects of motivational instruction on college students’ performance on low-stakes assessment. Educational Assessment, 20(2), 79–94. doi:10.1080/08957347.2021.1890741
   Web of Science ®Google Scholar
• Manstead, A., & van Eekelen, S. (1998). Distinguishing between perceived behavioral control and self-efficacy in the domain of academic achievement intentions and behaviors. Journal of Applied Social Psychology, 28(15), 1375–1392. doi:10.1111/j.1559-1816.1998.tb01682.x
   Web of Science ®Google Scholar
• Myers, A. J., & Finney, S. J. (2021a). Change in self-reported motivation before to after test completion: Relation with performance. The Journal of Experimental Education, 89(1), 74–94. doi:10.1080/00220973.2019.1680942
   Web of Science ®Google Scholar
• Myers, A. J., & Finney, S. J. (2021b). Does it matter if examinee motivation is measured before or after a low-stakes test? A moderated mediation analysis. Educational Assessment, 26(1), 1–19. doi:10.1080/10627197.2019.1645591
   Web of Science ®Google Scholar
• O’Neil, H., Sugrue, B., & Baker, E. L. (1995). Effects of motivational interventions on the national assessment of educational progress mathematics performance. Educational Assessment, 3(2), 135–157. doi:10.1207/s15326977ea0302_2
   Google Scholar
• Pastor, D. A., Foelber, K. J., Jacovidis, J. N., Fulcher, K. H., Sauder, D. C., & Love, P. D. (2019). University-wide assessment days: The James Madison University model. The Association for Institutional Research (AIR) Professional File, Article, 144(Spring 2019), 1–13. doi:10.34315/apf1442019
   Google Scholar
• Pastor, D. A., Patterson, C. R., & Finney, S. J. (2023). Development and internal validity of the student opinion scale: A measure of test-taking motivation. Journal of Psychoeducational Assessment, 41(2), 209–225. doi:10.1177/07342829221140957
   Web of Science ®Google Scholar
• Penk, C., & Richter, D. (2017). Change in test-taking motivation and its relationship to test performance in low-stakes assessments. Educational Assessment, Evaluation and Accountability, 29(1), 55–79. doi:10.1007/s11092-016-9248-7
   Web of Science ®Google Scholar
• Rios, J. (2021). Improving test-taking effort in low-stakes group-based educational testing: A meta-analysis of interventions. Applied Measurement in Education, 34(2), 85–106. doi:10.1080/08957347.2021.1890741
   Web of Science ®Google Scholar
• Rios, J., & Deng, J. (2021). Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis. Large-Scale Assessments in Education, 9(1), 18. doi:10.1186/s40536-021-00110-8
   Web of Science ®Google Scholar
• Rios, J., Guo, H., Mao, L., & Liu, O. L. (2017). Evaluating the impact of careless responses on aggregated scores: To filter unmotivated examinees or not? International Journal of Testing, 17(1), 74–104. doi:10.1080/15305058.2016.1231193
   Web of Science ®Google Scholar
• Rios, J., Liu, O. L., & Bridgeman, B. (2014). Identifying unmotivated examinees on student learning outcomes assessment: A comparison of two approaches. New Directions for Institutional Research, 2014(161), 69–82. doi:10.1002/ir.20068
   Google Scholar
• Roohr, K. C., Liu, H., & Liu, O. L. (2016). Investigating student learning gains in college: A longitudinal study. Studies in Higher Education, 42(12), 2284–2300. doi:10.1080/03075079.2016.1143925
   Web of Science ®Google Scholar
• Satkus, P., & Finney, S. J. (2021). Antecedents of examinee motivation during low-stakes tests: Examining the variability in effects across different research designs. Assessment and Evaluation in Higher Education, 46(7), 1065–1079. doi:10.1080/02602938.2020.1846680
   Web of Science ®Google Scholar
• Smith, L. F., & Smith, J. K. (2004). The influence of test consequences on national examinations. North American Journal of Psychology, 6(1), 13–25.
   Google Scholar
• Soland, J. (2018). The achievement gap or the engagement gap? Investigating the sensitivity of gaps estimates to test motivation. Applied Measurement in Education, 31(4), 312–323. doi:10.1080/08957347.2018.1495213
   Web of Science ®Google Scholar
• Soland, J., Kuhfeld, M., & Rios, J. (2021). Comparing different response time threshold setting methods to detect low effort on a large-scale assessment. Large-Scale Assessments in Education, 9(1), 8. doi:10.1186/s40536-021-00100-w
   Web of Science ®Google Scholar
• Spangenberg, E. R., Kareklas, I., Devezer, B., & Sprott, D. E. (2016). A meta-analytic synthesis of the question-behavior effect. Journal of Consumer Psychology, 26(3), 441–458. doi:10.1016/j.jcps.2015.12.004
   Web of Science ®Google Scholar
• Swerdzewski, P. J., Harmes, J. C., & Finney, S. J. (2009). Skipping the test: Using empirical evidence to inform policy related to students who avoid taking low-stakes assessment in college. Journal of General Education, 58(3), 167–195. doi:10.2307/27798137
   Google Scholar
• Swerdzewski, P. J., Harmes, J. C., & Finney, S. J. (2011). Two approaches for identifying low-motivated students in a low-stakes assessment context. Applied Measurement in Education, 24(2), 162–188. doi:10.1080/08957347.2011.555217
   Web of Science ®Google Scholar
• Thelk, A. D., Sundre, D. L., Horst, S. J., & Finney, S. J. (2009). Motivation matters: Using the student opinion scale to make valid inferences about student performance. The Journal of General Education, 58(3), 129–151. doi:10.2307/27798135
   Google Scholar
• U.S. Department of Education. (2006). A test of leadership: Charting the future of American higher education (report of the commission appointed by secretary of education Margaret spellings).
   Google Scholar
• Walton, G. M. (2014). The new science of wise psychological interventions. Current Directions in Psychological Science, 23(1), 73–82. doi:10.1177/0963721413512856
   Web of Science ®Google Scholar
• Walton, G. M., & Wilson, T. D. (2018). Wise interventions: Psychological remedies for social and personal problems. Psychological Review, 125(5), 617–655. doi:10.1037/rev0000115
   PubMed Web of Science ®Google Scholar
• Wilding, S., Conner, M., Sandberg, T., Prestwich, A., Lawton, R. … Sheeran, P. (2016). The question-behaviour effect: A theoretical and methodological review and meta-analysis. European Review of Social Psychology, 27(1), 196–230. doi:10.1080/10463283.2016.1245940
   Web of Science ®Google Scholar
• Wise, S. L. (2020). Six insights regarding test-taking disengagement. Educational Research & Evaluation, 26(5–6), 328–338. doi:10.1080/13803611.2021.1963942
   Web of Science ®Google Scholar
• Wise, S. L., & DeMars, C. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. doi:10.1207/s15326977ea1001_1
   Google Scholar
• Wise, S. L., & DeMars, C. E. (2010). Examinee non-effort and the validity of program assessment results. Educational Assessment, 15(1), 27–41. doi:10.1080/10627191003673216
   Google Scholar
• Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. doi:10.1207/s15324818ame1802_2
   Web of Science ®Google Scholar
• Wise, S. L., & Kuhfeld, M. R. (2021). Using retest data to evaluate and improve effort-moderated scoring. Journal of Educational Measurement, 58(1), 130–149. doi:10.1111/jedm.12275
   Web of Science ®Google Scholar
• Wise, S. L., Pastor, D. A., & Kong, X. J. (2009). Correlates of rapid-guessing behavior in low-stakes testing: Implications for test development and measurement practice. Applied Measurement in Education, 22(2), 185–205. doi:10.1080/08957340902754650
   Web of Science ®Google Scholar
• Wise, S. L., & Smith, L. F. (2011). A model of examinee test-taking effort. In J. A. Bovaird, K. F. Geisinger, & C. W. Buckendal (Eds.), High-stakes testing in education: Science and practice in K-12 settings (pp. 139–153). Washington, DC: American Psychological Association. doi:10.1037/12330-009
   Google Scholar
• Wise, S. L., & Smith, L. F. (2016). The validity of assessment when students don’t give good effort. In G. T. L. Brown & L. R. Harris (Eds.), Handbook of human and social conditions in assessment (pp. 204–220). New York, NY: Routledge.
   Google Scholar
• Wood, C., Conner, M. T., Miles, E., Sandberg, T., Taylor, N. J., Godin, G., & Sheeran, P. (2016). The impact of asking intention or self-prediction questions on subsequent behavior: A meta-analysis. Personality and Social Psychology Review, 20(3), 245–268. doi:10.1177/1088868315592334
   PubMed Web of Science ®Google Scholar