A resource efficient and reliable standard setting method for OSCEs: Borderline regression method using standardized patients as sole raters in clinical case encounters with medical students

References

• Berger AJ, Gillespie CC, Tewksbury LR, Overstreet IM, Tsai MC, Kalet AL, Ogilvie JB. 2012. Assessment of medical student clinical reasoning by “lay” vs physician raters: inter-rater reliability using a scoring guide in a multidisciplinary objective structured clinical examination. Am J Surg. 203(1):81–86.
   PubMed Web of Science ®Google Scholar
• Boulet JR, McKinley DW, Norcini JJ, Whelan GP. 2002. Assessing the comparability of standardized patient and physician evaluations of clinical skills. Adv Health Sci Educ Theory Pract. 7(2):85–97.
   PubMed Web of Science ®Google Scholar
• Boursicot KA, Roberts TE, Pell G. 2006. Standard setting for clinical competence at graduation from medical school: A comparison of passing scores across five medical schools. Adv Health Sci Educ Theory Pract. 11(2):173–183.
   PubMed Web of Science ®Google Scholar
• Boursicot KA, Roberts TE, Pell G. 2007. Using borderline methods to compare passing standards for OSCEs at graduation across three medical schools. Med Educ. 41(11):1024–1031.
   PubMed Web of Science ®Google Scholar
• Brennan R. 2010. Generalizability theory and classical test theory. Appl Measure Educ. 24(1):1–21.
   Web of Science ®Google Scholar
• Brennan RL, Kane MT. 1977. An index of dependability for mastery tests. J Educational Measurement. 14(3):277–289.
   Web of Science ®Google Scholar
• Cicchetti DV. 1994. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 6 (4):284–290.
   Google Scholar
• Cole S, Bird J. 2013. The medical interview: the three function approach. Philadelphia: Elsevier.
   Google Scholar
• Dauphinee WD, Blackmore DE, Smee S, Rothman AI, Reznick R. 1997. Using the judgments of physician examiners in setting the standards for a national multi-center high stakes OSCE. Adv Health Sci Educ Theory Pract. 2(3):201–211.
   PubMed Web of Science ®Google Scholar
• De Champlain AF. 2018. Standard setting in medical education: high stakes assessment. In: Swanwick T, Forrest K, O'Brien BC, editors. Understanding medical education, evidence,theory and practice. 3rd ed. Hoboken (NJ): Wiley Online Library; p. 347–359.
   Google Scholar
• Donohoe CL, Reilly F, Donnelly S, Cahill RA. 2020. Is there variability in scoring of student surgical OSCE performance based on examiner experience and expertise? J Surg Educ. 77(5):1202–1210.
   PubMed Web of Science ®Google Scholar
• Downing S, Yudkowsky R. 2009. Assessment in health professions education. New York: Routledge.
   Google Scholar
• Grochowalski JH, Liu Y, Siedlecki KL. 2016. Examining the reliability of ADAS-Cog change scores. Aging, Neuropsychology, and Cognition. 23(5):513–529.
   PubMed Web of Science ®Google Scholar
• Han JJ, Kreiter CD, Park H, Ferguson KJ. 2006. An experimental comparison of rater performance on an SP-based clinical skills exam. Teach Learn Med. 18(4):304–309.
   PubMed Web of Science ®Google Scholar
• Hejri SM, Jalili M, Muijtjens AM, Van Der Vleuten CP. 2013. Assessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination. J Res Med Sci. 18(10):887–891.
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Hawkins RE. 2008. Practical guide to the evaluation of clinical competence. Philadelphia: Mosby Elsevier.
   Google Scholar
• Homer M, Pell G. 2009. The impact of the inclusion of simulated patient ratings on the reliability of OSCE assessments under the borderline regression method. Med Teach. 31(5):420–425.
   PubMed Web of Science ®Google Scholar
• Howley LD. 2013. Standardized patients. In: Levine AI, DeMaria S, Schwartz AD, Sim AJ, editors. The comprehensive textbook of healthcare simulation. New York, NY. Springer; p. 173–190.
   Google Scholar
• Humphrey-Murto S, Macfadyen JC. 2002. Standard setting: a comparison of case-author and modified borderline-group methods in a small-scale OSCE. Acad Med. 77(7):729–732.
   PubMed Web of Science ®Google Scholar
• Kane MT. 2013. Validation as a pragmatic, scientific activity. J Educ Measure. 50(1):115–122.
   Web of Science ®Google Scholar
• Kilminster S, Roberts T. 2004. Standard setting for OSCEs: Trial of borderline approach. Adv Health Sci Educ Theory Pract. 9(3):201–209.
   PubMed Web of Science ®Google Scholar
• Kramer A, Muijtjens A, Jansen K, Dusman H, Tan L, Van der Vleuten C. 2003. Comparison of a rational and an empirical standard setting procedure for an OSCE. Med Educ. 37(6):574–574.
   Web of Science ®Google Scholar
• Lockyer J, Carraccio C, Chan M-K, Hart D, Smee S, Touchie C, Holmboe ES, Frank JR, ICBME Collaborators 2017. Core principles of assessment in competency-based medical education. Med Teach. 39 (6):609–616.
   PubMed Web of Science ®Google Scholar
• Mahmood O, Dagnaes J, Bube S, Rohrsted M, Konge L. 2018. Nonspecialist raters can provide reliable assessments of procedural skills. J Surg Educ. 75(2):370–376.
   PubMed Web of Science ®Google Scholar
• Malau-Aduli BS, Teague P-A, D'Souza K, Heal C, Turner R, Garne DL, van der Vleuten C. 2017. A collaborative comparison of objective structured clinical examination (OSCE) standard setting methods at Australian medical schools. Med Teach. 39(12):1261–1267.
   PubMed Web of Science ®Google Scholar
• May W, Park JH, Lee JP. 2009. A ten-year review of the literature on the use of standardized patients in teaching and learning: 1996–2005. Med Teach. 31(6):487–492.
   PubMed Web of Science ®Google Scholar
• McKinley D, Norcini J. 2014. How to set standards on performance-based examinations: AMEE Guide No. 85. Med Teach. 36(2):97–110.
   PubMed Web of Science ®Google Scholar
• Patnaik R, Anton NE, Stefanidis D. 2020. A video anchored rating scale leads to higher inter-rater reliability of inexperienced and expert raters in the absence of rater training. Am J Surg. 219(2):221–226.
   PubMed Web of Science ®Google Scholar
• Pell G, Fuller R, Homer M, Roberts T, International Association for Medical Education. 2010. How to measure the quality of the OSCE: a review of metrics – AMEE guide no. 49. Med Teach. 32(10):802–811.
   PubMed Web of Science ®Google Scholar
• Pradarelli JC, Gupta A, Hermosura AH, Murayama KM, Delman KA, Shabahang MM, Havens JM, Lipsitz S, Smink DS, Yule S. 2021. Non-technical skill assessments across levels of US surgical training. Surgery. 170(3):713–718.
   PubMed Web of Science ®Google Scholar
• Pradarelli JC, George E, Kavanagh J, Sonnay Y, Khoon TH, Havens JM. 2021. Training novice raters to assess nontechnical skills of operating room teams. J Surg Educ. 78(2):386–390.
   PubMed Web of Science ®Google Scholar
• Price T, Lynn N, Coombes L, Roberts M, Gale T, Regan de Bere S, Archer J. 2018. The international landscape of medical licensing examinations: a typology derived from a systematic review. Int J Health Policy Manag. 7(9):782–790.
   PubMed Web of Science ®Google Scholar
• Roch SG, Woehr DJ, Mishra V, Kieszczynska U. 2012. Rater training revisited: an updated meta-analytic review of frame-of-reference training. J Occup Organiz Psychol. 85(2):370–395.
   Web of Science ®Google Scholar
• Scheffer S, Muehlinghaus I, Froehmel A, Ortwein H. 2008. Assessing students' communication skills: validation of a global rating. Adv Health Sci Educ Theory Pract. 13(5):583–592.
   PubMed Web of Science ®Google Scholar
• Shrout PE, Fleiss JL. 1979. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 86(2):420–428.
   PubMed Web of Science ®Google Scholar
• Solomon D, Szauter K, Rosebraugh C, Callaway M. 2000. Global ratings of student performance in a standardized patient examination: is the whole more than the sum of the parts? Adv Health Sci Educ. 5(2):131–140.
   PubMed Web of Science ®Google Scholar
• Whelan GP, Boulet JR, McKinley DW, Norcini JJ, van Zanten M, Hambleton RK, Burdick WP, Peitzman SJ. 2005. Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG clinical skills assessment (CSA)®). Med Teach. 27(3):200–206.
   PubMed Web of Science ®Google Scholar
• Wilkinson TJ, Newble DI, Frampton CM. 2008. Standard setting in an objective structured clinical examination: use of global ratings of borderline performance to determine the passing score. Med Educ. 35(11):1043–1049.
   Google Scholar
• Wood TJ, Humphrey-Murto SM, Norman GR. 2006. Standard setting in a small scale OSCE: A comparison of the modified borderline-group method and the borderline regression method. Adv Health Sci Educ Theory Pract. 11(2):115–122.
   PubMed Web of Science ®Google Scholar
• Yousuf N, Violato C, Zuberi RW. 2015. Standard setting methods for pass/fail decisions on high-stakes objective structured clinical examinations: a validity study. Teach Learn Med. 27(3):280–291.
   PubMed Web of Science ®Google Scholar
• Yudkowsky R, Hyderi A, Holden J, Kiser R, Stringham R, Gangopadhyaya A, Khan A, Park YS. 2019. Can nonclinician raters be trained to assess clinical reasoning in post encounter patient notes? Acad Med. 94(11S):S21–S27.
   PubMedGoogle Scholar
• Zanten MV, Boulet JR, Mckinley D. 2007. Using standardized patients to assess the interpersonal skills of physicians: six years' experience with a high-stakes certification examination. Health Commun. 22(3):195–205.
   PubMed Web of Science ®Google Scholar