A resource efficient and reliable standard setting method for OSCEs: Borderline regression method using standardized patients as sole raters in clinical case encounters with medical students

Abstract

Finding a reliable, practical and low-cost criterion-referenced standard setting method for performance-based assessments has proved challenging. The borderline regression method of standard setting for OSCEs has been shown to estimate reliable scores in studies using faculty as raters. Standardized patients (SPs) have been shown to be reliable OSCE raters but have not been evaluated as raters using this standard setting method. Our study sought to find whether SPs could be reliably used as sole raters in an OSCE of clinical encounters using the borderline regression standard setting method.

SPs were trained for on a five-point global rating scale. In an OSCE for medical students, SPs completed skills checklists and the global rating scale. The borderline regression method was used to create case passing scores. We estimated the dependability of the final pass or fail decisions and the absolute dependability coefficients for global ratings, checklist scores, and case pass-score decisions using generalizability theory.

The overall dependability estimate is 0.92 for pass or fail decisions for the complete OSCE. Dependability coefficients (0.70–0.86) of individual case passing scores range demonstrated high dependability.

Based on our findings, the borderline regression method of standard setting can be used with SPs as sole raters in a medical student OSCE to produce a dependable passing score. For those already using SPs as raters, this can provide a practical criterion-referenced standard setting method for no additional cost or faculty time.

Keywords:

• Standard-setting
• OSCE
• standardized patients (SPs)
• global-rating
• borderline-regression
• generalizability theory

Acknowledgements

The authors would like to acknowledge Richard Fuller for his guidance and for sharing the global rating used in his studies and his invaluable help in adapting the global rating for use with standardized patients. Thanks to him and to Godfrey Pell for their informative webinar on the borderline regression method that got us launched using it at our institution. Thanks to Anna Lank, SP trainer extraordinaire who helped me train the SPs to use the global rating.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Glossary

Standard setting: The process of identifying the scores (cut scores) on a score scale that define the starting and ending points of various performance levels used for reporting test performance. For example, the process is used to determine the lowest score that could be obtained to categorize performance as ‘pass’ rather than ‘fail’.

Dependability coefficient: The ɸ coefficient (also known as Φ, or the dependability coefficient for absolute decisions) in particular is used to estimate the overall dependability, or proportion of universe score variance, of a set of scores used for absolute (or criterion referenced) decisions.

Angoff method: The Angoff method requires expert judges to discuss the issues involved in determining a pass mark and to evaluate the examination by using a well-defined and rational procedure. The Angoff method is based on the concept of the borderline or minimally competent candidate.

Additional information

Notes on contributors

Felise B. Milan

Felise Milan, MD, Professor of Medicine, Director of the clinicals skills center, assessment program, curriculum at Albert Einstein College of Medicine, a co-founder and past President of Directors of Clinicals Skills Courses.

Joseph H. Grochowalski

Joseph Grochowalski, PhD, is a Psychometrician at the College Board, Adjunct Faculty of Psychometrics and Quantitative Psychology at Fordham University.