An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions

References

• Albanese MA, Mitchell S. 1993. Problem-based learning: a review of literature on its outcomes and implementation issues. Acad Med. 68(1):52–52.
   PubMed Web of Science ®Google Scholar
• Anderson HL, Kurtz J, West DC. 2021. Implementation and use of workplace-based assessment in clinical learning environments: a scoping review. Acad Med. 96(11S):S164–S174.
   Google Scholar
• Ang I. 2011. Navigating complexity: From cultural critique to cultural intelligence. Continuum. 25(6):779–794.
   Web of Science ®Google Scholar
• ATLAS ti. version 9.1 Qualitative Data Analysis. 2020.
   Google Scholar
• Bok HG, Teunissen PW, Favier RP, Rietbroek NJ, Theyse LF, Brommer H, Haarhuis JC, van Beukelen P, van der Vleuten CP, Jaarsma DA. 2013. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 13(1):1–9.
   PubMed Web of Science ®Google Scholar
• Boyd VA, Whitehead CR, Thille P, Ginsburg S, Brydges R, Kuper A. 2018. Competency‐based medical education: the discourse of infallibility. Med Educ. 52(1):45–57.
   PubMed Web of Science ®Google Scholar
• Braun V, Clarke V. 2006. Using thematic analysis in psychology. Qual Res Psychol. 3(2):77–101.
   Google Scholar
• Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. 2002. Shifting paradigms: from Flexner to competencies. Acad Med. 77(5):361–367.
   PubMed Web of Science ®Google Scholar
• Chan T, Sherbino J, 2015. The McMaster Modular Assessment Program (McMAP): a theoretically grounded work-based assessment system for an emergency medicine residency program. Acad Med. 90(7):900–905.
   PubMed Web of Science ®Google Scholar
• Cleland J, MacLeod A, Ellaway RH. 2021. The curious case of case study research. Med Educ. 55(10):1131–1141.
   PubMed Web of Science ®Google Scholar
• Crabtree BF. 1999. Doing qualitative research. London: SAGE.
   Google Scholar
• Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. 2009. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 4(1):50–15.
   PubMed Web of Science ®Google Scholar
• Dannefer EF, Bierer SB, Gladding SP. 2012. Evidence within a portfolio-based assessment program: what do medical students select to document their performance? Med Teach. 34(3):215–220.
   PubMed Web of Science ®Google Scholar
• Dolmans DH, De Grave W, Wolfhagen IH, Van Der Vleuten CP. 2005. Problem‐based learning: future challenges for educational practice and research. Med Educ. 39(7):732–741.
   PubMed Web of Science ®Google Scholar
• Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. 2012. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 34(3):226–231.
   PubMed Web of Science ®Google Scholar
• Frambach JM, Driessen EW, Chan LC, van der Vleuten CP. 2012. Rethinking the globalisation of problem‐based learning: how culture challenges self‐directed learning. Med Educ. 46(8):738–747.
   PubMed Web of Science ®Google Scholar
• Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010. Competency-based medical education: theory to practice. Med Teach. 32(8):638–645.
   PubMed Web of Science ®Google Scholar
• Gordon L, Cleland JA. 2021. Change is never easy: how management theories can help operationalise change in medical education. Med Educ. 55(1):55–64.
   PubMed Web of Science ®Google Scholar
• Gosselin K, Norris JL, Ho M-J. 2016. Beyond homogenization discourse: reconsidering the cultural consequences of globalized medical education. Med Teach. 38(7):691–699.
   PubMed Web of Science ®Google Scholar
• Hall GH, Hord SM. 2001. Implementing change: patterns, principles, and potholes, 4th ed. Boston: Peason.
   Google Scholar
• Harris P, Snell L, Talbot M, Harden RM. 2010. Competency-based medical education: implications for undergraduate programs. Med Teach. 32(8):646–650.
   PubMed Web of Science ®Google Scholar
• Harrison CJ, Könings KD, Schuwirth LW, Wass V, Van der Vleuten CP. 2017. Changing the culture of assessment: the dominance of the summative assessment paradigm. BMC Med Educ. 17(1):1–14.
   PubMed Web of Science ®Google Scholar
• Harrison H, Birks M, Franklin R, Mills J. 2017, January. Case study research: foundations and methodological orientations. In Forum Qualitative Sozialforschung/Forum: Qualitative Social Research (Vol. 18, No. 1).
   Google Scholar
• Hawkins RE, Welcher CM, Holmboe ES, Kirk LM, Norcini JJ, Simons KB, Skochelak SE. 2015. Implementation of competency‐based medical education: are we addressing the concerns and challenges? Med Educ. 49(11):1086–1102.
   PubMed Web of Science ®Google Scholar
• Heeneman S, de Jong LH, Dawson LJ, Wilkinson TJ, Ryan A, Tait GR, Rice N, Torre D, Freeman A, van der Vleuten CP. 2021. Ottawa 2020 consensus statement for programmatic assessment–1. Agreement on the principles. Med Teacher. 43(10):1–10.
   PubMed Web of Science ®Google Scholar
• Heeneman S, Oudkerk Pool A, Schuwirth LW, van der Vleuten CP, Driessen EW. 2015. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 49(5):487–498.
   PubMed Web of Science ®Google Scholar
• Heeneman S, Schut S, Donkers J, van der Vleuten C, Muijtjens A. 2017. Embedding of the progress test in an assessment program designed according to the principles of programmatic assessment. Med Teacher. 39(1):44–52.
   PubMed Web of Science ®Google Scholar
• Hord SM, Roussin JL. 2013. Implementing change through learning: concerns-based concepts, tools, and strategies for guiding change. Thousand Oaks: Corwin Press.
   Google Scholar
• Iobst WF, Holmboe ES. 2020. Programmatic assessment: the secret sauce of effective CBME implementation.
   Google Scholar
• Jamieson J, Jenkins G, Beatty S, Palermo C. 2017. Designing programmes of assessment: a participatory approach. Med Teach. 39(11):1182–1188.
   PubMed Web of Science ®Google Scholar
• Jippes M, Driessen EW, Majoor GD, Gijselaers WH, Muijtjens AM, van der Vleuten CP. 2013. Impact of national context and culture on curriculum change: a case study. Med Teach. 35(8):661–670.
   PubMed Web of Science ®Google Scholar
• Kulasegaram K, Mylopoulos M, Tonin P, Bernstein S, Bryden P, Law M, Lazor J, Pittini R, Sockalingam S, Tait GR, et al. 2018. The alignment imperative in curriculum renewal. Med Teach. 40(5):443–448.
   PubMed Web of Science ®Google Scholar
• Mason M. 2009. Making educational development and change sustainable: insights from complexity theory. Int J Educ Dev. 29(2):117–124.
   Web of Science ®Google Scholar
• McGaghie WC. 2015. Mastery learning: it is time for medical education to join the 21st century. Acad Med. 90(11):1438–1441.
   PubMed Web of Science ®Google Scholar
• Mennin SP. 2013. Health professions education: complexity, teaching, and learning. In Handbook of systems and complexity in health. New York, NY: Springer; p. 755–766.
   Google Scholar
• Plsek PE, Wilson T. 2001. Complexity, leadership, and management in healthcare organisations. BMJ. 323(7315):746–749.
   PubMed Web of Science ®Google Scholar
• Roberts C, Shadbolt N, Clark T, Simpson P. 2014. The reliability and validity of a portfolio designed as a programmatic assessment of performance in an integrated clinical placement. BMC Med Educ. 14(1):1–11.
   PubMed Web of Science ®Google Scholar
• Ross S, Hauer KE, Wycliffe-Jones K, Hall AK, Molgaard L, Richardson D, Oswald A, Bhanji F, ICBME Collaborators. 2021. Key considerations in planning and designing programmatic assessment in competency-based medical education. Medical Teacher. 43(7):758–764.
   PubMed Web of Science ®Google Scholar
• Rosas SR. 2017. Systems thinking and complexity: considerations for health promoting schools. Health Promot Int. 32(2):301–311.
   PubMed Web of Science ®Google Scholar
• Sanger M, Giddings MM. 2012. A simple approach to complexity theory. J Social Work Educ. 48(2):369–376.
   Web of Science ®Google Scholar
• Schut S, Maggio LA, Heeneman S, van Tartwijk J, van der Vleuten C, Driessen E. 2021. Where the rubber meets the road – an integrative review of programmatic assessment in health care professions education. Perspect Med Educ. 10(1):6–13.
   PubMed Web of Science ®Google Scholar
• Schuwirth LWT, Van der Vleuten CPM. 2011. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 33(6):478–485.
   PubMed Web of Science ®Google Scholar
• Serrador P, Pinto JK. 2015. Does Agile work? – A quantitative analysis of agile project success. Int J Project Manage. 33(5):1040–1051.
   Web of Science ®Google Scholar
• Stakes RE. 1995. The art of case study research. Thousand Oaks, CA: SAGE.
   Google Scholar
• Thornton PH, Ocasio W. 2008. Institutional logics. The Sage Handbook of Organizational Institutionalism. 840(2008):99–128.
   Google Scholar
• Timmerman AA, Dijkstra J. 2017. A practical approach to programmatic assessment design. Adv Health Sci Educ Theory Pract. 22(5):1169–1182.
   PubMed Web of Science ®Google Scholar
• Torre D, Rice NE, Ryan A, Bok H, Dawson LJ, Bierer B, Wilkinson TJ, Tait GR, Laughlin T, Veerapen K, et al. 2021. Ottawa 2020 consensus statements for programmatic assessment 2: implementation and practice. Medical teacher43(10):1149–1160.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten CP, Schuwirth LW. 2005. Assessing professional competence: from methods to programmes. Med Educ. 39(3):309–317.
   PubMed Web of Science ®Google Scholar
• Van der Vleuten CP, Schuwirth LW. 2019. Assessment in the context of problem-based learning. Advin Health Sci Edu. 24(5):903–904.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten CP, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. 2015. Twelve tips for programmatic assessment. Med Teach. 37(7):641–646.
   PubMed Web of Science ®Google Scholar
• Waldrop MM, Gleick J. 1992. Complexity: the emerging science at the edge of order and chaos [m]. info. London: Viking.
   Google Scholar
• Waterval DG, Frambach JM, Driessen EW, Scherpbier AJ. 2015. Copy but not paste: a literature review of crossborder curriculum partnerships. J Stud Int Educ. 19(1):65–85.
   Web of Science ®Google Scholar
• Watling CJ, Ginsburg S. 2019. Assessment, feedback and the alchemy of learning. Med Educ. 53(1):76–85.
   PubMed Web of Science ®Google Scholar
• Wenger-Trayner E, Wenger-Trayner B. 2014. Learning in a landscape of practice: a framework. In Learning in landscapes of practice. New York: Routledge; p. 13–29.
   Google Scholar
• Wilkinson TJ, Tweed MJ. 2018. Deconstructing programmatic assessment. Adv Med Educ Pract. 9:191–197.
   PubMed Web of Science ®Google Scholar
• Wilkinson TJ, Tweed MJ, Egan TG, Ali AN, McKenzie JM, Moore M, Rudland JR. 2011. Joining the dots: conditional pass and programmatic assessment enhances recognition of problems with professionalism and factors hampering student progress. BMC Med Educ. 11(1):1–9.
   PubMed Web of Science ®Google Scholar
• Woodruff JN. 2019. Accounting for complexity in medical education: a model of adaptive behaviour in medicine. Med Educ. 53(9):861–873.
   PubMed Web of Science ®Google Scholar
• Yin RK. 2009. Case study research: design and methods (Vol. 5). Thousand Oaks: SAGE.
   Google Scholar