An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions

Abstract

Introduction

Programmatic assessment is an approach to assessment aimed at optimizing the learning and decision function of assessment. It involves a set of key principles and ground rules that are important for its design and implementation. However, despite its intuitive appeal, its implementation remains a challenge. The purpose of this paper is to gain a better understanding of the factors that affect the implementation process of programmatic assessment and how specific implementation challenges are managed across different programs.

Methods

An explanatory multiple case (collective) approach was used for this study. We identified 6 medical programs that had implemented programmatic assessment with variation regarding health profession disciplines, level of education and geographic location. We conducted interviews with a key faculty member from each of the programs and analyzed the data using inductive thematic analysis.

Results

We identified two major factors in managing the challenges and complexity of the implementation process: knowledge brokers and a strategic opportunistic approach. Knowledge brokers were the people who drove and designed the implementation process acting by translating evidence into practice allowing for real-time management of the complex processes of implementation. These knowledge brokers used a ‘strategic opportunistic’ or agile approach to recognize new opportunities, secure leadership support, adapt to the context and take advantage of the unexpected. Engaging in an overall curriculum reform process was a critical factor for a successful implementation of programmatic assessment.

Discussion

The study contributes to the understanding of the intricacies of implementation processes of programmatic assessment across different institutions. Managing opportunities, adaptive planning, awareness of context, were all critical aspects of thinking strategically and opportunistically in the implementation of programmatic assessment. Future research is needed to provide a more in-depth understanding of values and beliefs that underpin the assessment culture of an organization, and how such values may affect implementation.

Keywords:

• Assessment
• programmatic assessment
• implementation

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Glossary

Programmatic assessment: Is an approach to assessment aimed at optimizing the learning and decision function of assessment.

Additional information

Notes on contributors

Dario Torre

Dario Torre, MD, MPH, PHD, is professor of Medicine and Director of programs of assessment at University of Central Florida College of Medicine.

Lambert Schuwirth

Lambert Schuwirth, MD, PHD, is professor of medicine at Flinders University College of Medicine and Public Health, Adelaide, Australia.

Cees Van der Vleuten

Cees Van der Vleuten, PHD, is Professor of Education Department of Educational Development and Research, School of Health Profession Education, Maastricht University, The Netherlands.

Sylvia Heeneman

Sylvia Heeneman, PHD, is Professor of Education Department of Pathology, School Health Profession Education, Cardiovascular Research Institute Maastricht, Maastricht University, The Netherlands.