Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

References

• Thistlethwaite J, Davies H, Dornan T, et al. What is evidence? Reflections on the AMEE symposium, Vienna, August 2011. Med Teach. 2012;34(6):454–10. Epub 2012 Apr 11.
   PubMed Web of Science ®Google Scholar
• Gordon M. Are we talking the same paradigm? Considering methodological choices in health education systematic review. Med Teach. 2016 Jul;38(7):746–750.
   PubMed Web of Science ®Google Scholar
• Bearman M, Dawson P. Qualitative synthesis and systematic review in health professions education. Med Educ. 2013;47(3):252–260.
   PubMed Web of Science ®Google Scholar
• Denzin NK. The elephant in the living room: or extending the conversation about the politics of evidence. Qual Res. 2009;9(2):139–160.
   Web of Science ®Google Scholar
• Dixon-Woods M, Bonas S, Booth A, et al. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006;6(1):27–44.
   Google Scholar
• Kirkpatrick D. Great ideas revisited: techniques for evaluating training programs. Revisiting Kirkpatrick’s four-level model. Training Dev. 1996 Jan;50(1):54–59.
   Google Scholar
• Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE Guide No. 67. Med Teach. 2012;34(5):e288–e299.
   PubMed Web of Science ®Google Scholar
• Yardley S, Dornan T. Kirkpatrick’s levels and education evidence. Med Educ. 2012;46(1):97–106.
   PubMed Web of Science ®Google Scholar
• Campbell K, Taylor V, Douglas S. Effectiveness of online cancer education for nurses and allied health professionals: a systematic review using Kirkpatrick Evaluation Framework. J Cancer Educ. 2019;34(2):339–356.
   Web of Science ®Google Scholar
• Gorbanev I, Agudelo-Londoño S, González RA, et al. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online. 2018;23(1):1–9.
   Web of Science ®Google Scholar
• Johnston S, Coyer FM, Nash R. Kirkpatrick’s evaluation of simulation and debriefing in health care education: a systematic review. J Nurs Educ. 2018;57(7):393–398.
   PubMed Web of Science ®Google Scholar
• Moreau KA. Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox? Med Teach. 2017;39(9):999–1001.
   PubMed Web of Science ®Google Scholar
• Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s four levels of training evaluation. Alexandria (VA): ATD Press; 2016.
   Google Scholar
• Lechner SK. Evaluation of teaching and learning strategies. Med Educ Online. 2001;6:1–4.
   Google Scholar
• Gordon M, Carneiro AV, Patricio MF. Enhancing the impact of BEME systematic reviews on educational practice [letter]. Med Teach. 2015;37(8):789–790.
   PubMed Web of Science ®Google Scholar
• Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Syst Rev. 2015;4(36):1–6.
   Web of Science ®Google Scholar
• Maudsley G, Taylor DCM, Allam O, et al. A Best Evidence Medical Education (BEME) systematic review of: what works best for health professions students using mobile (hand-held) devices for educational support on clinical placements? BEME Guide No. 52. Med Teach. 2019 Feb;41(2):125–140.
   PubMed Web of Science ®Google Scholar
• Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. 2008 Feb;42(2):128–133.
   PubMed Web of Science ®Google Scholar
• Colthart I, Bagnall G, Evans A, et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide No. 10. Med Teach. 2008;30(2):124–145.
   PubMed Web of Science ®Google Scholar
• Hammick M, Dornan T, Steinert Y. Conducting a best evidence systematic review. Part 1: from idea to data coding: BEME Guide No. 13. Med Teach. 2010;32(1):3–15.
   PubMed Web of Science ®Google Scholar
• Harden RM, Grant J, Buckley G, et al. BEME Guide No. 1: best evidence medical education. Med Teach. 1999;21(6):553–562.
   PubMed Web of Science ®Google Scholar
• Maxwell RJ. Quality assessment in health. Br Med J. 1984;288:1470–1472.
   PubMed Web of Science ®Google Scholar
• Maxwell R. Dimensions of quality revisited: from thought to action. Qual Health Care. 1992;1(3):171–177.
   Google Scholar
• Regehr G. It’s NOT rocket science: rethinking our metaphors for research in health professions education. Med Educ. 2010;44(1):31–39.
   Web of Science ®Google Scholar
• Creswell JW. Research design: qualitative, quantitative, and mixed methods approaches. 2nd ed. London: Sage Publications; 2003.
   Google Scholar
• Green J, Thorogood N. Qualitative methods for health research. 3rd ed. London: Sage Publications; 2014.
   Google Scholar
• Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013 Sep;15(3):398–405. Epub 2013 Mar 11.
   PubMed Web of Science ®Google Scholar
• Dixon-Woods M, Agarwal S, Jones D, et al. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005;10(1):45–53.
   PubMedGoogle Scholar
• Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol. 1993 May;46(5):423–429.
   PubMed Web of Science ®Google Scholar
• Looney SW, Hagan JL. Chapter 5: validation of biomarkers. In: Analysis of biomarker data: A practical guide. Hoboken (NJ): John Wiley & Sons; 2015. p. 255–331.
   Google Scholar
• Sarstedt M, Mooi E. Chapter 9: cluster analysis. In: A concise guide to market research. Springer Texts in Business and Economics. 3rd ed. Berlin: Springer-Verlag; 2019. p. 301–354.
   Google Scholar
• Bhanbhro S, Drennan VM, Grant R, et al. Assessing the contribution of prescribing in primary care by nurses and professionals allied to medicine: a systematic review of literature. BMC Health Serv Res. 2011;11:10.
   Web of Science ®Google Scholar
• Gillam SJ, Siriwardena AN, Steel N. Pay-for-performance in the United Kingdom: impact of the Quality and Outcomes Framework: a systematic review. Ann Fam Med. 2012 Sep–Oct;10(5):461–468.
   PubMed Web of Science ®Google Scholar
• Halter M, Wheeler C, Pelone F, et al. Contribution of physician assistants/associates to secondary care: a systematic review. BMJ Open. 2018 Jun 19;8(6):e019573:1-21.
   Web of Science ®Google Scholar
• Reio TG, Rocco TS, Smith DH, et al. A critique of Kirkpatrick’s evaluation model. New Horiz Adult Educ Hum Resour Dev. 2017;29(2):35–53.
   Web of Science ®Google Scholar
• Klassen A, Miller A, Anderson N, et al. Performance measurement and improvement frameworks in health, education and social services systems: a systematic review. Int J Qual Health Care. 2010;22(1):44–69.
   PubMed Web of Science ®Google Scholar
• Eva KW. Broadening the debate about quality in medical education research. Med Educ. 2009 Apr;43(4):294–296.
   PubMed Web of Science ®Google Scholar
• Maudsley G. What issues are raised by evaluating problem-based undergraduate medical curricula? Making healthy connections across the literature. J Eval Clin Pract. 2001;7(3):311–324.
   PubMed Web of Science ®Google Scholar
• Petticrew M. Why certain systematic reviews reach uncertain conclusions. Br Med J. 2003;326(7392):756–758.
   PubMed Web of Science ®Google Scholar
• Threlfall AG, Meah S, Fischer AJ, et al. The appraisal of public health interventions: the use of theory. J Public Health. 2015;37(1):166–171.
   Web of Science ®Google Scholar
• Taylor D. Quality and professionalism in health care: a review of current initiatives in the NHS. Br Med J. 1996;312(7031):626–629.
   Google Scholar
• Raven JH, Tolhurst RJ, Tang SL, et al. What is quality in maternal and neonatal health care? Midwifery. 2012;28(5):E676–E683.
   Web of Science ®Google Scholar
• Institute of Medicine (US National Academy of Sciences) Committee on Quality of Health Care in America; Richardson WC, Berwick DM, Bisgard JC, et al. Crossing the quality chasm: a new health system for the 21st Century [brief report]. Washington (DC): National Academies Press; 2001. p. 1–8. [cited 2020 Feb]; Available from: http://www.nationalacademies.org/hmd/~/media/Files/Report%20Files/2001/Crossing-the-Quality-Chasm/Quality%20Chasm%202001%20%20report%20brief.pdf
   Google Scholar
• Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44(3):166–206. Reprinted: Milbank Q. 2005;83(4):691–729.
   PubMedGoogle Scholar
• Berwick D, Fox DM. “Evaluating the quality of medical care”: Donabedian’s classic article 50 years later. Milbank Q. 2016;94(2):237–241.
   PubMed Web of Science ®Google Scholar
• Mulley A, Coulter A, Wolpert M, et al. New approaches to measurement and management for high integrity health systems. Br Med J. 2017 Mar;356:j1401:1–7.
   Google Scholar
• Harries U, Landes R, Popay J. Visual disability among older-people: a case-study in assessing needs and examining services. J Public Health Med. 1994;16(2):211–218.
   Google Scholar
• Rajpura A, Sethi S, Taylor M. An evaluation of two rapid access chest pain clinics in central Lancashire, UK. J Eval Clin Pract. 2007 Jun;13(3):326–336.
   Web of Science ®Google Scholar
• Chideka K, Klass C, Dunne S, et al. Listening to older adults: community consultation on a new dental service. Community Dent Health J. 2015 Dec;32(4):231–236.
   Web of Science ®Google Scholar
• Worsley DJ, Marshman Z, Robinson PG, et al. Evaluation of the telephone and clinical NHS urgent dental service in Sheffield. Community Dent Health. 2016 Mar;33(1):9–14.
   Web of Science ®Google Scholar
• Maudsley G. Mixing it but not mixed-up: mixed methods research in medical education (a critical narrative review) [Web-paper]. Med Teach. 2011;33(2):e92–e104.
   PubMed Web of Science ®Google Scholar
• Gordon M, Darbyshire D, Baker P. Separating the wheat from the chaff: the role of systematic review in medical education. Med Educ. 2013 Jun;47(6):632.
   PubMed Web of Science ®Google Scholar
• Gordon M, Vaz Carneiro A, Patricio M, et al. Missed opportunities in health care education evidence synthesis [letter]. Med Educ. 2014 Jun;48(6):644–645.
   Web of Science ®Google Scholar
• Eva KW. On the limits of systematicity. Med Educ. 2008;42(9):852–853.
   PubMed Web of Science ®Google Scholar
• Gordon M, Daniel M, Patricio M. What do we mean by ‘systematic’ in health education systematic reviews and why it matters! Med Teach. 2019 Aug;41(8):956–957.
   PubMed Web of Science ®Google Scholar
• Glass GV. Primary, secondary, and meta-analysis of research. Educ Researcher. 1976;5(10):3–8.
   Google Scholar
• Wilson DB, Lipsey MW. The role of method in treatment effectiveness research: evidence from meta-analysis. Psychol Methods. 2001;6(4):413–429.
   PubMed Web of Science ®Google Scholar
• Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012 Jun 9;1(28):1–9.
   Google Scholar