Does developing multiple-choice Questions Improve Medical Students’ Learning? A Systematic Review

Figures & data

Table 1. Inclusion & exclusion criteria

	Inclusion criteria	Exclusion criteria
Population	Under-graduate/graduate medical students	Other health-related students: biomedical, nursing, and dental students
Intervention	Writing MCQs	Writing questions other than MCQs
Outcome	learning in terms of perceptions, performance, behavior	Entirely descriptive papers without an evaluation section

Figure 1. Flow-chart of the study selection.

Table 2. Demographics, interventions, and outcome of the included studies

Author, year	Country	Study design	Participants	Subject	Intervention	Co-intervention	MCQtype	MCQ Quality	Instrument	Findings
Palmer Eet al. 2006 [10]	Australia	RCT	4th year N = 51	Surgery	Students were split randomly into 2 groups, the first group had to write a case report while the second had to write 3 MCQs in addition to the case report. Students were guided to write good MCQs	NR	NR	61% of MCQs: acceptable quality.Only 25% tested higher-order skills	Pre/post-test.	no significant difference
									Survey (learning methods ranking)	Both groups ranked learning methods similarly
	Australia	Single group cross-sectional	5th year N = 53	Diverse	Students were asked to rank their preferred learning methods before and after an activity in which they had to do research on a topic witha presentation and construct 3 MCQs for their peers to answer	NR	NR	Students created good quality MCQs.	Survey	a significant difference for the MCQ as a learning exercise option (p = 0.04) but the ranking among other activities remains poor
Chamberlain S et al. 2006 [43]	UK	Single-group cross-sectional	1st year N = 3, 2nd year N = 3	Diverse	After item-writing training Students worked singly or with peers to create (MCQs).students were getting paid to write MCQs	Feedback on each option given.	Type A	NR	Qualitative feedback	Students reported that the method helped them to consolidate their knowledge and comprehension of the curriculum
Gooi ACet al., 2014 [46]	Canada	Single-cross-sectional	1st year N = 113	Oto-laryngology	First step: Introductory session to how to write a high quality MCQs. Second step: Self-study sessionLast step: students created MCQs.MCQs were reviewed by instructors	Studentsreviewed each other’s MCQs	Type A	NR	Survey	Creating MCQs valuable = 78%.Reviewing MCQs valuable = 79% Class-created MCQs is a valuable resource = 91%Interested in collaborating in future session = 86%
Grainger Ret al. 2018 [30]	NewZealand	Singlegroup cross-sectional	4th year N = 106	Anatomic pathology	Students were instructed to write MCQs using bloom taxonomy and were required to write at least 2 in each module of anatomic pathology using PeerWise®	Rate and answer peer-generated MCQs, explain the answer and distractors.	Type A	74% of the MCQs were tested high order skills	Survey.	Students that were not satisfied with the process = 81%. Only 37% of students believed that interaction with peers was useful
Shah MPet al. 2019 [25]	USA	Single-group cross-sectional	2nd year N = 11	Cardio-pulmonary-renal	Students tooka workshop on how to write good quality MCQs and were asked to write MCQs on the lecture topics they were given	Giveexplanations of answers and distractors	NR	NR	Qualitative feedback	Studentsreported that creating MCQs and explaining answers helped them review key objectives and refined their test-taking strategies and would like to engage in similar activities
Walsh J et al. 2016 [44]	UK	Cohort	Final year (5th) N = 20.	Diverse	Students were instructed to write and create a bank of MCQs, Questions produced were arranged into a series of tests. the performance of question writers was compared to the rest of the students	Students had to meet for peer and senior clinician review of their MCQs	Type A	NR	Post-test	Students who wrote and reviewed questions scored higher than average compared to the rest of the students at the end of year summative exam (p ≤ 0.001)
Kurtz JB et al. 2019 [38]	USA	Single group cross-sectional	2nd yearN = 18	Cardiology	Participants were randomly divided into 6 groups of 3 students each, then students had to write 2 MCQs from exam blueprint subjects	Review MCQs with peers and faculty	Type A	NR	Qualitative analysis (n = 8).	Students found this activity beneficial on how to strategically approach MCQ examinations.Students voiced frustration about the time consumption
									Open-ended survey (n = 10).	Students found the activity beneficial for their learning (mean = 3.9 ± 0.3), They did not agree or deny that this was an efficient method to review cardiology (mean = 2.9 ± 1.1)
Benjamin HL et al. 2015 [47]	UK	Single group cross-sectional	5th year N = 20	Diverse	Students were asked to volunteer to create an online MCQ database, students had to write MCQs in a standard format, then the senior clinician was asked to review and approve MCQs	Students checked and reviewed each other’s MCQs	Type A	High quality	Mixed method feedback	100% positive feedback students reported question writing and/or peer review to be valuable for learning and useful for preparation and described it as enjoyable
Herrero JIet al. 2019 [26]	Spain	RCT	2nd & 3rd year N = 75 & N = 109	General pathology & Physiopathology	Students were invited to write 4 MCQs on a topic that was randomly chosen. They were offered an extra 0.25 point if their questions were good enough. The best 2 questions on each topic were selected to be included in the exam	NR	NR	Poor quality	Post-test	Students performed significantly better when writing MCQs on certain modules compared to others
Rajendiren S et al. 2014 [27]	India	Single-group Pre and post-test	First-year N = 135	Biochemistry	Students were classified into three categories: high, medium, and low performers. They took a pre-test, then the 3 different groups were given MCQ stems of the same subject and were asked to create distractors and the right answer separately, then they were tested again	NR	Distractors and answer	Good quality	Pre and post-tests.	A significant difference between pre and post-tests in both high and low performers
									Students feedback	55% of students found the exercise to be challenging and must be used as a learning exercise
Bobby Zet al. 2012 [48]	India	Single-group Pre and post-test	First-year N = 84	Biochemistry	Students took a pre-test then they were given 4 distractors in which one could be the answer. They were then asked to individually write the stem based on given keywords	Students engaged in small group discussions to review and modify MCQs of the group	Stems of MCQs	High quality	Pre and post-tests.	A significant difference between pre and the two post-tests in all students’ categories
									Students Feedback	95% of students wanted a second session in the future, 99% felt the exercise was not a burden
Sircar SSet al. 1999 [52]	India	Single-group cross-sectional	First-year N = 37	Physiology	A contest in which students had to write MCQs was organized. the contributors of the best and highest number of MCQs would be awarded certificates. And the best MCQs would be included in exams	NR	assertion-reasons MCQs	NR	Students Feedback	Most students agreed the contest was useful to their learning, though some found it time-consuming
McLeod PJ & Snell L. 1996 [28]	Canada	Single-group cross-sectional	2nd & 3rd year N = 150	Diverse	Students were divided into 3 groups, each one spent in rotation a 10 weeks clinical course. Each one was expected to write two to five MCQs. All accepted student-generated questions were included in the summative exams	NR	24% of MCQs were clinical case-based	Good quality	Students Feedback	Students appreciated how involved they were in the learning process. They recognized the benefits of reading when formulating a question
Papinczak T et al. 2012 [49]	Australia	Cohort	1st & 2nd year N = 384	Diverse	In small groups, students were asked to devote at least 1 hour a week to write MCQs type assertion-reason.MCQs and answers were reviewed by one or 2 academics before being loaded to a questions bank	Students were also asked to write short-answer & complex patient-based questions	assertion-reasonMCQs	Good quality	Pre and post-test.	Slight drop in grades
									Questionnaire	26% of students found the activity time consuming and challenging than expected. 77.3% supported the continuation of the project.
Stone MRet al. 2017 [31]	USA	Single-group cross-sectional	1st & 2nd year (N = 39).	Diverse	Students were asked to participate in crowd-sourced practice quizzes made of MCQs, matching, and true/false questions, based on the material taught. Each participant had to write questions for a certain number of lectures	Students took quizzes made of questions they generated	NR	Good quality	Post-test	No statistically significant difference between participants and non-participants. low performers benefited more from the process. Writing and taking tests were more effective than each one alone
									Survey	81.3% of students stated they felt more positive when they wrote MCQ
Walsh JLet al. 2018 [29]	UK	Cohort	1st & 2nd year N = 603	Diverse	PeerWise® was introduced to the first-year class of 2014. Over 2 years, students were asked to write MCQs.	Students were also asked to comment and rate their peers’ MCQs	NR	Acceptable quality	Post-test	There were significant correlations between writing, answering, and commenting frequency with summative examination performance (p < 0.001, R = 0.24, 0.13, and 0.15, respectively). PeerWise® users performed significantly better than non-users (p < 0.001)
									Students feedback	Students appreciated curricular specificity, and they were worried about the quality of student-authored questions
Jobs A et al. 2013 [45]	Germany	Single-group cross-sectional	4th year N = 102	Internal Medicine	Internal Medicine was divided into 4 sections which students had to take an exam on each. Students wrote MCQs 3 weeks before exams instructed by an approved manual, exam included some questions written by students	NR	Type A	Poor quality	Post-test	Low performers did significantly better while high performers didn’t have a measurable advantage
									Questionnaire	students spent less time designing MCQs compared to other methods. No apparent beneficial effects on learning habits

MCQs: Multiple-choice questions; N: Number; NR: Not reported; RCT: Randomized controlled trial

Table 3. Methodological quality of included studies according to MERSQI

		Sampling			Validity evidence for evaluation instrument scores			Data analysis		Outcome (j)
Authors	Study design (a)	Institutions studied (b)	Response rate (c)	Type of data (d)	Content (e)	Internal structure (f)	Relationships to other variables (g)	Appropriateness of analysis (h)	Complexity of analysis (i)	Satisfaction, Attitudes, perceptions	Knowledge,Skills	Behaviors	Patient/healthCare outcomes	Total
Palmer E et al., 2006 [9]	3	0.5	1.5	3	1	0	0	1	2	-	1.5	-	-	13.5
	1	0.5	0.5	1	0	0	0	1	2	1	-	-	-	7
Chamberlain S et al. 2006 [43]	-	-	-	-	-	-	-	-	-	-	-	-	-	-
Gooi AC et al., 2014 [46]	1	0.5	1.5	1	0	0	0	1	1	1	-	-	-	7
Grainger R et al. 2018 [30]	1	0.5	1	1	1	1	1	1	1	1	-	-	-	9.5
Shah MP et al. 2019 [25]	-	-	-	-	-	-	-	-	-	-	-	-	-	-
Walsh J et al. 2016 [44]	2	0.5	1.5	3	1	0	0	1	2	-	1.5	-	-	12.5
Kurtz JB et al. 2019 [38]	1	0.5	1.5	1	0	0	0	1	1	1	-	-	-	7
Benjamin H. L. 2015 [47]	1	0.5	1.5	1	0	0	0	1	1	1	-	-	-	7
Herrero JI et al. 2019 [26]	3	0.5	1.5	3	1	0	0	1	2	-	1.5	-	-	13.5
Rajendiren S et al. 2014 [27]	1.5	0.5	1.5	3	1	0	0	1	2	-	1.5	-	-	12
Zachariah Bobby, 2012 [48]	1.5	0.5	1.5	3	1	0	0	1	2	-	1.5	-	-	12
Sircar SS et al. 1999 [52]	-	-	-	-	-	-	-	-	-	-	-	-	-	-
McLeod PJ & Snell L. 1996 [28]	-	-	-	-	-	-	-	-	-	-	-	-	-	-
Papinczak T et al. 2012 [49]	2	0.5	1.5	3	1	0	0	1	2	-	1.5	-	-	12.5
Stone MR et al. 2017 [31]	1.5	0.5	1.5	3	1	1	1	1	2	-	1.5	-	-	14
Walsh JL et al. 2018 [29]	1.5	0.5	1.5	3	1	0	0	1	2	-	1.5	-	-	12
Alexander Jobs, 2013 [45]	1	0.5	1	3	1	0	0	1	2	-	1.5	-	-	11

Details of MERSQI Scoring:

a. Study design: Single group cross-sectional/post-test only (1); single group pre- and post-test (1.5); nonrandomized 2 groups (2); randomized controlled experiment (3).

b. Sampling: Institutions studied: Single institution (0.5); 2 institutions (1); More than 2 institutions (1.5).

c. Sampling: Response rate: Not applicable (0); Response rate < 50% or not reported (0.5); Response rate 50–74% (1); Response rate > 75% (1.5).

d. Type of data: evaluation by study participants (1); Objective measurement (3).

e. Validity evidence for evaluation instrument scores: Content: Not reported/ Not applicable (0); Reported (1).

f. Validity evidence for evaluation instrument scores: Internal structure: Not reported/ Not applicable (0); Reported (1).

g. Validity evidence for evaluation instrument scores: Relationships to other variables: Not reported/ Not applicable (0); Reported (1).

h. Appropriateness of analysis: Inappropriate (0); appropriate (1)

i. Complexity of analysis: Descriptive analysis only (1); Beyond descriptive analysis (2).

j. Outcome: Satisfaction, attitudes, perceptions (1); Knowledge, skills (1.5); Behaviors (2); Patient/health care outcome (3)

Palmer E, Devitt P. Constructing multiple choice questions as a method for learning. Ann Acad Med Singap. 2006 Sep;35(9):604–608.

 PubMed Web of Science ®Google Scholar

Chamberlain S, Freeman A, Oldham J, et al. Innovative learning: employing medical students to write formative assessments. Med Teach. 2006 Nov;28(7):656–659.

 PubMed Web of Science ®Google Scholar

Gooi ACC, Sommerfeld CS. Medical school 2.0: how we developed a student-generated question bank using small group learning. Med Teach. 2015;37(10):892–896.

 PubMed Web of Science ®Google Scholar

Grainger R, Dai W, Osborne E, et al. Medical students create multiple-choice questions for learning in pathology education: a pilot study. BMC Med Educ. 2018 Aug;18(1):201.

 PubMedGoogle Scholar

Shah MP, Lin BR, Lee M, et al. Student-written multiple-choice questions—a practical and educational approach. Med.Sci.Educ 2019 Mar;29(1):41–43.

 PubMedGoogle Scholar

Walsh J, Harris B, Tayyaba S, et al. Student-written single-best answer questions predict performance in finals. Clin Teach. 2016 Oct;13(5):352–356.

 PubMed Web of Science ®Google Scholar

Kurtz JB, Lourie MA, Holman EE, et al. Creating assessments as an active learning strategy: what are students’ perceptions? A mixed methods study. Med Educ Online. 2019 Dec;24(1):1630239.

 PubMed Web of Science ®Google Scholar

Harris BHL, Walsh JL, Tayyaba S, et al. A novel student-led approach to multiple-choice question generation and online database creation, with targeted clinician input. Teach Learn Med. 2015;27(2):182–188.

 PubMed Web of Science ®Google Scholar

Herrero JI, Lucena F, Quiroga J. Randomized study showing the benefit of medical study writing multiple choice questions on their learning. BMC Med Educ. 2019 Jan;19(1):42.

 PubMedGoogle Scholar

Rajendiren S, Dhiman P, Rajendiren S, et al. Making concepts of medical biochemistry by formulating distractors of multiple choice questions: growing mighty oaks from small acorns. J Contem Med Edu. 2014 Jun;2(2):123–127.

 Google Scholar

Bobby Z, Radhika MR, Nandeesha H, et al. Formulation of multiple choice questions as a revision exercise at the end of a teaching module in biochemistry. Biochem Mol Biol Educ. 2012 May;40(3):169–173.

 PubMed Web of Science ®Google Scholar

Sircar SS, Tandon OP. Involving students in question writing: a unique feedback with fringe benefits. Am J Physiol. 1999 Dec;277(6 Pt 2):S84–91.

 PubMedGoogle Scholar

McLeod PJ, Snell L. Student-generated MCQs. Med Teach. 1996 Jan;18(1):23–25.

 Web of Science ®Google Scholar

Papinczak T, Peterson R, Babri AS, et al. Using student-generated questions for student-centred assessment. Assess Eval Higher Educ. 2012 Jun;37(4):439–452.

 Web of Science ®Google Scholar

Rick Stone M, Kinney M, Chatterton C, et al. A crowdsourced system for creating practice questions in a clinical presentation medical curriculum. Med.Sci.Educ 2017 Dec;27(4):685–692.

 Google Scholar

Walsh JL, Harris BHL, Denny P, et al. Formative student-authored question bank: perceptions, question quality and association with summative performance. Postgrad Med J. 2018 Feb;94(1108):97–103.

 PubMed Web of Science ®Google Scholar

Jobs A, Twesten C, Göbel A, et al. Question-writing as a learning tool for students – outcomes from curricular exams. BMC Med Educ. 2013 Jun;13(1):89.

 PubMedGoogle Scholar

Olde Bekkink M, Donders ARTR, Kooloos JG, et al. Uncovering students’ misconceptions by assessment of their written questions. BMC Med Educ. 2016 Aug;16(1):221.

 PubMedGoogle Scholar