Item-analysis of the multiple-choice questions used in the formative assessment of introductory posting examination in Medicine and Surgery at a medical university in Southern Nigeria.
Main Article Content
Abstract
Introduction: The aim of this study was to undertake quality assurance as post-examination analysis of multiple-choice questions (MCQs) used in formative assessment.
Materials & Methods: Classical Test theory (CTT) and Item Response Theory (IRT) of 500 items (100 keys & 400 distractors) in single-best answer MCQs (A-type) in introductory medicine (IM) and introductory surgery (IS) from 62 medical students was done post-examination. Anonymised answer-scripts had item responses made binary as 0 and 1 and analysed using Microsoft Excel
spreadsheet&JMetrik psychometric software to determine difficulty index, discrimination index, distractor efficiency and Cronbach's alpha coefficient.
Results: The mean score in IM was 60.83 ± 9.48 (95% C.I. 58.42 – 63.24)] Fifty-four students (87.10%) [95% C.I. 76.15 – 94.26] passed and 8 (12.9%0 [95% C.I 5.74 - 23.85] failed. Thirteen (20.7%) attained a score of 70 and above. The mean score in IS was 63.5 ± 7.1 (95% C.I 61.70 – 65.32). Sixty (96.77%) [95% C.I 88.83 – 99.61] passed, and 2 (3.23%) [95% C.I 0.39 – 11.17] failed. Twelve (19.3%) scored 70 and above. Difficulty index (DIF-I) of keys set at <0.3 (too hard) was 10% in IM & 14% in IS. DIF-I > 0.8 (too easy) was 22% in IM and 40% in IS respectively. Discrimination index (DI) of keys set <0.1 (poor) was 44% in IM & 48% in IS and >0.3(good) was 10% in IM & 2% in IS respectively. The Cronbach's alpha coefficient was 0.62 in IM & 0.45 in IS respectively. Nonfunctioning and ineffective distractors (NFD) with a score of zero (0) was 28.8% in IM & 45.2% in IS respectively.
Conclusion: Item analysis in this study showed many easy questions with poor discrimination, low reliability index and poor distractor efficiency. We recommend post-examination item-analysis as part of quality assurance matrix after formative assessment.
Downloads
Article Details
Issue
Section
All articles published in the journal are licensed under a Creative Commons Attribution (CC BY) license(i.e CC Attribution-NonCommercial-ShareAlike 4.0), allowing others to share, distribute, and build upon the work, provided the original author(s) and source are properly cited. Authors retain the copyright of their work.
How to Cite
References
1.Guidelines on minimum standards for undergraduate medical & dental education in Nigeria. Medical & Dental Council of Nigeria.
Accessed 23rd November 2024
2.Al-Wardy NM. Assessment methods in undergraduate medical education. Sultan Qaboos Univ Med J. Aug 2010;10(2):203-9.
3.Bhat SK, Prasad KHL. Item analysis and optimizing multiple-choice questions for a viable question bank in ophthalmology: A cross-
sectional study. Indian Journal of Ophthalmology. 2021;69(2)
4.Wood TJ, Cunnington JP, Norman GR. Assessing the measurement properties of a clinical reasoning exercise. Teach Learn Med. Fall 2000; 12(4): 196-200. doi:10.1207/S15328015TLM1204_6
5.Coughlin PA, Featherstone CR. How to Write a High Quality Multiple Choice Question (MCQ): A Guide for Clinicians. European Journal of
Vascular and Endovascular Surgery. 2017;54(5):654-658. doi:10.1016/j.ejvs.2017.07.012
6.Ventista O. Time to increase the quality of the multiple-choice questions you use! 2017; Accessed 23rd November 2024
7.Ndu IK, Ekwochi U, Di Osuorah C, et al. Negative Marking and the Student Physician--A Descriptive Study of Nigerian Medical Schools. J Med
Educ Curric Dev. Jan - Dec 2016;3doi:10.4137/jmecd.S40705
8.Lesage E, Valcke M, Sabbe E. Scoring methods for multiple choice assessment in higher education Is it still a matter of number right
scoring or negative marking? Studies in Educational Evaluation. 2013/09/01/ 2013;39(3):188-193.
9.Knowles M. The modern practice of adult education: From pedagogy to andragogy. Englewood Cliffs, NJ. Cambridge Adult Education; 1980.
10.Roine I, Molina Y, Caneo M. A psychometric appraisal of the dundee ready education environment measure in a medical school in Chile. Educ Health (Abingdon). Sep - Dec 2018; 31(3): 148 - 154. doi: 10. 4103/efh.EfH_17_18
11. Tavakol M, Dennick R. Post - examination analysis of objective tests. Medical Teacher. 2011/06/01 2011;33(6):447-458. doi:10.
3109/0142159X.2011.564682
12. Tavakol M, Dennick R. Post - examination interpretation of objective test data: Monitoring and improving the quality of high-stakes examinations: AMEE Guide No. 66. Medical Teacher. 2012/03/01 2012;34(3):e161-e175. doi:10. 3109/0142159X.2012.651178
13.Gasmalla HEE, Mohamed Tahir MEM. A-Type MCQs. In: Gasmalla HEE, Ibrahim AAM, Wadi MM, Taha MH, eds. Written Assessment in
Medical Education. Springer International Publishing; 2023:73-89.
14.Meyer JP. JMetrik (Version 4.1) Computer Software. 2014; Retrieved from.
15.Brown JD. Testing In Language Programs: A Comprehensive Guide To English Language Assessment. Upper Saddle River, NJ. McGraw-Hill College; 2005.
16.Etobro BA, Taiwo O, Alawaye M. Pragmatic Perspective of Item Analysis Using Microsoft Office Excel Data Analysis Tools. Ilorin Journal of Education. 04/08 2024;44(2):279 - 292.
17.Reynolds CR, Altmann RA, Allen DN. Item Analysis: Methods for Fitting the Right Items to the Right Test. In: Reynolds CR, Altmann RA,
Allen DN, eds. Mastering Modern Psychological Testing: Theory and Methods. Springer International Publishing; 2021:263-289.
18. Understanding Item Analyses. University Of Washington; Accessed 23rd November 2024
19. Tavakol M, Dennick R. Post - examination analysis of objective tests. Med Teach. 2011;33(6):447- 58. doi:10.3109/0142159X.2011.564682
20.Lord FM. The relation of the reliability of multiple-choice tests to the distribution of item difficulties. Psychometrika. 1952;17(2):181-
194. doi:10.1007/BF02288781
21.Ebel RL, Frisbie DA. Essentials of Educational Measurement. Prentice-Hall; 1986.
22.Testa S, Toscano A, Rosato R. Distractor Efficiency in an Item Pool for a Statistics Classroom Exam: Assessing Its Relation With Item
Cognitive Level Classified According to Blooms Taxonomy. Original Research. Frontiers in Psychology. 2018-August-28 2018;9doi:10.3389/fpsyg.2018.01585
23.Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16, 297-334
24.Biggs JB, Tang CKC. Teaching for quality learning at university: what the student does. Maidenhead. McGraw-Hill; 2011.
25.Vegi VAK, Sudhakar PV, Bhimarasetty DM, et al. Multiple-choice questions in assessment: Perceptions of medical students from low-resource setting. J Educ Health Promot. 2022;11:103. doi:10.4103/jehp.jehp_621_21
26.Loh KY, Elsayed I, Nurjahan MI, Roland GS. Item Difficulty and Discrimination Index in Single Best Answer MCQ: End of Semester Examinati-
ons at Taylors Clinical School. Springer Singapore; 2018:167-171.
27.Tarrant M, Ware J, Mohammed AM. An assessment of functioning and non-functioning distractors in multiple-choice questions: a
descriptive analysis. BMC Medical Education. 2009/07/07 2009;9 (1):40. doi:10.1186/1472-6920-9-40
28.Pan Q, Jiang Z. Examining distractor qualities of pediatrics subject tests from a national assessment. Front Med (Lausanne). 2022;9:921719.
doi:10.3389/fmed.2022.921719
29.Nnodim JO. Multiple-choice testing in anatomy. Med Educ. Jul 1992;26 (4):301-9. doi:10.1111/j.1365-2923.1992.tb00173.x