Development and a validation study of comprehensive prescription writing rubrics for medical students
DOI:
https://doi.org/10.46542/pe.2024.241.403417Keywords:
Drug prescription, Generalisability theory, Medical student, RubricAbstract
Background: Prescribing is a complex task for physicians, with many global reports of errors. This study evaluates a comprehensive rubric for medical student prescribing skills regarding validity and reliability.
Methods: Twenty-one third-year medical students participated in three separate prescribing exams. Two pharmacology professors rated the students' prescriptions using a rubric covering ten criteria. Messick validity framework was utilised to enhance the study’s validity. Generalisability theory (G-theory) helped determine the source of variance and the optimal number of raters and test occasions.
Results: Content validity was ensured by three experts and alignment with the Thai Medical Council criterion. The Inter-rater and test-retest reliability were acceptable. The rubric had a Cronbach's alpha 0.70 with item-test correlation, all above 0.40. G-theory indicated that 54.93% of the total variance was due to performance and 27.57% to the interaction between performance and occasions, with a minimal residual variance of 4.28%. To reach an acceptable Phi-coefficient (≥0.70), three occasions with one rater (Phi-coefficient=0.76) or two occasions with two raters (Phi-coefficient=0.72) are needed. Conversely, the Phi-coefficient was low on a single occasion.
Conclusion: The study introduces a comprehensive rubric and description of a prescription writing programme to minimise potential prescribing errors in pre-clinical years. Furthermore, more assessment opportunities enhance knowledge retention and assessment reliability.
References
Andersen, S. A. W., Nayahangan, L. J., Park, Y. S., & Konge, L. (2021). Use of generalizability theory for exploring reliability of and sources of variance in assessment of technical skills: A systematic review and meta-analysis. Academic Medicine, 96(11), 1609–1619. https://doi.org/10.1097/ACM.0000000000004150
Aronson, J. K. (2012). Balanced prescribing – Principles and challenges. British Journal of Clinical Pharmacology, 74(4), 566–572. https://doi.org/10.1111/j.1365-2125.2012.04413.x
Baranski, B., Bolt, J., Albers, L., Siddiqui, R., Bell, A., & Semchuk, W. (2017). Development of a documentation rubric and assessment of pharmacists’ competency for documentation in the patient health record. The Canadian Journal of Hospital Pharmacy, 70(6). https://doi.org/10.4212/cjhp.v70i6.1710
Bloch, R., & Norman, G. (2012). Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68. Medical Teacher, 34(11), 960–992. https://doi.org/10.3109/0142159X.2012.703791
Brennan, R. L. (2010). Generalizability theory and classical test theory. Applied Measurement in Education, 24(1), 1–21. https://doi.org/10.1080/08957347.2011.532417
Briesch, A. M., Swaminathan, H., Welsh, M., & Chafouleas, S. M. (2014). Generalizability theory: A practical guide to study design, implementation, and interpretation. Journal of School Psychology, 52(1), 13–35. https://doi.org/10.1016/j.jsp.2013.11.008
Brinkman, D. J., Tichelaar, J., Graaf, S., Otten, R. H. J., Richir, M. C., & van Agtmael, M. A. (2018). Do final-year medical students have sufficient prescribing competencies? A systematic literature review. British Journal of Clinical Pharmacology, 84(4), 615–635. https://doi.org/10.1111/bcp.13491
Cassidy, C. E., Boulos, L., McConnell, E., Barber, B., Delahunty-Pike, A., Bishop, A., Fatima, N., Higgins, A., Churchill, M., Lively, A., MacPhee, S. P., Misener, R. M., Sarty, R., Wells, R., & Curran, J. A. (2023). E-prescribing and medication safety in community settings: A rapid scoping review. Exploratory Research in Clinical and Social Pharmacy, 12, 100365. https://doi.org/10.1016/j.rcsop.2023.100365
Donnon, T., McIlwrick, J., & Woloschuk, W. (2013). Investigating the reliability and validity of self and peer assessment to measure medical students’ professional competencies. Creative Education, 04(06), 23–28. https://doi.org/10.4236/ce.2013.46A005
Fanikos, J., Jenkins, K. L., Piazza, G., Connors, J., & Goldhaber, S. Z. (2014). Medication use evaluation: Pharmacist rubric for performance improvement. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy, 34(S1). https://doi.org/10.1002/phar.1506
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
Gautam, P. (2013). Minimizing medication errors: Moving attention from individual to system. Journal of Anaesthesiology Clinical Pharmacology, 29(3), 293. https://doi.org/10.4103/0970-9185.117037
George, D., & Mallery, P. (2019). IBM SPSS statistics 26 step by step: A Simple Guide and Reference (16th ed.). Routledge. https://doi.org/10.4324/9780429056765
Gupta, M., Tikoo, D., & Pal, S. (2020). Assessment of Prescription writing skills and impact of an educational intervention on safe prescribing among the first-year postgraduate medical students of tertiary care hospital. AMEI’s Current Trends in Diagnosis & Treatment, 4(1), 51–59. https://doi.org/10.5005/jp-journals-10055-0087
Hamstra, S. J., & Yamazaki, K. (2021). A validity framework for effective analysis and interpretation of milestones data. Journal of Graduate Medical Education, 13(2s), 75–80. https://doi.org/10.4300/JGME-D-20-01039.1
Heaton, A., Webb, D. J., & Maxwell, S. R. J. (2008). Undergraduate preparation for prescribing: The views of 2413 UK medical students and recent graduates. British Journal of Clinical Pharmacology, 66(1), 128–134. https://doi.org/10.1111/j.1365-2125.2008.03197.x
Hill, J., Ogle, K., Gottlieb, M., Santen, S. A., & Artino, A. R. (2022). Educator’s blueprint: A how‐to guide for collecting validity evidence in survey‐ based research. AEM Education and Training, 6(6). https://doi.org/10.1002/aet2.10835
James, H., Tayem, Y. I. Y., Al Khaja, K. A. J., Veeramuthu, S., & Sequeira, R. P. (2016). Prescription writing in small groups as a clinical pharmacology educational intervention: Perceptions of preclerkship medical students. The Journal of Clinical Pharmacology, 56(8), 1028–1034. https://doi.org/10.1002/jcph.692
Keijsers, C. J. P. W., Leendertse, A. J., Faber, A., Brouwers, J. R. B. J., de Wildt, D. J., & Jansen, P. A. F. (2015). Pharmacists’ and general practitioners’ pharmacology knowledge and pharmacotherapy skills. The Journal of Clinical Pharmacology, 55(8), 936–943. https://doi.org/10.1002/jcph.500
Khaja, K. A. J. Al, Handu, S. S., James, H., Mathur, V. S., & Sequeira, R. P. (2005). Assessing prescription writing skills of pre-clerkship medical students in a problem-based learning curriculum. International Journal of Clinical Pharmacology and Therapeutics, 43(09), 429–435. https://doi.org/10.5414/CPP43429
Khodi, A. (2021). The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution. Language Testing in Asia, 11(1), 30. https://doi.org/10.1186/s40468-021-00134-5
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Linton, K. D., & Murdoch-Eaton, D. (2020). Twelve tips for facilitating medical students prescribing learning on clinical placement. Medical Teacher, 42(10), 1134–1139. https://doi.org/10.1080/0142159X.2020.1726309
Loskutova, N. Y., Lutgen, C., Smail, C., Staton, E. W., Nichols, S. D., & Pinckney, R. G. (2022). Stimulant prescribing error assessment rubric development. Journal of Patient Safety, 18(1), e282–e289. https://doi.org/10.1097/PTS.0000000000000775
Malau-Aduli, B. S., Hays, R. B., D’Souza, K., Saad, S. L., Rienits, H., Celenza, A., & Murphy, R. (2023). Twelve tips for improving the quality of assessor judgements in senior medical student clinical assessments. Medical Teacher, 45(11), 1228–1232. https://doi.org/10.1080/0142159X.2023.2216364
Martínez-Domínguez, J., Sierra-Martínez, O., Galindo-Fraga, A., Trejo-Mejía, J. A., Sánchez-Mendiola, M., Ochoa-Hein, E., Vázquez-Rivera, M., Gutiérrez-Cirlos, C., Naveja, J., & Martínez-González, A. (2022). Antibiotic prescription errors: the relationship with clinical competence in junior medical residents. BMC Medical Education, 22(1), 456. https://doi.org/10.1186/s12909-022-03499-0
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Mokrzecki, S. M., Mallett, A., Sen Gupta, T., Perks, S., & Pain, T. (2023). Do educational interventions improve prescribing skills of medical students compared to no additional learning? A systematic review. Medical Education Online, 28(1). https://doi.org/10.1080/10872981.2023.2259166
Odukoya, O. K., Stone, J. A., & Chui, M. A. (2014). E-prescribing errors in community pharmacies: Exploring consequences and contributing factors. International Journal of Medical Informatics, 83(6), 427–437. https://doi.org/10.1016/j.ijmedinf.2014.02.004
Peeters, M. J., Cor, M. K., Petite, S. E., & Schroeder, M. N. (2021). Validation evidence using generalizability theory for an objective structured clinical examination. INNOVATIONS in Pharmacy, 12(1), 15. https://doi.org/10.24926/iip.v12i1.2110
Ross, S., & Loke, Y. K. (2009). Do educational interventions improve prescribing by medical students and junior doctors? A systematic review. British Journal of Clinical Pharmacology, 67(6), 662–670. https://doi.org/10.1111/j.1365-2125.2009.03395.x
Rothwell, C., Burford, B., Morrison, J., Morrow, G., Allen, M., Davies, C., Baldauf, B., Spencer, J., Johnson, N., Peile, E., & Illing, J. (2012). Junior doctors prescribing: Enhancing their learning in practice. British Journal of Clinical Pharmacology, 73(2), 194–202. https://doi.org/10.1111/j.1365-2125.2011.04061.x
Ryan, C., Ross, S., Davey, P., Duncan, E. M., Francis, J. J., Fielding, S., Johnston, M., Ker, J., Lee, A. J., MacLeod, M. J., Maxwell, S., McKay, G. A., McLay, J. S., Webb, D. J., & Bond, C. (2014). Prevalence and causes of prescribing errors: The Prescribing Outcomes for Trainee Doctors Engaged in Clinical Training (PROTECT) Study. PLoS ONE, 9(1), e79802. https://doi.org/10.1371/journal.pone.0079802
Salmasi, S., Khan, T. M., Hong, Y. H., Ming, L. C., & Wong, T. W. (2015). Medication errors in the Southeast Asian countries: A systematic review. PLOS ONE, 10(9), e0136545. https://doi.org/10.1371/journal.pone.0136545
Sam, A. H., Fung, C. Y., Wilson, R. K., Peleva, E., Kluth, D. C., Lupton, M., Owen, D. R., Melville, C. R., & Meeran, K. (2019). Using prescribing very short answer questions to identify sources of medication errors: A prospective study in two UK medical schools. BMJ Open, 9(7), e028863. https://doi.org/10.1136/bmjopen-2018-028863
Sequeira, R. (2015). Patient safety in medical education: Medication safety perspectives. Indian Journal of Pharmacology, 47(2), 135. https://doi.org/10.4103/0253-7613.153417
Sottiyotin, T., Uitrakul, S., Sakdiset, P., Sukkarn, B., Sangfai, T., Chuaboon, L., & Palee, P. (2023). Effective formative assessment for pharmacy students in Thailand: Lesson learns from a school of pharmacy in Thailand. BMC Medical Education, 23(1), 300. https://doi.org/10.1186/s12909-023-04232-1
Thenrajan, P., & Murugan, Pr. (2016). Impact of patient-based teaching in improving prescription writing skills of II MBBS students. International Journal of Applied and Basic Medical Research, 6(3), 174. https://doi.org/10.4103/2229-516X.186954
Wiernik, P. H. (2015). A dangerous lack of pharmacology education in medical and nursing schools: A policy statement from the American College of Clinical Pharmacology. The Journal of Clinical Pharmacology, 55(9), 953–954. https://doi.org/10.1002/jcph.539
Yaman, M. N., Zamzam, R., Mohamad, N., Besar, M. N. A., & Kamarudin, M. A. (2012). Evaluation of case write-up: Assessment of prescription writing skills of fifth year medical students at UKM medical centre. Procedia - Social and Behavioral Sciences, 60, 249–253. https://doi.org/10.1016/j.sbspro.2012.09.375
