RESEARCH ARTICLE: Evaluation of faculty inter-variability OSCE grade scoring on overall student performance in a laboratory course
Keywords:Faculty, Inter-variability, Objective Structured Clinical Examination (OSCE), Performance
Introduction: Objective structured clinical examinations (OSCEs) are considered the gold standard for evaluating pharmacy students’ clinical skills due to their reliability and validity.
Aim: The purpose of this study was to determine whether faculty inter-variability in OSCE grading had a significant impact on a student’s overall performance.
Methods: A retrospective analysis was conducted using data from two cohorts of third-year pharmacy students. Descriptive statistics, simple linear regression, and multivariate linear regression analyses were conducted.
Results: There were 120 students that participated in the OSCE with a mean score of 66.7%. Higher scores in the Integrated Therapeutics (IT) 2 lecture series and the IT 2 lab course corresponded to better OSCE scores. Out of 17 evaluators, six were found to rate students significantly lower and one was found to rate students significantly higher in comparison to a reference evaluator who evaluated students closest to the overall mean.
Conclusion: It is likely that standardised grading, and possibly additional training, may be needed to ensure a fair and appropriate evaluation of OSCE performance.
Ali, M, Pawluk, S.A., Rainkie, D.C., & Wilby, K,J. (2019) Pass-Fail decisions for borderline performers after a summative objective structured clinical examination. American Journal of Pharmacy Education, 83(2), Article 6849. https://doi.org/10.5688/ajpe6849
Barman, A. (2005) Critiques on the objective structured clinical examination. Annals – Academic Medicine of Singapore, 34(8), 478–482
Bartman, I., Smee, S., & Roy, M. (2013). A method for identifying extreme OSCE examiners. The clinical teacher, 10 (1), 27-31. https://doi.org/10.1111/j.1743-498X.2012.00607.x
Binh, P.D., An, P.L., Tuan, T.D., & Leppink, J.(2018). Inter-Rater reliability of a professionalism OSCE developed in family medicine training University of Medicine and Pharmacy. MedPharmRes, 2(1), 20-24. https://doi.org/10.32895/UMP.MPR.2.1.20
Brannick, M.T., Erol-Korkmaz, H.T., & Prewett, M. (2011). A systematic review of the reliability of objective structured clinical examination scores. Medical Education, 45(12), 1181-1189. https://doi.org/10.1111/j.1365-2923.2011.04075.x
Cauthon, K.A., Attridge, R.L., Urteaga, E.M., Witte, A.P. (2017) An Objective Structured Clinical Examination to Assess Pharmacy Resident Performance. Innovations in Pharmacy, 8(2), Article 5. https://doi.org/10.24926/IIP.V812.510
Corbo, M., Patel, J.P., Abdel Tawab R, Davies JG. (2006). Evaluating clinical skills of undergraduate pharmacy students using objective structured clinical examinations (OSCEs). Pharmacy Education. 6, 53–58. https://doi.org/10.1080/15602210500528372
Curtis, S., Smith, K.J., & Taylor, J. (2019) Implementation of objective structured clinical examinations (OSCE) in a doctor of pharmacy program. Currents in Pharmacy Teaching and Learning, 11, 832-837. https://doi.org/10.1016/j.cptl.2019.04.011
Duanmu, Y, Henwood, P.C., Takhar, S.S., Chan, W., Rempell, J.S., Liteplo, A.S., Koskenoja, V., Noble, V.E., & Kimberly, H.H. (2019). Correlation of OSCE performance and point-of-care ultrasound scans numbers among a cohort of emergency medicine residents. The ultrasound Journal, 11(1):3. https://doi.org/10.1186/s13089-019-0118-7
Kilminster, S., Roberts, T., Morris, P. (2007). Incorporating patients’ assessment into objective structured clinical examinations. Education for Health, 20(6)
Kirton, S.B., & Kravitz, L. (2011). Objective Structured Clinical Examinations (OSCEs) Compared With Traditional Assessment Methods. American Journal of Pharmacy Education, 75(6), Article 111. https://doi.org/10.5688/ajpe756111
Mafinejad, M.K., Rastegarpanah, M, Moosavi, F, Shirazi, M. (2017). Training and Validation of Standardized Patients for Assessing Communication and Counseling Skills of Pharmacy Students: A Pilot Study. Journal of Research in Pharmacy Practice, 6(2), 83-88. https://doi.org/10.4103/jrpp.JRPP_17_20
Reid, K, Smallwood, D, Collins, M, Sutherland, R, Dodds, A. (2016). Taking OSCE examiner training on the road: reaching the masses. Medical Education Online, 21. https://doi.org/10.3402/meo.v21.32389
Salinitri, F.D., O'Connell, M.B., Garwood, C.L., Lehr, V.T., & Abdallah, K. (2012). An objective structured clinical examination to assess problem-based learning. American Journal of Pharmacy Education, 76(3), 44. https://doi.org/10.5688/ajpe76344
Shahzad, A., Saeed, M.H.B., Paiker, S. (2017). Dental students' concerns regarding OSPE and OSCE: a qualitative feedback for process improvement. BDJ Open. 9, 3, 17009. https://doi.org/10.1038/bdjopen.2017.9
Shirwaikar, A. Objective structured clinical examination (OSCE) in pharmacy education - a trend. (2015). Pharmacy Practice, 13(4), 627. https://doi.org/10.18549/PharmPract.2015.04.627
Sobh, A.H., Austin, Z, Izham, M.I.M., Diab, M.I., Wilby, K.J. (2017). Application of a Systematic Approach to Evaluating Psychometric Properties of a Cumulative Exit-From-Degree Objective Structured Clinical Examination (OSCE). Currents in Pharmacy Teaching and Learning, 9(6), 1091–1098. https://doi.org/10.1016/j.cptl.2017.07.011
Solà, M., Pulpón, A.M., Morin, V., Sancho, R., Clèries, X., Fabrellas, N. (2017). Towards the implementation of OSCE in undergraduate nursing curriculum: A qualitative study. Nurse Education Today. 49, 163-167. https://doi.org/10.1016/j.nedt.2016.11.028. 7
Stowe, C.D., & Gardner, S.F. (2005). Real-Time Standardized Participant Grading of an Objective Structured Clinical Examination. American Journal of Pharmacy Education, 69(3) Article 41. https://doi.org/10.5688/aj690341
Sturpe, D.A. (2010). Objective structured clinical examinations in the doctor of pharmacy programs in the United states. American Journal of Pharmacy Education, 74(8) Article 148. https://doi.org/10.5688/aj7408148
Tamblyn, R.M., Klass, D.J., Schnabl, G.K., & Kopelow, M.L. (1991). Sources of unreliability and bias in standardized‐patient rating, Teaching and Learning in Medicine, 3(2), 74-85. https://doi.org/10.1080/10401339109539486
Vanka, A., Wali, O., Akondi, B.R., Vanka, S., Ravindran, S. (2018). OSCE: A New Assessment Method for Pharmaceutical Education. Indian Journal of Pharmaceutical Education and Research, 52(4s), S1-S6. https://doi.org/10.5530/ijper.52.4s.68