Reliability of a modified single-point rubric for assessment of integrated pharmacotherapy cases
DOI:
https://doi.org/10.46542/pe.2024.241.730738Keywords:
Assessment, Pharmacotherapy case, Pharmacy education, RubricAbstract
Background: The Pharmacists’ Patient Care Process (PPCP) offers a method to integrate pharmacotherapy cases between disciplines. However, establishing a consistent grading standard across multiple faculty and disciplines creates a challenge. The purpose of this study was to describe the reliability of a single-point rubric for assessing integrated cases among interdisciplinary graders.
Methods: A single-point rubric was developed to provide students with formative and summative feedback on existing integrated cases requiring written PPCP notes. The rubric was calibrated retrospectively on a sample of deidentified cases from two integrated pharmacotherapy courses. Following calibration, 20 submissions were evaluated by an interdisciplinary team of faculty. The following year, four new evaluators graded 12 submissions to ensure continued reliability.
Results: Fleiss' kappa was used to determine if there was an agreement between instructors’ judgement on the final grade classification of the original 20 submissions. There was almost perfect agreement between the instructors’ judgements, κ= .868. Each section of the PPCP and overall numerical score all showed significant agreement using Kendall’s W. Results were similar the following year with four new graders.
Conclusion: This study demonstrates the continued reliability of a modified single-point rubric to evaluate written cases involving multiple disciplines.
References
Barnett, S. G., Allen, S. M., Bastianelli, K. M., Chen, J. S., Dula, C. a. C., Kachlic, M. D., Goliak, K. L., Knockel, L. E., Matthews, D. E., Volino, L. R., Lasarev, M. R., & Reist, J. C. (2022). Determination of interrater reliability of a universal evaluator rubric to assess student pharmacist communication skills. American Journal of Pharmaceutical Education, 86(1), 8544. https://doi.org/10.5688/ajpe8544
Blommel, M. L., & Abate, M. A. (2007). A rubric to assess critical literature evaluation skills. American Journal of Pharmaceutical Education, 71(4), 63. https://doi.org/10.5688/aj710463
Bray, B. S., Willson, M. N., Robinson, J. D., Matsuura, G. T., Schwartz, C. R., & Weeks, D. L. (2013). A tool to assess student performance in a Clostridium difficile infection simulation scenario. American Journal of Pharmaceutical Education, 77(7), 149. https://doi.org/10.5688/ajpe777149
Chao, I. C., King, S., Gotch, C. M., & Roberts, M. R. (2021). Exploring the educational impact of using a single-point rubric through validation in interprofessional education. Journal of Allied Health, 50(4), 253–262. https://www.ncbi.nlm.nih.gov/pubmed/34845481
Fluckiger, J. (2010). Single point rubric: A tool for responsible student self-assessment. Teacher Education Faculty Publications, 76(4), 18. https://digitalcommons.unomaha.edu/cgi/viewcontent.cgi?article=1004&context=tedfacpub
Gonzalez, J. (2023, November 1). Know your terms: Holistic, analytic, and single-point rubrics. Cult of Pedagogy. https://www.cultofpedagogy.com/holistic-analytic-single-point-rubrics/
Marshall, L. L., & Nykamp, D. (2010). Active-learning assignments to integrate basic science and clinical course material. American Journal of Pharmaceutical Education, 74(7), 119. https://doi.org/10.5688/aj7407119
McKeirnan, K. C., Willson, M. N., & Bray, B. S. (2020). Striving for interrater reliability with a patient counseling assessment rubric: Lessons learned. Currents in Pharmacy Teaching and Learning, 12(6), 603–613. https://doi.org/10.1016/j.cptl.2020.01.032
Peeters, M. J., Sahloff, E. G., & Stone, G. E. (2010). A standardized rubric to evaluate student presentations. American Journal of Pharmaceutical Education, 74(9), 171. https://doi.org/10.5688/aj7409171
Ripp, P. (2019). Using the single point rubric for better assessment conversations. Pernille Ripp. https://pernillesripp.com/2019/02/24/using-the-single-point-rubric-for-better-assessment-conversations/
St. Jean, C. R., King, S., & Roberts, M. R. (2023). Validity evidence for the use of a single-point rubric to support interprofessional teaching and learning. Journal of Interprofessional Education & Practice, 32, 100631. https://doi.org/10.1016/j.xjep.2023.100631
Joint Commission of Pharmacy Practitioners. (2018, October 26). The pharmacists’ patient care process. JCPP. https://jcpp.net/patient-care-process/
Villa, K. R., Sprunger, T. L., Walton, A. M., Costello, T. J., & Isaacs, A. N. (2020). Inter-rater reliability of a clinical documentation rubric within pharmacotherapy problem-based learning courses. American Journal of Pharmaceutical Education, 84(7), ajpe7648. https://doi.org/10.5688/ajpe7648
