RESEARCH ARTICLE: Comprehensive assessment of reliability and validity for the clinical cases in simulated community pharmacy
DOI:
https://doi.org/10.46542/pe.2022.221.183190Keywords:
Community pharmacy, Objective structured clinical examinations, Reliability, Simulation, ValidityAbstract
Background: The objective structured clinical examination (OSCE) is used to measure the clinical competence of pharmacy students in a community pharmacy setup. However, the OSCE needs to be standardised to assess the clinical competence of the student accurately.
Objectives: The present study was aimed to assess the reliability and validity of two clinical cases used in the simulated community pharmacy.
Methods: OSCE simulation was performed by the students with two clinical cases in a simulated community pharmacy. The reliability was measured using Cronbach’s α and Mc Donald’s ω. Exploratory factor analysis (EFA) and Confirmatory factor analysis (CFA) were used to measure the validity of the cases.
Results: Among the two cases, the first case scenario was found to have a good model fit. However, the second case scenario has a poor model fit which was determined by the CFA. The inadequate sample size and factor loading in EFA were the main reasons for poor model fit in the second case scenario.
Conclusion: The internal consistency, sample adequacy, factor loading, test for an exact fit, and fit measurements should be ensured for the clinical cases included in OSCE. This will help the academician to ensure the accurate assessment of the clinical competence of the student in a simulated community pharmacy.
References
Abdi, A.M., Meštrović, A., Demirdamar, R., & Basgut, B. (2019). Preparing competent graduates for delivering pharmaceutical care: an experience from Northern Cyprus. BMC medical education, 19(1), 442. https://doi.org/10.1186/s12909-019-1875-5
Achary, A., & Khanal D.P. (2020). Compliance on Good Pharmacy Practice (GPP) Guideline in Community Pharmacy of Western Nepal, Nepalgunj. Journal of Pharmacy Practice,3 No.2,25
Alavi, M., Visentin, D.C., Thapa, D.K., Hunt, G.E., Watson, R., & Cleary, M. (2020). Chi-square for model fit in confirmatory factor analysis. Journal of advanced nursing, 76(9), 2209-2211. https://doi.org/10.1111/jan.14399
Al-Haqan, A., Al-Taweel, D., Koshy, S., & Alghanem, S. (2021). Evolving to Objective Structured Clinical Exams (OSCE): Transitional experience in an undergraduate pharmacy program in Kuwait. Saudi pharmaceutical journal, 29(1), 104–113. https://doi.org/10.1016/j.jsps.2020.12.013
Al-Jedai, A., Qaisi, S., & Al-Meman, A. (2016). Pharmacy Practice and the Health Care System in Saudi Arabia. The Canadian journal of hospital pharmacy, 69(3), 231–237. https://doi.org/10.4212/cjhp.v69i3.1561
Almaghaslah, D., Alsayari, A., Almanasef, M., & Asiri, A. (2021). A Cross-Sectional Study on Pharmacy Students' Career Choices in the Light of Saudi Vision 2030: Will Community Pharmacy Continue to Be the Most Promising, but Least Preferred, Sector? International journal of environmental research and public health, 18(9), 4589. https://doi.org/10.3390/ijerph18094589
Andrew, F.H., Jacob, J.C. (2020). Use Omega Rather than Cronbach’s Alpha for Estimating Reliability. But…, Communication Methods and Measures, 14(1), 1-24. https://doi.org/10.1080/19312458.2020.1718629
Badro, D.A., Sacre, H., Hallit, S., Amhaz, A., & Salameh, P. (2020). Good pharmacy practice assessment among community pharmacies in Lebanon. Pharmacy practice, 18(1), 1745. https://doi.org/10.18549/PharmPract.2020.1.1745
Bartlett, M.S. (1954) A Note on the Multiplying Factors for Various χ2 Approximations. Journal of the Royal Statistical Society: Series B, 16, 296–8. https://doi.org/10.1111/j.25176161.1954.tb00174.x
Brennan, P.A., Konieczny, K., Groves, J., Parker, M., Sherman, K. P., Foulkes, J., Hills, S., & Featherstone, C. (2015). Development, validation and initial outcomes of a questionnaire to examine human factors in postgraduate surgical objective structured clinical examinations. The British journal of surgery, 102(4), 423–430. https://doi.org/10.1002/bjs.9721
Chesser, A.M., Laing, M.R., Miedzybrodzka, Z.H., Brittenden, J., & Heys, S.D. (2004). Factor analysis can be a useful standard setting tool in a high stakes OSCE assessment. Medical education, 38(8), 825–831. https://doi.org/10.1111/j.1365-2929.2004.01821.x
Costello, A.B., Osborne, J. (2005). Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from Your Analysis. Practical Assessment Research & Evaluation, 10,1-9
Dagnall, N., Denovan, A., Parker, A., Drinkwater, K., & Walsh, R. S. (2018). Confirmatory Factor Analysis of the Inventory of Personality Organization-Reality Testing Subscale. Frontiers in psychology, 9, 1116. https://doi.org/10.3389/fpsyg.2018.01116
Deng, L., & Chan, W. (2017). Testing the Difference Between Reliability Coefficients Alpha and Omega. Educational and psychological measurement, 77(2), 185–203. https://doi.org/10.1177/0013164416658325
Dizon, S., Malcolm, J.C., Rethans, J.J., & Pugh, D. (2021). Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism. BMC medical education, 21(1), 288. https://doi.org/10.1186/s12909-021-02653-4
Flora, D.B. (2020) Your Coefficient Alpha Is Probably Wrong, but Which Coefficient Omega Is Right? A Tutorial on Using R to Obtain Better Reliability Estimates. Advances in Methods and Practices in Psychological Science, 484-501. https://doi.org/10.1177/2515245920951747
Furmedge, D.S., Smith, L.J., & Sturrock, A. (2016). Developing doctors: what are the attitudes and perceptions of year 1 and 2 medical students towards a new integrated formative objective structured clinical examination?. BMC medical education, 16, 32. https://doi.org/10.1186/s12909-
Guadagnoli, E., & Velicer, W.F. (1988). Relation of sample size to the stability of component patterns. Psychological bulletin, 103(2), 265–275. https://doi.org/10.1037/0033-
Hancock, G.R., & French, B.F. (2013). Power Analysis in Structural Equation Modeling. In G.R. Hancock, & R.O. Mueller (Eds.), Structural Equation Modeling: A Second Course (2nd ed., pp. 117-159)
Htay, M., Shrivastava, S., Sahoo, S. (2019) 'Development and Validation of Medical Students’ Professionalism Assessment Scale (MSPAS) In OSCE: Simulated Patients’ rating scale', MedEdPublish, 8(2), 24. https://doi.org/10.15694/mep.2019.000094.1
Jackson, D.L. (2003). Revisiting Sample Size and Number of Parameter Estimates: Some Support for the N:q Hypothesis. Structural Equation Modeling, 10, 128-141. https://doi.org/10.1207/S15328007SEM1001_6
Kaiser, H.F. (1974). An index of factorial simplicity. Psychometrika, 39, 31–6. https://doi.org/10.1007/BF02291575
Knekta, E., Runyon, C., & Eddy, S. (2019). One Size Doesn't Fit All: Using Factor Analysis to Gather Validity Evidence When Using Surveys in Your Research. CBE life sciences education, 18(1). https://doi.org/10.1187/cbe.18-04-0064
Kline, R.B. (2005). Principles and practice of structural equation modeling (2nd ed.). New York, NY: Guilford Press
Kyriazos, T. (2018). Applied Psychometrics: Sample Size and Sample Power Considerations in Factor Analysis (EFA, CFA) and SEM in General. Psychology, 9, :2207-2230. https://doi.org/10.4236/psych.2018.98126
Lee, T., Cai, L., & MacCallum, R. (2012). Power Analysis for Tests of Structural Equation Models. In R. H. Hoyle (Ed.), Handbook of Structural Equation Modeling (pp. 181-194). New York: Guilford Press
Lim, A.S., Lee, S., Karunaratne, N., & Caliph, S. (2020). Pharmacy Students' Perceptions and Performance on the Use of an Online Virtual Experience Tool for Practicing Objective Structured Clinical Examinations. American journal of pharmaceutical education, 84(11), 7920. https://doi.org/10.5688/ajpe7920
Liou, C.Y., Musicus, B.R. (2008). "Cross Entropy Approximation of Structured Gaussian Covariance Matrices". IEEE Transactions on Signal Processing, 56 (7), 3362–3367. https://doi.org/10.1109/TSP.2008.917878
Majumder, M., Kumar, A., Krishnamurthy, K., Ojeh, N., Adams, O.P., & Sa, B. (2019). An evaluative study of objective structured clinical examination (OSCE): students and examiners perspectives. Advances in medical education and practice, 10, 387–397. https://doi.org/10.2147/AMEP.S197275
Paul F.Wimmers., & Guido F.Schauer. (2017). Validating OSCE Performance: The Impact of General Intelligence. Health Professions Education, 3(2), 79-84. https://doi.org/10.1016/j.hpe.2016.12.002
Peeters, M.J., Cor, M.K., Petite, S.E., & Schroeder, M.N. (2021). Validation evidence using generalizability theory for an objective structured clinical examination. Innovations in pharmacy, 12(1). https://doi.org/10.24926/iip.v12i1.2110
Peterson, R.A. (2000). A Meta-Analysis of Variance Accounted for and Factor Loadings in Exploratory Factor Analysis. Marketing Letters, 11, 261–275. https://doi.org/10.1023/A:1008191211004
Piumatti, G., Cerutti, B., & Perron, N.J. (2021). Assessing communication skills during OSCE: need for integrated psychometric approaches. BMC medical education, 21(1), 106. https://doi.org/10.1186/s12909-021-02552-8
Rasheed, M.K., Alqasoumi, A., Hasan, S.S., & Babar, Z.U. (2020). The community pharmacy practice change towards patient-centered care in Saudi Arabia: a qualitative perspective. Journal of pharmaceutical policy and practice, 13, 59. https://doi.org/10.1186/s40545-020-00267-7
Ravinder, B., Saraswathi, A.B. (2020). Literature Review Of Cronbach alpha coefficient (Α) And Mcdonald's Omega Coefficient (Ω). European Journal of Molecular & Clinical Medicine, 7(6), 2943-2949
Said Said Elshama. (2021). How to Design and Apply an Objective Structured Clinical Examination (OSCE) in Medical Education? Iberoamerican Journal of Medicine, 1, 51-55. https://doi.org/10.53986/ibjm.2021.0009
Savage, A., Minshew, L.M., Anksorus, H.N., & McLaughlin, J. E. (2021). Remote OSCE Experience: What First Year Pharmacy Students Liked, Learned, and Suggested for Future Implementations. Pharmacy, 9(1), 62. https://doi.org/10.3390/pharmacy9010062
Schumacker, R.E., & Lomax, R.G. (2015). A Beginner’s Guide to Structural Equation Modeling (4th ed.). New York, Routledge. https://doi.org/10.4324/9781315749105
Shirwaikar, A. (2015). Objective structured clinical examination (OSCE) in pharmacy education - a trend. Pharmacy practice, 13(4), 627. https://doi.org/10.18549/PharmPract.2015.04.627
Susi, A.K., & Yosef, W. (2019) Assessment of Pharmacy Students’ Clinical Skills using Objective Structured Clinical Examination (OSCE): A Literature Review. Systematic Reviews in Pharmacy, 10(1), 55-60. https://doi.org/10.5530/srp.2019.1.9
Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). A Journey around alpha and omega to estimate internal consistency reliability. Anales de Psicología, 33(3), :755–782. https://doi.org/10.6018/analesps.33.3.268401
Wang, L.L., Watts, A.S., Anderson, R.A., & Little, T.D. (2013). Common Fallacies in Quantitative Research Methodology. In T.D. Little (Ed.), The Oxford Handbook of Quantitative Methods (pp. 718-758). https://doi.org/10.1093/oxfordhb/9780199934898.013.0031
Watkins, M.W. (2018). Exploratory Factor Analysis: A Guide to Best Practice. Journal of Black Psychology, 44(3), 219-246. https://doi.org/10.1177/0095798418771807
World Health Organization. (2011). Joint FIP/WHO Guidelines on Good Pharmacy Practice:Standards for quality of pharmacy services. Geneva
Yazbeck Karam, V., Park, Y.S., Tekian, A., & Youssef, N. (2018). Evaluating the validity evidence of an OSCE: results from a new medical school. BMC medical education, 18(1), 313. https://doi.org/10.1186/s12909-018-1421-x
