Reliability, validity, and generalizability of an objective structured clinical examination (OSCE) for assessment of entry-to-practice in pharmacy

Authors

  • Lila Quero Munoz Pharmacy Examining Board of Canada, Toronto, Ont., Canada M5B 2E7
  • Carol O'Byrne Pharmacy Examining Board of Canada, Toronto, Ont., Canada M5B 2E7
  • John Pugsley Pharmacy Examining Board of Canada, Toronto, Ont., Canada M5B 2E7
  • Zubin Austin Pharmacy Examining Board of Canada, Toronto, Ont., Canada M5B 2E7, and 2 Leslie Dan Faculty of Pharmacy University of Toronto, 19 Russell Street, Toronto, Ontario, Canada M5S 1A1

Keywords:

OSCE, assessment, pharmacy education, clinical skills evaluation, performance assessment

Abstract

This paper describes the evaluation of an objective structured clinical examination (OSCE) and the assessment outcomes for reliability, validity and generalizability for the entry-to-practice context in pharmacy in Canada.  A total of 190 participants were involved: 153 entry-to-practice candidates and 37 pharmacists who were already licensed. Two balanced forms of an OSCE were developed, consisting of 26 stations (18 interactive and 8 non-interactive stations). Descriptive analysis for all data was undertaken, and detailed analysis of data from Form I of the OSCE (including generalizability and dependability studies) are reported. Based on findings of this study, conclusions were made regarding OSCEs for entry-to-practice assessment in pharmacy. A key finding of this study was that a 15-station OSCE, using one pharmacist-assessor per station, yielded consistent and dependable scores when holistic scoring was used to assess both qualifying candidates and practising pharmacists.

References

Abedi, J. (2004). The no child left behind act and English language learners: Assessment and accountability issues. Education Research, 33(1), 4–14.

Austin, Z., Croteau, D., Marinia, A., & Violato, C. (2003). Continuous professional development: The Ontario experience in professional self-regulation through quality assurance and peer review. American Journal of Pharmaceutical Education, 62(2), 56–63.

Benson, J. A. (1991). Certification and recertification: One approach to professional accountability. Annals of Internal Medicine, 114, 228–232.

Brennan, R. L. (1995). Generalizability of performance assessment. Educational Measurement: Issues and Practice, 9–12, Winter.

Campbell, C., Parboosingh, J., & Slotnick, H. B. (1999). Outcomes related to physicians practice-based learning. Journal of Continuing Education of the Health Professions, 19(4), 234–241.

Carpenter, J. L. (1995). Cost analysis of OSCEs (Review). Academic Medicine, 70, 828–833.

Colliver, J. A., & Swartz, M. H. (1997). Assessing clinical performance with standardized patients. JAMA, 278, 790–791.

Dupras, D. M., & Li, J. T. (1995). Use of an OSCE to determine clinical competence. Academic Medicine, 70,1029–1034.

Fielding, D. W., Page, G. G., Fevang, L. C., & Thomas, N. S. (1981). Competency assessment: A progress report on British Columbia’s program. American Journal of Pharmaceutical Education, 45, 178–183.

Fielding, D. W., Page, G. G., Schulzer, M., Rogers, W. T., & O’Byrne, C. C. (1992). Assuring continuing competency: Identification and validation of practice-based assessment blueprint. American Journal of Pharmaceutical Education, 56, 21–29.

Fielding, D., Page, G., Rogers, W., O’Byrne, C., Schulzer, M., Moody, K. G., & Dyer, S. (1997). Application of objective structured clinical examinations in an assessment of pharmacists’ continuing competency. American Journal of Pharmaceutical Education, 61, 117–126.

Fielding, D. W., Rogers, W. T., Tench, E., O’Byrne, C. C., Page, G. G., & Schulzer, M. (2001). Predictors of continuing competence. American Journal of Pharmaceutical Education, 65, 107–118.

Grand’Maison, P., Brailovsky, C. A., Lescop, J. J., & Rainsberry, P. C. (1997). Using standardized patients in licensing/certification examinations: A comparison of two tests in Canada. Family Medicine, 29, 27–32.

Regeher, G., MacRae, H., Reznick, R. K., & Szalay, D. (1998). Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine, 73(9), 993–997.

Robb, K. V., & Rothman, A. (1985). Assessment of clinical skills in general medical residents: Comparison of the objective structured clinical examination to a conventional oral examination. Annals of the Royal College of Physicians and Surgeons of Canada., 18, 235–238.

Singer, P. A., Robb, A., Cohen, R., Norman, G., & Turnbull, J. (1996). Performance-based assessment of clinical ethics using an objective structured clinical examination. Academic Medicine, 21, 495–498.

Sloan, D. A., Donnelly, M. B., Schwartz, R. W., & Strodel, W. E. (1995). The objective structured clinical examination: The new gold standard for evaluating postgraduate clinical performance. Annals of Surgery, 222(6), 735–742.

Solomon, D. J., Szauter, K., Rosebraugh, C. J., & Callaway, M. R. (1999). Global ratings of student performance in a standardized patient examination: Is the whole more than the sum of the parts? Advances in Health Sciences Education, 4, 1–10.

Woodburn, J., & Sutcliffe, N. (1996). The reliability, validity and evaluation of the objective structured clinical examination in podiatry (chiropody). Assessment and Evaluation in Higher Education, 21, 131–146.

Downloads

Published

08-12-2004

How to Cite

Quero Munoz, L., O’Byrne, C., Pugsley, J., & Austin, Z. (2004). Reliability, validity, and generalizability of an objective structured clinical examination (OSCE) for assessment of entry-to-practice in pharmacy. Pharmacy Education, 5(1). Retrieved from https://pharmacyeducation.fip.org/pharmacyeducation/article/view/79

Issue

Section

Research Article