RESEARCH ARTICLE: Competency assessors’ cognitive map of practice when assessing practice based encounters
DOI:
https://doi.org/10.46542/pe.2021.211.443448Keywords:
Assessment, Competency, WorkforceAbstract
Introduction: There is growing evidence that inconsistencies exist in how competencies are conceptualised and assessed.
Aim: This study aimed to determine the reliability of pharmacist assessors when observing practice-based encounters and to compare and contrast assessors’ cognitive map of practice with the guiding competency framework.
Methods: This was a qualitative study with verbal protocol analysis. A total of 25 assessors were recruited to score and verbalise their assessments for three videos depicting practice-based encounters. Verbalisations were coded according to the professional competency framework.
Results: Protocols from 24 participants were included. Interrater reliability of scoring was excellent. Greater than 75% of assessment verbalisations were focused on 3 of the 27 competencies: communicate effectively, consults with the patient, and provide patient counselling.
Conclusion: Findings support the notion that assessment completed within practice could be largely informed by a single component of the interaction or more specifically, what ‘catches the eye’ of the assessor.
References
Berendonk, C., Stalmeijer, R., Schuwirth, L.W.T. (2013). Expertise in performance assessment: assessors’ perspectives. Advances in Health Sciences Education, 18, 559-571
Cate, O.T., Chen, H.C., Hoff, R.G., Peters, H., Bok, H., van der Schaaf, M. (2015). Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Medical Teacher, 37, 983-1002
Eva K. (2018). Cognitive influences on complex performance assessment: lessons from the interplay between medicine and psychology. Journal of Applied Research in Memory and Cognition, 7, 177-188
Frank, J.R., Snell, L.S., Cate, O.T., Holmboe, E.S., Carraccio, C., Swing, S.R., Harris, P., Glasgow, N.J., Campbell, C., Dath, D., Harden, R.M., Iobst, W., Long, D.M., Mungroo, R., Richardson, D.L., Sherbino, J., Silver, I., Taber, S., Talbot, M., Harris, K.A. (2010). Competency-based medical education: theory to practice. Medical Teacher, 32, 638-645. https://doi.org/10.3109/0142159X.2010.501190
Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., Holmboe, E. (2014). Seeing the ‘black box’ differently: assessor cognition from three research perspectives. Medical Education, 48, 1055-1068
Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., Regehr, G. (2010). Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Academic Medicine, 85, 780-786
Ginsburg, S., van der Vleuten, C.P.M., Eva, K.W. (2017). The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Academic Medicine, 92, 1617-1621
Hanson, J.L., Resenberg, A.A., Lane, J.L. (2013). Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States. Frontiers in Psychology, 4, Article 668
Holmboe, E.S., Sherbino, J., Long, D.M., Swing, S.R., Frank, J.R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32, 676-682. https://doi.org/10.3109/0142159X.2010.500704
Holmboe, E.S., Ward, D.S., Reznick, R.K., Katsufrakis, P.J., Leslie, K.M., Patel, V.L., Ray, D.D., Nelson. E.A. (2011). Faculty development in assessment: the missing link in competency-based medical education. Academic Medicine, 86, 460-467. https://doi.org/10.1097/ACM.0b013e31820cb2a7
Iobst, W.F., Sherbino, J., Cate, O.T., Richardson, D.L., Dath, D., Swing, S.R., Harris, P., Mungroo, R., Holmboe, E.S., Frank, J.R. (2010). Competency-based medical education in post-graduate medical education. Medical Teacher, 32, 651-656. https://doi.org/10.3109/0142159X.2010.500709
Kogan, J.R., Conforti, L.N., Bernabeo, E., Iobst, W., Holmboe, E. (2015). How faculty members experience workplace-based assessment rater training: a qualitative study. Medical Education, 49, 692-708
Nash, R.E., Chalmers, L., Stupans, I., Brown, N. (2016). Knowledge, use and perceived relevance of a profession’s competency standards; implications for pharmacy education. International Journal of Pharmacy Practice, 24, 390-402
Oudkerk Pool, A., Govaerts, M.J.B., Jaarsma, D.A.D.C., Driessen, E.W. (2018). From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio. Advances in Health Sciences Education, 23, 275-287
Pharmacy Council of New Zealand. (2015). Competence standards for the pharmacy profession January 2015. Available at: https://www.pharmacycouncil.org.nz/dnn_uploads/Documents/standardsguidelines/CompStds2015Web.pdf?ver=2017-02-20-104344-177
Tavares, W., Eva, K.W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18, 291-303
