RESEARCH ARTICLE: Competency assessors’ cognitive map of practice when assessing practice based encounters


  • Madhuriksha Reddy University of Otago, Dunedin, New Zealand
  • Jared Davidson University of Otago, Dunedin, New Zealand
  • Carla Dillon University of Otago, Dunedin, New Zealand
  • Kyle John Wilby Dalhousie University, Halifax NS, Canada



Assessment, Competency, Workforce


Introduction: There is growing evidence that inconsistencies exist in how competencies are conceptualised and assessed. 

Aim: This study aimed to determine the reliability of pharmacist assessors when observing practice-based encounters and to compare and contrast assessors’ cognitive map of practice with the guiding competency framework. 

Methods: This was a qualitative study with verbal protocol analysis. A total of 25 assessors were recruited to score and verbalise their assessments for three videos depicting practice-based encounters. Verbalisations were coded according to the professional competency framework. 

Results: Protocols from 24 participants were included. Interrater reliability of scoring was excellent. Greater than 75% of assessment verbalisations were focused on 3 of the 27 competencies: communicate effectively, consults with the patient, and provide patient counselling. 

Conclusion: Findings support the notion that assessment completed within practice could be largely informed by a single component of the interaction or more specifically, what ‘catches the eye’ of the assessor.

Author Biographies

Madhuriksha Reddy, University of Otago, Dunedin, New Zealand

School of Pharmacy

Jared Davidson, University of Otago, Dunedin, New Zealand

School of Pharmacy

Carla Dillon, University of Otago, Dunedin, New Zealand

School of Pharmacy

Kyle John Wilby, Dalhousie University, Halifax NS, Canada

Faculty of Health


Berendonk, C., Stalmeijer, R., Schuwirth, L.W.T. (2013). Expertise in performance assessment: assessors’ perspectives. Advances in Health Sciences Education, 18, 559-571

Cate, O.T., Chen, H.C., Hoff, R.G., Peters, H., Bok, H., van der Schaaf, M. (2015). Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Medical Teacher, 37, 983-1002

Eva K. (2018). Cognitive influences on complex performance assessment: lessons from the interplay between medicine and psychology. Journal of Applied Research in Memory and Cognition, 7, 177-188

Frank, J.R., Snell, L.S., Cate, O.T., Holmboe, E.S., Carraccio, C., Swing, S.R., Harris, P., Glasgow, N.J., Campbell, C., Dath, D., Harden, R.M., Iobst, W., Long, D.M., Mungroo, R., Richardson, D.L., Sherbino, J., Silver, I., Taber, S., Talbot, M., Harris, K.A. (2010). Competency-based medical education: theory to practice. Medical Teacher, 32, 638-645.

Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., Holmboe, E. (2014). Seeing the ‘black box’ differently: assessor cognition from three research perspectives. Medical Education, 48, 1055-1068

Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., Regehr, G. (2010). Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Academic Medicine, 85, 780-786

Ginsburg, S., van der Vleuten, C.P.M., Eva, K.W. (2017). The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Academic Medicine, 92, 1617-1621

Hanson, J.L., Resenberg, A.A., Lane, J.L. (2013). Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States. Frontiers in Psychology, 4, Article 668

Holmboe, E.S., Sherbino, J., Long, D.M., Swing, S.R., Frank, J.R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32, 676-682.

Holmboe, E.S., Ward, D.S., Reznick, R.K., Katsufrakis, P.J., Leslie, K.M., Patel, V.L., Ray, D.D., Nelson. E.A. (2011). Faculty development in assessment: the missing link in competency-based medical education. Academic Medicine, 86, 460-467.

Iobst, W.F., Sherbino, J., Cate, O.T., Richardson, D.L., Dath, D., Swing, S.R., Harris, P., Mungroo, R., Holmboe, E.S., Frank, J.R. (2010). Competency-based medical education in post-graduate medical education. Medical Teacher, 32, 651-656.

Kogan, J.R., Conforti, L.N., Bernabeo, E., Iobst, W., Holmboe, E. (2015). How faculty members experience workplace-based assessment rater training: a qualitative study. Medical Education, 49, 692-708

Nash, R.E., Chalmers, L., Stupans, I., Brown, N. (2016). Knowledge, use and perceived relevance of a profession’s competency standards; implications for pharmacy education. International Journal of Pharmacy Practice, 24, 390-402

Oudkerk Pool, A., Govaerts, M.J.B., Jaarsma, D.A.D.C., Driessen, E.W. (2018). From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio. Advances in Health Sciences Education, 23, 275-287

Pharmacy Council of New Zealand. (2015). Competence standards for the pharmacy profession January 2015. Available at:

Tavares, W., Eva, K.W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18, 291-303





Research Article