Halos and Horns in the Assessment of Undergraduate Medical Students: A Consistency-Based Approach

Margaret MACDOUGALL
Simon C. RILEY
Helen S. CAMERON
Brian MCKINSTRY

Keywords

halo effect, horn effect, intra-class correlation coefficient, second marker, supervisor bias, undergraduate assessment, Zegers-ten Berge general association coefficient

Abstract

The authors introduce a consistency-based approach to detecting examiner bias. On comparing intra-class correlation coefficients on transformed data for supervisor continuous performance and report marks (ICC1*) with those for supervisor continuous performance and second marker report marks (ICC2*), a highly significant difference was obtained for both the entire cohort (ICC1* = .72, ICC2* = .30, F = 2.47, p < .0005 (N = 1085)) and the subgroup with high supervisor ratings for continuous performance (ICC1* = .62, ICC2* = .24, F = 1.97, p < .0005 (n = 952)). A strong halo effect was detected and preliminary evidence was obtained for the presence of a strong horn effect for students with lower scores, thus providing a basis for future research.