TY - JOUR
T1 - Beyond kappa: A review of interrater agreement measures
AU - Banerjee, Mousumi
AU - Capozzoli, Michelle
AU - McSweeney, Laura
AU - Sinha, Debajyoti
PY - 1999/1/1
Y1 - 1999/1/1
N2 - In 1960, Cohen introduced the kappa coefficient to measure chance‐corrected nominal scale agreement between two raters. Since then, numerous extensions and generalizations of this interrater agreement measure have been proposed in the literature. This paper reviews and critiques various approaches to the study of interrater agreement, for which the relevant data comprise either nominal or ordinal categorical ratings from multiple raters. It presents a comprehensive compilation of the main statistical approaches to this problem, descriptions and characterizations of the underlying models, and discussions of related statistical methodologies for estimation and confidence‐interval construction. The emphasis is on various practical scenarios and designs that underlie the development of these measures, and the interrelationships between them.
AB - In 1960, Cohen introduced the kappa coefficient to measure chance‐corrected nominal scale agreement between two raters. Since then, numerous extensions and generalizations of this interrater agreement measure have been proposed in the literature. This paper reviews and critiques various approaches to the study of interrater agreement, for which the relevant data comprise either nominal or ordinal categorical ratings from multiple raters. It presents a comprehensive compilation of the main statistical approaches to this problem, descriptions and characterizations of the underlying models, and discussions of related statistical methodologies for estimation and confidence‐interval construction. The emphasis is on various practical scenarios and designs that underlie the development of these measures, and the interrelationships between them.
KW - Kappa
KW - coefficient
KW - intraclass correlation
KW - log‐linear models
KW - nominal data
KW - ordinal data
UR - https://digitalcommons.fairfield.edu/mathandcomputerscience-facultypubs/60
UR - https://fairfield.primo.exlibrisgroup.com/discovery/openurl?institution=01FUNI_INST&vid=01FUNI_INST:MAIN&sid=google&auinit=M&aulast=Banerjee&atitle=Beyond%20kappa:%20A%20review%20of%20interrater%20agreement%20measures&id=doi:10.2307%2F3315487&title=The%20Canadian%20journal%20of%20statistics%20%5Belectronic%20resource%5D%20%3D%20La%20revue%20canadienne%20de%20statistique.&volume=27&issue=1&date=1999&spage=3&issn=0319-5724
U2 - 10.2307/3315487
DO - 10.2307/3315487
M3 - Article
VL - 27
JO - The Canadian Journal of Statistics
JF - The Canadian Journal of Statistics
ER -