Intraclass correlations

An alternative, to the kappa statistic, which uses an analysis of variance output to estimate rater reliability is the intraclass correlation coefficient (ICC).

For a repeated measures anova involving k raters it follows

$$\mbox{ICC} = \frac{\mbox{MS(subjects)–MS(subjects x raters)}}{\mbox{MS(subjects) + (k-1)MS(subjects x raters)}}$$

where MS is the Mean square from the repeated measures analysis of variance. It follows that the intra-class correlation (ICC), unlike the Pearson correlation, is useful for pooling paired data each having three or more observations. Einfield and Tonge (1992, p 12) prefer using the ICC to the Pearson as it is more conservative owing to that fact it "takes account of the absolute as well as the relative difference between the scores of two raters".

Examples of its use in SPSS [http://www.nyu.edu/its/statistics/Docs/intracls.html are available here] and [attachment:ICC.doc here.]

The fixed ICC correlations called sfsingle, sf random and sffixed in the above article are of form

$$\frac{\mbox{true inter-rater variance}}{\mbox{true inter-rater variance + common error in rating variance}}$$

as mentioned as a reliability correlation in the two rater case, for example, in [http://www-users.york.ac.uk/%7Emb55/talks/oxtalk.htm a paper by Martin Bland and Doug Altman.]

An overview of approaches to inter rater reliability including the ICC is given by Darroch and McCloud (1986).

• [:FAQ/iccpr: Inferiority of using a Pearson correlation compared to an ICC]

References

Darroch JN, McCloud PI (1986) Category distinguishability and observer agreement Australian Journal of Statistics 28 371-88.

Einfield, SL and Tonge, BJ (1992) Manual for the developmental hebaviour checklist (DBC)(Primary Carer version). Melbourne:School of Psychiatry, Unievrsity of new South Wales, and Centre for Developmental Psychiatry, Monash University, Clayton, Victoria.

Shrout, PE and Fleiss, JL (1979). Intraclass Correlations: Uses in Assessing Rater Reliability, Psychological Bulletin, 86 (2) 420-428. (A good primer showing how anova output can be used to compute ICCs).