Intraclass correlations

An alternative, to the kappa statistic, which uses an analysis of variance output to estimate rater reliability is the intraclass correlation coefficient (ICC).

For a repeated measures anova involving k raters it follows

$$\mbox{ICC} = \frac{MS(subjects)–MS(subjects x raters)}{MS(subjects) + (k-1)MS(subjects x raters)}$$

where MS is the Mean square from the repeated measures analysis of variance.

Examples of its use in SPSS [http://www.nyu.edu/its/statistics/Docs/intracls.html are available.]

An overview of approaches to inter rater reliability including the ICC is given by Darroch and McCloud(1986).

Reference:

Darroch JN, McCloud PI (1986) Category distinguishability and observer agreement Australian Journal of Statistics 28 371-88.