A Generalized Anova Model for Estimating the Reliability of Categorical Judgments.

Enger, John M.; Whitney, Douglas R.

There are few existing or widely known measures of agreement applicable when data is nominal or categorical. Most such coefficients are applicable only when judges classify objects or subjects into a single category. A wider range of applications, including those where judges (1) place probabilities on subjects belonging to mutually exclusive and exhaustive nominal categories, or (2) rank order the applicability of categories to subjects, is desirable. A generalized ANOVA model is presented which allows the estimation of various reliability coefficients of interest for all classification tasks described. (Author)