Kappa is a
popular agreement statistic used to estimate the accuracy of two or more
observers. Kappa is a consistent estimator of accuracy only when the
assumptions and contextual factors outlined by Cohen (1960) are met. In this
presentation, the response of kappa to differing base rates of the observed
event is examined.† Methods for
interpreting the accuracy of observers are presented. The methods for
interpreting kappa will be demonstrated using observational data. A review of
software designed to analyze agreement data will be presented.†

Topics

1.
†Agreement statistics

2.
†Kappa

3.
†Base rate

4.
†Accuracy

Goals

1. †Participants will be able to determine if a data
space is mutually exclusive and††††
exhaustive.

2. †Participants will be able to calculate kappa
from a two-by-two table using available software.

3. †Participants will be able to calculate the
best estimate of the base rate of a code from a two-by-two table.

4. †Participants will be able to use the best
estimate of the base rate and an obtained kappa to interpolate the accuracy of
raters

5. †Participants will be able to justify to a
reviewer the acceptance of a kappa below the criterion value of .6, if rater accuracy
is sufficient.

Intended
Audience:

P.I.ís,
students, staff. The only prerequisite is a basic understanding of agreement
coefficients like kappa.

Speaker
Description

Cornelia
Taylor is a doctoral student in Early Childhood Special Education. She received
her masters in Child and Family Development from the University of Georgia. While at
Vanderbilt, she has worked with Dr. Paul Yoder on several projects
investigating the validity and reliability of observational measures of young
children with disabilities. Both classical and modern test theory have been
used in these projects. She is currently completing her dissertation project
comparing children with ASD to typically developing children on the probability
of correctly responding to items on a parent report vocabulary inventory
controlling for expressive and receptive vocabulary.††††††