re: st: Checking reliability of a measurement device

Hi Statalisters,
a Dentistry PhD student did some measurements on 12
teeth with varying conditions and he asked me how
could he show that the device used for the
measurements is reliable. More specifically each one
of the 12 teeth has been measured by this device by 2
raters (a and b) X 2 time points (week 1 and week 2) X
6 relative positions = 24 measurements. The goal is
to show that the discrepancies among measurements are
not statistically significant.

That is interesting.

I'm not sure whether the student is interested in reliability (variance
component estimation?) or the student wishes to compare performance of
a new method to an accepted method (mean comparisons with certain
assumptions about the variance between methods?).

If the latter, maybe you want a planned one-sided comparison against a
"gold standard". You are only interested in whether your new method
mean is higher (or lower) than an accepted method's mean performance.
You might look at Hsu (1996) Multiple Comparisons: Theory and methods
(Chapman and Hall).

If the former, don't your hypotheses center around the rater means and
or variances? There is a lot of literature on rater reliability. Seems
like the design has a lot of information within two raters, but very
little between rater information from the population of possible
raters.

In your model, level of variation to think about is the person. Is each
tooth from a separate individual? Teeth are correlated with person if
not.

My first thought was to use a 2-level model
(measurements nested within tooth) and test the rater,
time, position effects and maybe their interractions.
Something like
xi:xtreg length i.rater i.time i.position,i(tooth)
assuming that a random intercept structure is adequate
for this experiment.
The problem is that this is the first time I deal with
a problem where the goal is to "prove" that some
factors have no significant effects, thus I am not
very confident with the aforementioned method.
Any thoughts?
Nikos Pantazis
Biostatistician