On 11-02-22 06:45 PM, Derek Sonderegger wrote:
> I am trying to fully understand the trade-offs between the two analyses and
> if one method is superior in the case of only two observations per random
> effect.
>> Full experimental set-up: I have 6 field plots and three of them receive a
> treatment (applied to the plot) and three serve as controls. In each plot,
> I have measurements from 2 individual plants. Since this is a desert
> ecosystem, the variance within a plot is quite large compared the the
> variance from plot to plot.
>> The 'old-school' analysis method would be to calculate plot average for each
> plot and do an ANOVA using those plot means as data and accept that my
> inference is only at the plot level. If I go with a fancy mixed model
> approach, my inference is on the level of an individual plant, and I get an
> estimate of within plot variability.
>> But the rub is this: the plot means approach gives me a fairly significant
> treatment effect (p = .014) while the mixed model approach gives me a
> non-significant result (p=.34). My feeling is that trying to estimate the
> random effect (in addition to the treatment effect) is taking a large amount
> of our statistical power in a relatively data poor problem. I did a quick
> simulation study and it appears that in the case of high within plot
> variability and a small number of observations per plot, the mixed model
> approach has substantially reduced power compared to the plot means
>>>From a scientific point of view, we actually don't mind making inferences at
> the plot level as we are primarily concerned with landscape effects of the
> treatment.
>> At this point I think that the plot means approach is the best choice, but I
> want to make sure that I'm not missing anything.
>
The plot means approach seems fine to me (see e.g. Murtaugh 2007
Ecology), but I have to admit I'm a little bit concerned by the big
difference in the p values. In principle (I think) the 'modern' mixed
model approach should converge to the method-of-moments ('old school')
result in the classical case (balanced design, nested effects only,
etc.) ... how are you getting those different p values? Are you perhaps
doing a likelihood ratio test in one case and an F test in the other?
I would also say that you must be seeing some pretty big effects to
be getting so much power from what is effectively 6 data points ...
Ben Bolker