Interpreting "Inconsistent" Statistical Testing Results

Q contains multiple different systems for computing and displaying statistical testing results. Each of these systems is doing different things and, consequently, their results can seem to be inconsistent. However, unless the user has modified default settings, in general the seeming inconsistencies are in misinterpretations of outputs rather than genuine inconsistencies.

In the first row, no letters are shown, and instead the _ symbol is shown, which indicates that no columns have been compared. This is because, by default, columns are only compared if they are in the same span. That is, in this table, the Non-workers columns are compared with each other and the Work columns are compared with each other (see also How to Specify Columns to be Compared). However, the font color, which represents the Cell comparisons, shows that in June, the percentage for If I wanted to eat fried chicken, I would go to KFC is higher among the Work segment than the Non-workers segment, as the cell comparisons do not take the spans into account.

The second row of the table more starkly illustrates the difference between the column comparisons and the cell comparisons. The font colors show us that the non-workers in general find the side orders more appealing than the workers. The column comparisons show that among the non-workers, the June score was significantly higher than the April score.

Planned comparisons (alpha) versus Cell and Column comparisons

In the second row of the table below, the only signficant cell comparisons are for May (i.e., these are the only comparisons in color). But, if you select the April cell and press , the result is shown as being Significant, as is shown below. On face value, this appears to be inconsistent.

The reason for the difference is that by default in Q, Cell and Column comparisons have multiple comparison corrections automatically applied to them, whereas planned tests do not. If you look at the output shown below, you will see that this seeming inconsistency is also described in the sentence immediately below where it is marked as significant. This difference in behavior is by design (see Multiple Comparisons (Post Hoc Testing)).

Note also that when selecting a single cell and pressing Q will show the results of a Cell comparison, whereas if you select a pair of cells Q will show a Column comparison.