Comparing all pairwise comparisons in a between subjects anova (with a suggested generalization to repeated measures!)

Ramsey PH and Ramsey PP (2008) recommend using the Tukey-Kramer procedure to compare all possible group means assuming homogeneity of group variances and allowing different group sizes. It is found to give the best any-pair power if the overall F test is not significant. This procedure is computed for upto 10 means using this [attachment:mcneq+hf.xls spreadsheet.] It may also be computed using the mc or mcneq procedures found by typing mc and mcneq at a UNIX prompt. Ramsey and Ramsey (2008) further recommend using the more conservative Hayter-Fisher modification of the Tukey-Kramer procedure for maximizing any-pair power in the presence of a significant overall F value. This is also computed in the spreadsheet.

The Games-Howell approach is recommended by Field, 2005 and Howell, 2002 for all pairwise comparisons when there are also heterogeneous group variances in addition to, possibly, unequal group sizes. The Games-Howell approach, as well as all the other post-hoc procedures mentioned on this page, may be computed in SPSS using the GLM:univariate procedure which handles between subject post-hoc comparisons or by using this [attachment:gamesh.xls spreadsheet.]

A flow chart detailing issues involved in the choice of post-hoc tests for between subject designs is [attachment:flow.pdf here.] The chart suggests two alternative tests to the Tukey-Kramer mentioned above for groups with homogeneous variances but of different size. In particular, an adjustment suggested by Gabriel to the Tukey HSD test is suggested when the sample sizes are close, emabling the use of the harmonic mean which averages group sizes when they are not too dissimilar. The REGWQ test, incorporating Gabriel's adjustment, is computed by this [attachment:tuk+regw.xls spreadsheet.] Tukey's (and the REGWQ) test (see pages 399-400 of Howell (2002)) may also be used for comparing all pairwise group difference based on a single repeated measures factor. For a worked example of using Tukey's HSD with groups from repeated measures data see [http://www.uvm.edu/~dhowell/StatPages/More_Stuff/RepMeasMultComp/RepMeasMultComp.html here.]

Note (just in case you were wondering!): The above spreadsheets quote the, perhaps, more familiar t-statistic rather than the studentised range statistic, q, (which is quoted in the output of e.g. the mc and mcneq UNIX programs at CBSU) although we, equivalently, still use the studentised range statistic for testing. In fact, as Howell (2002) points out, q and t can be used interchangeably since q = $$sqrt(2)$$t.

A spreadsheet to compute Hochberg's GT2, mentioned in the above flow chart for situations where groups have homogeneous variances but are of markedly different sizes, should, hopefully, be appearing on this page in due course.

References

Field A (2005) Discovering statistics using SPSS. Second edition. Sage:London.

Howell DC (2002) Statistical methods for psychologists. Fifth edition. Wadsworth:Pacific Grove, CA.

Ramsey PH and Ramsey PP (2008) Power of pairwise comparisons in the equal variance and unequal sample size case. British Journal of Mathematical and Statistical Psychology 61(1) 115-131.