Parametric Post-Hoc Tests: A Refined Examination of Significance and Errors in Small Sample Sizes

Abstract

Introduction: Post-hoc analysis is the process of examining data after an initial statistical test conducted on more than two groups, such as ANOVA or Kruskal-Wallis tests. Aim: This study aimed to perform a comparative analysis of ten parametric post-hoc tests, used in eight different scenarios, with a focus on small sample sizes, regarding the occurrence of type I and type II errors. Methods: An analytical study was conducted in which we evaluated 10 parametric post-hoc tests, namely Duncan, Tukey HSD, Bonferroni, Fisher’s LSD, Scheffé, Dunnett, Šidák, Gabriel, Games-Howell and Tamhane T2, using a personal database. The dataset contained synthetic values based on experimental data, and a Monte Carlo-like simulation approach was used to simulate different experimental scenarios. For each set of variables, data was randomly modified based on real values, with adjustments made (either by adding or subtracting) to meet the specified criteria of interest (normality, equal/unequal variance and group size, difference/no difference between groups, with ANOVA tests p-value distant or closer to the significance threshold). Results: According to the scenario where no differences between groups were found, variance was equal, and the p-value from ANOVA tests was close to the significance threshold, both Duncan and Fisher’s LSD post-hoc tests presented type I errors in both equal and unequal group sizes. In contrast, the Dunnett test showed type I errors only in the case of equal group sizes. For the same scenario, but in cases where variance was unequal, all tests presented type I error except for Tamhane T2. Conclusion: For small sample sizes, equal variances and p-value close to the significance threshold, Tukey HSD and Bonferroni presented good control of type I errors, while Tamhane T2 was appropriate for unequal variance.