Type I error in multiple comparison tests in analysis of variance

ABSTRACT. In a hypothesis test, a researcher initially fixes a type I error rate, that is, the probability of rejecting the null hypothesis given that it is true. In the case of means tests, it is important to present a type I error that is equal to the nominal pre-fixed level, such that this error remains unchanged across various scenarios, including the number of treatments, number of repetitions, and coefficient of variation. The purpose of this study is to analyse and compare the following multiple comparison tests for the control of both conditional and unconditional type I error rates, depending on a significant F-test in the analysis of variance: Tukey, Duncan, Fisher’s least significant difference, Student-Newman-Keuls (SNK), and Scheffé. As an application, we present a motivation study and develop a simulation study using the Monte Carlo method for a total of 64 scenarios. In each simulated scenario, we estimate the comparison-wise and experiment-wise error rates, conditional and unconditional on a significant result of the overall F-test of analysis of variance for each of the five multiple comparison tests evaluated. The results indicate that the application of the means tests based only on the significance of the F-test should be considered when determining the error rates, as this can change them. In addition, we find that Fisher’s test controls for the comparison-wise error rate, the Tukey and SNK tests control for the experiment-wise error rate, and the Duncan and Fisher tests control for the conditional experiment-wise error rate. Scheffé’s test does not control for any of the error rates considered.

摘要。在假设检验中，研究者首先会设定第一类错误率（type I error rate），即原假设（null hypothesis）为真时拒绝原假设的概率。对于均值检验（means tests）而言，确保第一类错误率等于名义预设水平至关重要，如此可令该错误率在多种实验情形下保持稳定，涵盖处理组数、重复次数与变异系数（coefficient of variation）等维度。本研究旨在针对基于方差分析（analysis of variance）显著F检验的多重比较检验，分析并比较其对条件和非条件第一类错误率的控制效果，涉及的检验方法包括图基检验（Tukey）、邓肯检验（Duncan）、费希尔最小显著差异法（Fisher’s least significant difference）、学生-纽曼-克鲁斯尔检验（Student-Newman-Keuls，简称SNK）以及谢弗检验（Scheffé）。作为应用示例，我们开展了一项动机研究，并通过蒙特卡洛方法（Monte Carlo method）构建了共计64种场景的模拟研究。在每个模拟场景中，针对评估的五种多重比较检验，我们分别估计了两两比较错误率（comparison-wise error rate）、试验整体错误率（experiment-wise error rate），以及基于方差分析整体F检验显著结果的条件和非条件错误率。研究结果表明，在确定错误率时，应考虑仅基于F检验显著性的均值检验应用，因为这一操作可能会改变错误率。此外，我们发现费希尔检验可控制两两比较错误率，图基检验与SNK检验可控制试验整体错误率，而邓肯检验与费希尔检验可控制条件试验整体错误率。谢弗检验则无法对所考虑的任意一种错误率实现控制。