From One Environment to Many: The Problem of Replicability of Statistical Inferences

Among plausible causes for replicability failure, one that has not received sufficient attention is the environment in which the research is conducted. Consisting of the population, equipment, personnel, and various conditions such as location, time, and weather, the research environment can affect treatments and outcomes, and changes in the research environment that occur when an experiment is redone can affect replicability. We examine the extent to which such changes contribute to replicability failure. Our framework is that of an initial experiment that generates the data and a follow-up experiment that is done the same way except for a change in the research environment. We assume that the initial experiment satisfies the assumptions of the two-sample <i>t</i>-statistic and that the follow-up experiment is described by a mixed model which includes environmental parameters. We derive expressions for the effect that the research environment has on power, sample size selection, <i>p</i>-values, and confidence levels. We measure the size of the environmental effect with the environmental effect ratio (EER) which is the ratio of the standard deviations of environment by treatment interaction and error. By varying EER, it is possible to determine conditions that favor replicability and those that do not.

在实验可重复性失败的诸多合理诱因中，有一个尚未得到足够关注：即研究开展时所处的环境。研究环境涵盖研究对象总体、实验设备、研究人员，以及地点、时间、天气等各类条件，其可对实验处理与研究结果产生影响；而当实验重复开展时，研究环境发生的变化同样会对可重复性造成干扰。本研究旨在探究此类环境变化对可重复性失败的影响程度。本研究的分析框架包含两类实验：一是生成原始数据的初始实验，二是除研究环境发生变化外，其余操作均与初始实验保持一致的重复实验。我们假设初始实验满足两样本t统计量（two-sample t-statistic）的假设条件，而重复实验则可通过包含环境参数的混合效应模型（mixed model）进行描述。我们推导出研究环境对检验功效、样本量选择、p值以及置信水平的影响表达式。本研究采用环境效应比（environmental effect ratio, EER）来量化环境效应的大小，该比值为环境-处理交互作用的标准差与误差标准差之比。通过调整EER的取值，可分别确定有利于与不利于实验可重复性的条件。