Beyond p-values: Utilizing Multiple Estimates to Evaluate Evidence

Null hypothesis significance testing (NSHT) is cited as a threat to validity and reproducibility. While many individuals suggest we focus on altering the p-value at which we deem an effect significant, we believe this suggestion is short-sighted. Alternative procedures (i.e., Bayesian analyses and Observation Oriented Modeling: OOM) can be more powerful and meaningful to our discipline. However, these methodologies are less frequently utilized and are rarely discussed in combination with NHST. Herein, we discuss three methodologies (NHST, Bayesian Model comparison, and OOM), then compare the possible interpretations of three analyses (ANOVA, Bayes Factor, and an Ordinal Pattern Analysis) in various data environments using a frequentist simulation study. We found that changing significance thresholds had little effect on conclusions. Further, we suggest that evaluating multiple estimates as evidence of an effect allows for more robust and nuanced interpretations of results and implies the need to redefine evidentiary value and reporting practices.

零假设显著性检验（Null hypothesis significance testing，简称NSHT）被视为对研究有效性和可重复性的威胁。尽管许多学者建议我们关注改变我们判定效应显著的p值，但我们认为这一建议具有短视之嫌。替代性程序（例如贝叶斯分析和面向观察建模：OOM）可能对我们的学科更具影响力和意义。然而，这些方法的使用频率较低，并且很少与NHST结合进行讨论。在本研究中，我们讨论了三种方法（NHST、贝叶斯模型比较和OOM），然后利用频率主义模拟研究，对比了在多种数据环境下三种分析（方差分析、贝叶斯因子和有序模式分析）的可能解释。我们发现，改变显著性阈值对结论的影响微乎其微。此外，我们提出，将多个估计值作为效应的证据，可以使得对结果的解释更加稳健和细腻，这也暗示了我们需要重新定义证据价值和报告实践。