Second-generation p-values: Improved rigor, reproducibility, & transparency in statistical analyses

Verifying that a statistically significant result is scientifically meaningful is not only good scientific practice, it is a natural way to control the Type I error rate. Here we introduce a novel extension of the p-value—a second-generation p-value (pδ)–that formally accounts for scientific relevance and leverages this natural Type I Error control. The approach relies on a pre-specified interval null hypothesis that represents the collection of effect sizes that are scientifically uninteresting or are practically null. The second-generation p-value is the proportion of data-supported hypotheses that are also null hypotheses. As such, second-generation p-values indicate when the data are compatible with null hypotheses (pδ = 1), or with alternative hypotheses (pδ = 0), or when the data are inconclusive (0 pδ p-values provide a proper scientific adjustment for multiple comparisons and reduce false discovery rates. This is an advance for environments rich in data, where traditional p-value adjustments are needlessly punitive. Second-generation p-values promote transparency, rigor and reproducibility of scientific results by a priori specifying which candidate hypotheses are practically meaningful and by providing a more reliable statistical summary of when the data are compatible with alternative or null hypotheses.

验证一项具备统计学显著性的结果是否具有科学意义，不仅是合规的科研实践规范，亦是控制一类错误率（Type I error rate）的自然路径。本文提出了p值（p-value）的一种全新扩展形式——第二代p值（pδ），该指标可正式纳入科学相关性考量，并依托上述天然的一类错误控制机制。该方法基于预先指定的区间零假设，该假设涵盖了所有科学上无意义或实际可视为无效的效应量（effect size）集合。第二代p值指数据支持的假设中同时属于零假设的比例。据此，第二代p值可用于判断数据与零假设相容（pδ=1）、与备择假设相容（pδ=0），或是数据尚无定论（0 < pδ < 1）。第二代p值可对多重比较（multiple comparisons）实施恰当的科学校正，并降低错误发现率（false discovery rates）。这一改进适配了数据富集的科研场景，在此类场景中传统的p值校正手段往往显得过于严苛。第二代p值通过预先指定哪些候选假设具备实际科学意义，并为判断数据与备择假设或零假设相容的情况提供更可靠的统计总结，从而提升科研结果的透明度、严谨性与可重复性。