An Excel file with 5 worksheets listing the sources for the datasets analysed in the paper: Medical, Particle, Nuclear, Interlab, Constants. from Not Normal: the uncertainties of scientific measurements

Judging the significance and reproducibility of quantitative research requires a good understanding of relevant uncertainties, but it is often unclear how well these have been evaluated and what they imply. Reported scientific uncertainties were studied by analysing 41 000 measurements of 3200 quantities from medicine, nuclear and particle physics, and interlaboratory comparisons ranging from chemistry to toxicology. Outliers are common, with 5 σ disagreements up to five orders of magnitude more frequent than naively expected. Uncertainty-normalized differences between multiple measurements of the same quantity are consistent with heavy-tailed Student's t-distributions that are often almost Cauchy, far from a Gaussian Normal bell curve. Medical research uncertainties are generally as well evaluated as those in physics, but physics uncertainty improves more rapidly, making feasible simple significance criteria such as the 5 σ discovery convention in particle physics. Contributions to measurement uncertainty from mistakes and unknown problems are not completely unpredictable. Such errors appear to have power-law distributions consistent with how designed complex systems fail, and how unknown systematic errors are constrained by researchers. This better understanding may help improve analysis and meta-analysis of data, and help scientists and the public have more realistic expectations of what scientific results imply.

评判定量研究的显著性与可复现性，需要充分理解其相关不确定度，但学界往往难以明确这些不确定度的评估程度与实际内涵。本研究针对已报道的科学不确定度展开分析：共梳理了来自医学、核物理与粒子物理领域的3200个物理量的41000次测量数据，以及覆盖化学、毒理学等领域的实验室间比对结果。异常值（outlier）十分常见，5σ偏差的出现频率远超直觉预期，可达预期值的五个数量级以上。对同一物理量的多次测量结果经不确定度归一化后的差值，符合重尾学生t分布（Student's t-distribution），这类分布往往近似柯西分布（Cauchy distribution），与高斯正态钟形曲线相去甚远。医学研究中的不确定度评估质量总体上与物理学领域相当，但物理学领域的不确定度评估质量提升更快，这使得粒子物理领域采用的5σ发现准则这类简易显著性判定标准具备了可行性。由失误与未知问题引发的测量不确定度贡献项并非完全不可预测。这类误差的分布符合幂律分布（power-law distribution），其特征与人工设计的复杂系统失效模式，以及研究者对未知系统误差的约束规律相一致。这一更为深入的认知，有望助力数据分析与元分析（meta-analysis）方法的优化，并帮助科研人员与公众更理性地认知科学研究结果的实际内涵。