Data from: Not normal: the uncertainties of scientific measurements

Judging the significance and reproducibility of quantitative research requires a good understanding of relevant uncertainties, but it is often unclear how accurately these are estimated and what they imply. Reported scientific uncertainties were studied by analysing 41000 measurements of 3200 quantities from medicine, nuclear and particle physics, and inter-laboratory comparisons ranging from chemistry to toxicology. Outliers are common, with 5σ disagreements up to five orders of magnitude more frequent than naively expected. Uncertainty-normalized differences between multiple measurements of the same quantity follow heavy-tailed Student- t distributions that are often almost Cauchy, far from a Gaussian Normal bell curve. Medical research uncertainties are generally as well calibrated as those in physics, but physics precision improves more rapidly, making feasible simple significance criteria such as the 5σ discovery convention in particle physics. Contributions to measurement uncertainty from mistakes and unknown problems are not completely unpredictable. Such errors appear to have power-law distributions consistent with how designed complex systems fail, and how unknown systematic errors are constrained by researchers. This better understanding may help improve analysis and meta-analysis of data, and help scientists and the public have more realistic expectations of what scientific results imply.

评判定量研究的显著性与可重复性，需要充分理解其相关不确定度，但目前人们往往难以明确这些不确定度的估算精度与实际意义。本研究通过分析来自医学、核物理与粒子物理领域的3200个物理量的41000次测量数据，以及从化学到毒理学的跨实验室比对结果，对已报道的科学不确定度展开了研究。异常值（outlier）十分常见，5σ偏差的出现频率远超直觉预期，最高可达预期值的五个数量级。同一物理量的多次测量结果经不确定度归一化后的差值，服从厚尾的学生t分布（Student-t分布），且该分布往往近似柯西分布（Cauchy distribution），与高斯正态（Gaussian Normal）钟形曲线相去甚远。医学研究中的不确定度校准精度通常与物理领域相当，但物理领域的测量精度提升速度更快，这使得诸如粒子物理领域中5σ发现惯例这类简单显著性判定标准具备了可行性。由失误与未知问题引发的测量不确定度贡献因素，并非完全不可预测。这类误差的分布似乎服从幂律分布（power-law distributions），这与人工设计的复杂系统的失效规律，以及研究者对未知系统误差（systematic error）的约束机制相一致。这一更为深入的认知，或可助力改进数据的分析与元分析（meta-analysis）流程，同时帮助科研人员与公众更客观地理解科学研究结果的实际意义。