Data from: Not Normal: the uncertainties of scientific measurements

Judging the significance and reproducibility of quantitative research
requires a good understanding of relevant uncertainties, but it is often
unclear how well these have been evaluated and what they imply. Reported
scientific uncertainties were studied by analysing 41 000 measurements of
3200 quantities from medicine, nuclear and particle physics, and
interlaboratory comparisons ranging from chemistry to toxicology. Outliers
are common, with 5σ disagreements up to five orders of magnitude more
frequent than naively expected. Uncertainty-normalized differences between
multiple measurements of the same quantity are consistent with
heavy-tailed Student’s t-distributions that are often almost Cauchy, far
from a Gaussian Normal bell curve. Medical research uncertainties are
generally as well evaluated as those in physics, but physics uncertainty
improves more rapidly, making feasible simple significance criteria such
as the 5σ discovery convention in particle physics. Contributions to
measurement uncertainty from mistakes and unknown problems are not
completely unpredictable. Such errors appear to have power-law
distributions consistent with how designed complex systems fail, and how
unknown systematic errors are constrained by researchers. This better
understanding may help improve analysis and meta-analysis of data, and
help scientists and the public have more realistic expectations of what
scientific results imply.