Toward Replicability With Confidence Intervals for the Exceedance Probability

Several scientific fields including psychology are undergoing a replication crisis. There are many reasons for this problem, one of which is a misuse of <i>p</i>-values. There are several alternatives to <i>p</i>-values, and in this article we describe a complement that is geared toward replication. In particular, we focus on confidence intervals for the probability that a parameter estimate will exceed a specified value in an exact replication study. These intervals convey uncertainty in a way that <i>p</i>-values and standard confidence intervals do not, and can help researchers to draw sounder scientific conclusions. After briefly reviewing background on <i>p</i>-values and a few alternatives, we describe our approach and provide examples with simulated and real data. For linear models, we also describe how confidence intervals for the exceedance probability are related to <i>p</i>-values and confidence intervals for parameters.