Incompatibility of Data with Hypotheses, or Hypotheses with Data? Beware the Bayesian Within You

Neo-Fisherian P-values are conceived as indices of (in)compatibility between data and statistical models. However, some linguistic devices risk creating an inversion fallacy, resulting in the use of P-values as indices of (in)compatibility between models and data rather than the other way around. While this switch is defensible and may appear merely semantic or philosophical, frequentist inference treats models as provisionally true, quantifying the probability density of the data under those models, not vice versa. Since the neo-Fisherian school itself has witnessed how easily inversion fallacies arise—such as reading P-values as posterior probabilities of hypotheses given the data—this paper argues that future work should adopt less ambiguous language to preserve the correct directionality of statistical (in)compatibility. Note that the inversion problem may intensify when adopting measures such as the S-value, used to assess how “surprising” a result is under the model’s assumptions by comparison with the “surprise” of obtaining ‘s’ heads in ‘s’ coin tosses under the assumption of no bias toward heads. Indeed, referring to a hypothesis as more/less incompatible with the data legitimizes expressions such as ‘the hypothesis is more/less “surprising” compared with the data,’ which in turn may be misconstrued as a Bayesian assessment.