Identifying the best approximating model in Bayesian phylogenetics: Bayes factors, cross-validation or wAIC?

There is still no consensus as to how to select models in Bayesian
phylogenetics, and more generally in applied Bayesian statistics. Bayes
factors are often presented as the method of choice, yet other approaches
have been proposed, such as cross-validation or information criteria. Each
of these paradigms raises specific computational challenges, but they also
differ in their statistical meaning, being motivated by different
objectives: either testing hypotheses or finding the best-approximating
model. These alternative goals entail different compromises, and as a
result, Bayes factors, cross-validation and information criteria may be
valid for addressing different questions. Here, the question of Bayesian
model selection is revisited, with a focus on the problem of finding the
best-approximating model. Several model selection approaches were
re-implemented, numerically assessed and compared: Bayes factors,
cross-validation (CV), in its different forms (k-fold or leave-one-out),
and the widely applicable information criterion (wAIC), which is
asymptotically equivalent to leave-one-out cross validation (LOO-CV).
Using a combination of analytical results and empirical and simulation
analyses, it is shown that Bayes factors are unduly conservative. In
contrast, cross-validation represents a more adequate formalism for
selecting the model returning the best approximation of the
data-generating process and the most accurate estimates of the parameters
of interest. Among alternative CV schemes, LOO-CV and its asymptotic
equivalent represented by the wAIC, stand out as the best choices,
conceptually and computationally, given that both can be simultaneously
computed based on standard MCMC runs under the posterior distribution.