Hierarchical heuristic species delimitation under the multispecies coalescent model with migration

The multispecies coalescent (MSC) model accommodates genealogical
fluctuations across the genome and provides a natural framework for
comparative analysis of genomic sequence data from closely related species
to infer the history of species divergence and gene flow. Given a set of
populations, hypotheses of species delimitation (and species phylogeny)
may be formulated as instances of MSC models (e.g., MSC for one species
versus MSC for two species) and compared using Bayesian model selection.
This approach, implemented in the program bpp, has been found to be prone
to over-splitting. Alternatively heuristic criteria based on population
parameters (such as population split times, population sizes, and
migration rates) estimated from genomic data may be used to delimit
species. Here we develop hierarchical merge and split algorithms for
heuristic species delimitation based on the genealogical divergence index
(𝑔𝑑𝑖) and implement them in a python pipeline called hhsd. We characterize
the behavior of the 𝑔𝑑𝑖 under a few simple scenarios of gene flow. We
apply the new approaches to a dataset simulated under a model of isolation
by distance as well as three empirical datasets. Our tests suggest that
the new approaches produced sensible results and were less prone to
over-splitting. We discuss possible strategies for accommodating
paraphyletic species in the hierarchical algorithm, as well as the
challenges of species delimitation based on heuristic criteria.