Five-leaf generalizations of the D-statistic reveal the directionality of admixture

Over the past 15 years, the D-statistic, a four-taxon test for organismal
admixture (hybridization, or introgression) which incorporates single
nucleotide polymorphism data with allelic patterns ABBA and BABA, has seen
considerable use. This statistic seeks to discern significant deviation
from either a given species tree assumption, or from the balanced
incomplete lineage sorting that could otherwise defy this species tree.
However, while the D-statistic can successfully discriminate admixture
from incomplete lineage sorting, it is not a simple matter to determine
the directionality of admixture using only four-leaf tree models. As such,
methods have been developed that use 5 leaves to evaluate admixture. Among
these, the DFOIL method, which tests allelic patterns on the “symmetric”
tree S = (((1,2),(3,4)),5), succeeds in finding admixture direction for
many five-taxon examples. However, DFOIL does not make full use of all
symmetry, nor can DFOIL function properly when ancient samples are
included because of the reliance on singleton patterns (such as BAAAA and
ABAAA). Here, we take inspiration from DFOIL to develop a new and
completely general family of five-leaf admixture tests, dubbed
Δ-statistics, that can either incorporate or exclude the singleton allelic
patterns depending on individual taxon and age sampling choices. We
describe two new shapes that are also fully testable, namely the
“asymmetric” tree A = ((((1,2),3),4),5) and the “quasisymmetric” tree Q =
(((1,2),3),(4,5)), which can considerably supplement the “symmetric“ S =
(((1,2),(3,4)),5) model used by DFOIL. We demonstrate the consistency of
Δ-statistics under various simulated scenarios, and provide empirical
examples using data from black, brown and polar bears, the latter also
including two ancient polar bear samples from previous studies. Recently
DFOIL and one of these ancient samples was used to argue for a dominant
polar bear → brown bear introgression direction. However, we find, using
both this ancient polar bear and our own, that by far the strongest signal
using both DFOIL and Δ-statistics on tree S is actually bidirectional gene
flow of indistinguishable direction. Further experiments on trees A and Q
instead highlight what were likely two phases of admixture: one with
stronger brown bear → polar bear introgression in ancient times, and a
more recent phase with predominant polar bear → brown bear directionality.
Code and documentation available at
https://github.com/KalleLeppala/Delta-statistics.