Data from: Constraints on the FST–heterozygosity outlier approach

The FST-heterozygosity outlier approach has been a popular method for identifying loci under balancing and positive selection since Beaumont and Nichols first proposed it in 1996 and recommended its use for studies sampling a large number of independent populations (at least 10). Since then, their program FDIST2 and a user-friendly program optimized for large datasets, LOSITAN, have been used widely in the population genetics literature, often without the requisite number of samples. We observed empirical datasets whose distributions could not be reconciled with the confidence intervals generated by the null coalescent island model. Here, we use forward-in-time simulations to investigate circumstances under which the FST-heterozygosity outlier approach performs poorly for next-generation single-nucleotide polymorphism (SNP) datasets. Our results show that samples involving few independent populations, particularly when migration rates are low, result in distributions of the FST-heterozygosity relationship that are not described by the null model implemented in LOSITAN. In addition, even under favorable conditions LOSITAN rarely provides confidence intervals that precisely fit SNP data, making the associated p-values only roughly valid at best. We present an alternative method, implemented in a new R package named fsthet, which uses the raw empirical data to generate smoothed outlier plots for the FST-heterozygosity relationship.自1996年Beaumont与Nichols首次提出FST-杂合性离群值方法以来，该方法便成为识别平衡选择与正选择作用下基因座的常用方法，且二人推荐该方法适用于采样至少10个独立种群的研究。此后，他们开发的FDIST2程序，以及针对大型数据集优化的易用工具LOSITAN，已在种群遗传学领域的文献中被广泛应用，但相关研究往往未满足其所需的样本量要求。我们发现，部分经验数据集的分布无法与零值趋同岛屿模型生成的置信区间相契合。本研究借助正向时间模拟，探究FST-杂合性离群值方法在处理下一代单核苷酸多态性（Single-Nucleotide Polymorphism, SNP）数据集时表现不佳的场景。研究结果显示，当独立种群样本量较少，尤其是迁移率较低时，FST与杂合性的相关性分布无法被LOSITAN中采用的零模型所描述。此外，即便在理想条件下，LOSITAN生成的置信区间也极少能精准适配SNP数据，致使其关联的p值至多仅具备粗略有效性。本文提出一种替代方法，该方法已在名为fsthet的全新R包中实现，可利用原始经验数据为FST-杂合性相关性生成平滑的离群值绘图。