Measuring population differentiation using GST or D? A simulation study with microsatellite DNA markers under a finite island model and nonequilibrium conditions

Genetic differentiation of populations is a key question in population genetic investigations. Wright’s FST (and its relatives such as GST) has been a standard measure of differentiation. However, the deficiencies of these indexes and their significance have been increasing realized in recent years, leading to some new measures being proposed, such as Jost’s (2008) D. This has also stimulated some considerable debate which, in certain sense, makes empirical biologists even more confused, for example, on statistics which should be used for estimating population differentiation. Here we report a simulation study with neutral microsatellite DNA loci under a finite island model to compare the performance of GST and D, under non-equilibrium conditions, in particular. Our results suggest that there exist fundamental differences between the two statistics and neither GST nor D operate satisfactorily in all situations for quantifying differentiation. D is very sensitive to mutation models but G...