The evolutionary response of mating system to heterosis

Isolation allows populations to diverge and to fix different alleles. Deleterious alleles that reach locally high frequencies contribute to genetic load, especially in inbred or selfing populations, in which selection is relaxed. In the event of secondary contact, the recessive portion of the genetic load is masked in the hybrid offspring, producing heterosis. This advantage, only attainable through outcrossing, should favor evolution of greater outcrossing even if inbreeding depression has been purged from the contributing populations. Why, then, are selfing-to-outcrossing transitions not more common? To evaluate the evolutionary response of mating system to heterosis, we model two monomorphic populations of entirely selfing individuals, introduce a modifier allele that increases the rate of outcrossing, and investigate whether the heterosis among populations is sufficient for the modifier to invade and fix. We find that the outcrossing mutation invades for many parameter choices, but it rarely fixes unless populations harbor extremely large unique fixed genetic loads. Reversions to outcrossing become more likely as the load becomes more polygenic, or when the modifier appears on a rare background, such as by dispersal of an outcrossing genotype into a selfing population. More often, the outcrossing mutation instead rises to moderate frequency, which allows recombination in hybrids to produce superior haplotypes that can spread without the mutation's further assistance. The transience of heterosis can therefore explain why secondary contact does not commonly yield selfing-to-outcrossing transitions.