Epistasis, inbreeding depression and the evolution of self-fertilization

Inbreeding depression resulting from partially recessive deleterious
alleles is thought to be the main genetic factor preventing
self-fertilizing mutants from spreading in outcrossing hermaphroditic
populations. However, deleterious alleles may also generate an advantage
to selfers in terms of more efficient purging, while the effects of
epistasis among those alleles on inbreeding depression and mating system
evolution remain little explored. In this paper, we use a general model of
selection to disentangle the effects of different forms of epistasis
(additive-by-additive, additive-by-dominance and dominance-by-dominance)
on inbreeding depression and on the strength of selection for selfing.
Models with fixed epistasis across loci, and models of stabilizing
selection acting on quantitative traits (generating distributions of
epistasis) are considered as special cases. Besides its effects on
inbreeding depression, epistasis may increase the purging advantage
associated with selfing (when it is negative on average), while the
variance in epistasis favors selfing through the generation of linkage
disequilibria that increase mean fitness. Approximations for the strengths
of these effects are derived, and compared with individual-based
simulation results.