Originally published as Genetics Published Articles Ahead of Print on July 27, 2008.

Genetics, Vol. 179, 2091-2112, August 2008, Copyright © 2008
doi:10.1534/genetics.107.084418

Frequency-Dependent Selection and the Evolution of Assortative Mating

Sarah P. Otto*,1, Maria R. Servedio{dagger} and Scott L. Nuismer{ddagger}

* Department of Zoology, University of British Columbia, Vancouver, British Columbia V6T 1Z4, Canada, {dagger} Department of Biology, University of North Carolina, Chapel Hill, North Carolina 27599-3280 and {ddagger} Department of Biological Sciences, University of Idaho, Moscow, Idaho 83844-3051

1 Corresponding author: Department of Zoology, University of British Columbia, 6270 University Blvd., Vancouver, BC V6T 1Z4, Canada.
E-mail: otto{at}zoology.ubc.ca

A long-standing goal in evolutionary biology is to identify the conditions that promote the evolution of reproductive isolation and speciation. The factors promoting sympatric speciation have been of particular interest, both because it is notoriously difficult to prove empirically and because theoretical models have generated conflicting results, depending on the assumptions made. Here, we analyze the conditions under which selection favors the evolution of assortative mating, thereby reducing gene flow between sympatric groups, using a general model of selection, which allows fitness to be frequency dependent. Our analytical results are based on a two-locus diploid model, with one locus altering the trait under selection and the other locus controlling the strength of assortment (a "one-allele" model). Examining both equilibrium and nonequilibrium scenarios, we demonstrate that whenever heterozygotes are less fit, on average, than homozygotes at the trait locus, indirect selection for assortative mating is generated. While costs of assortative mating hinder the evolution of reproductive isolation, they do not prevent it unless they are sufficiently great. Assortative mating that arises because individuals mate within groups (formed in time or space) is most conducive to the evolution of complete assortative mating from random mating. Assortative mating based on female preferences is more restrictive, because the resulting sexual selection can lead to loss of the trait polymorphism and cause the relative fitness of heterozygotes to rise above homozygotes, eliminating the force favoring assortment. When assortative mating is already prevalent, however, sexual selection can itself cause low heterozygous fitness, promoting the evolution of complete reproductive isolation (akin to "reinforcement") regardless of the form of natural selection.