## Abstract

Evolutionary biologists have identified several factors that could explain the widespread phenomena of sex and recombination. One hypothesis is that host–parasite interactions favor sex and recombination because they favor the production of rare genotypes. A problem with many of the early models of this so-called Red Queen hypothesis is that several factors are acting together: directional selection, fluctuating epistasis, and drift. It is thus difficult to identify what exactly is selecting for sex in these models. Is one factor more important than the others or is it the synergistic action of these different factors that really matters? Here we focus on the analysis of a simple model with a single mechanism that might select for sex: fluctuating epistasis. We first analyze the evolution of sex and recombination when the temporal fluctuations are driven by the abiotic environment. We then analyze the evolution of sex and recombination in a two-species coevolutionary model, where directional selection is absent (allele frequencies remain fixed) and temporal variation in epistasis is induced by coevolution with the antagonist species. In both cases we contrast situations with weak and strong selection and derive the evolutionarily stable (ES) recombination rate. The ES recombination rate is most sensitive to the period of the cycles, which in turn depends on the strength of epistasis. In particular, more virulent parasites cause more rapid cycles and consequently increase the ES recombination rate of the host. Although the ES strategy is maximized at an intermediate period, some recombination is favored even when fluctuations are very slow. By contrast, the amplitude of the cycles has no effect on the ES level of sex and recombination, unless sex and recombination are costly, in which case higher-amplitude cycles allow the evolution of higher rates of sex and recombination. In the coevolutionary model, the amount of recombination in the interacting species also has a large effect on the ES, with evolution favoring higher rates of sex and recombination than in the interacting species. In general, the ES recombination rate is less than or equal to the recombination rate that would maximize mean fitness. We also discuss the effect of migration when sex and recombination evolve in a metapopulation. We find that intermediate parasite migration rates maximize the degree of local adaptation of the parasite and lead to a higher ES recombination rate in the host.

THE vast majority of species reproduce sexually, at least occasionally. Such a widespread success of sex and recombination is problematic given the strong fitness costs associated with sexual reproduction (*e.g*., the twofold cost of sex, the cost of breaking a favorable combination of genes, the cost of finding and courting a mate, etc.). This problem has attracted considerable theoretical attention (Maynard-Smith 1978; Kondrashov 1993; Barton and Charlesworth 1998). Although some theories invoke *proximate* (or *mechanistic*) explanations (*e.g*., sex might be induced to promote DNA repair), we focus on *evolutionary* (or *generative*) hypotheses that focus on the effects of sex and recombination on genetic associations. The modifier theory approach, introduced by Nei (1967), has helped to clarify the conditions under which sex and recombination might evolve by considering the dynamics at genes that alter (“modify”) the mode of reproduction and the frequency of crossover events.

In a general model of recombination evolution, Barton (1995) showed that two conditions favor a modifier allele that increases recombination. First, recombination can be favored if it breaks apart less-fit combinations of genes and, consequently, increases the mean fitness of the descendants, a so-called *short-term* benefit of recombination (because a modifier inducing more recombination immediately increases in frequency). Second, recombination can be favored if it increases the variance in fitness of the descendants and, consequently, the efficacy of selection, a so-called *long-term* benefit of recombination (because the frequency of the modifier changes by hitchhiking with the most fit of the variants produced, which take several generations to spread). In either case, selection on the modifier of recombination is indirect, with frequency changes at the modifier locus occurring via associations with other loci that are directly under selection.

The main determinant of whether these short-term and long-term effects actually do benefit recombination is the type of genetic associations (linkage disequilibria) that exist among selected alleles. In general, evolutionary hypotheses can be classified according to the forces that generate these genetic associations (Kondrashov 1993). Here, we review four leading hypotheses:

If selection is constant in time, linkage disequilibrium develops with the same sign as the multiplicative epistasis, which is a measure of the curvature of the fitness surface measured on a log scale (Felsenstein 1965; Eshel and Feldman 1970). Basically, selection builds up allelic combinations that work well together, which causes disequilibrium to have the same sign as epistasis and implies that genetic associations among alleles tend to increase fitness, on average. In this case, the main short-term effect of recombination is to decrease the average fitness of offspring. Nevertheless, evolution favors higher rates of recombination via a long-term benefit, as long as epistasis is negative and sufficiently weak that the short-term costs are not too severe (Feldman

*et al*. 1980; Barton 1995; Otto and Feldman 1997). With negative epistasis, advantageous alleles at one locus become associated with disadvantageous alleles at a second locus (negative disequilibrium), which hinders a population's response to selection. Higher rates of recombination can be favored in this situation because it breaks this linkage disequilibrium and thus facilitates the response to natural selection.If epistasis fluctuates over time, a lag between epistasis and linkage disequilibrium can develop, leading to a mismatch at some points in time between which combinations of alleles are most fit and are most common (Sturtevant and Mather 1938; Charlesworth 1976; Maynard Smith 1978; Barton 1995). In this situation, recombination can be favored because it breaks apart the currently maladapted allele combinations and increases the mean fitness of descendants (a short-term benefit). Because such mismatches must occur often to have much influence, however, this mechanism works only under restrictive parameter values (Charlesworth 1976; Barton 1995). In particular, Barton (1995) found that epistasis must fluctuate very rapidly for this mechanism to work: epistasis must change sign every 2–5 generations, implying cycles with a period of 4–10 generations (the so-called “Barton zone”; Peters and Lively 1999) to account for high rates of recombination.

When the population is finite, the interaction between genetic drift and selection yields negative linkage disequilibrium (Hill and Robertson 1966). In this case, increased recombination can evolve because recombination increases the variance in fitness of the descendants (a long-term benefit) through the production of high-fitness genotypes that are rare or absent due to genetic drift (Felsenstein and Yokoyama 1976; Otto and Barton 2001; Iles

*et al*. 2003; Barton and Otto 2005; Keightley and Otto 2006; Martin*et al*. 2006). This process can select for recombination over a range of both positive and negative epistasis, but it requires that drift is neither too weak (no effect on linkage disequilibrium) nor too strong (polymorphisms are rapidly lost). Nevertheless, a drift-based advantage to recombination can occur even in very large populations as long as they are spatially structured (Martin*et al*. 2006) and/or selection acts on a large number of loci (Iles*et al*. 2003).Selection that varies over space can promote the evolution of recombination in the absence of genetic drift. With migration among populations, spatial heterogeneity in selection can generate positive or negative linkage disequilibrium and select for or against recombination, depending on the sign of epistasis (Pylkov

*et al*. 1998; Lenormand and Otto 2000). Modifier theory thus provides a general framework that allows us to formalize and compare different evolutionary hypotheses for the evolution of sex and recombination.

All of the theories reviewed above focus on a single species in which recombination is evolving. Where do species interactions fit within this framework? The Red Queen hypothesis posits that the biotic environment of a species is continually changing due to the coevolution of surrounding species, including parasites, pathogens, predators, competitors, etc. Within this changing biotic environment, sex and recombination might be favored as mechanisms that generate rare combinations of alleles to which antagonistic species are not adapted (akin to the advantage of recombination in an abiotically fluctuating environment). Unfortunately, the intrinsic complexity of coevolutionary dynamics impedes a general analytical treatment of the problem. In fact, most theoretical articles on the Red Queen hypothesis are based on the exploration of complex simulation models, making it difficult to determine exactly what selects for sex (Otto and Michalakis 1998). For example, in one of the first simulation models used to formalize the Red Queen hypothesis in a metapopulation (Ladle *et al*. 1993), all four of the factors listed above may have been responsible for the evolution of host recombination: (i) directional selection, (ii) fluctuating epistasis, (iii) interaction between drift and selection, and (iv) spatial covariance in selection. Is it the synergistic action of these multiple factors that explains the success of sex in these models (West *et al*. 1999) or could each factor work in isolation?

To answer this question one needs to analyze simpler models with fewer factors affecting the evolution of sex and recombination. Recent studies of the Red Queen hypothesis (Peters and Lively 1999; Otto and Nuismer 2004) have focused on two forces: directional selection and fluctuating epistasis. Otto and Nuismer (2004) developed a general model of species interactions and showed that under weak selection, the epistasis generated by most genetic models underlying species interactions (matching-genotypes model, gene-for-gene model, a quantitative trait model) is too strong relative to the strength of selection to favor the evolution of high rates of recombination. When epistasis is strong relative to selection, the main effect of sex and recombination is to break apart the good gene combinations that allowed parents to survive and reproduce in the face of the current suite of coevolving species. Consequently, the short-term cost of sex (the recombination load) is too severe relative to the long-term benefit (the production of rare genotypes). Nevertheless when selection is strong, simulations revealed that recombination can be favored (Peters and Lively 1999; Otto and Nuismer 2004). What force selects for recombination in these cases? Peters and Lively (1999) examined the coevolutionary dynamics of a matching-alleles model, where parasites must match a host's alleles to infect, and found that there is often a discrepancy between which combinations of alleles are currently present (disequilibria) and which are currently favored (epistasis). They thus concluded that fluctuating epistasis is the primary force driving the evolution of recombination when selection is strong, although directional selection also contributed (see also Lythgoe 2000). In summary, when selection is weak, recombination induces a short-term cost by breaking apart good combinations of alleles, which prevents the evolution of recombination despite the potential for long-term benefits. In contrast, when selection is strong, the short-term effect of recombination is to break apart currently maladapted gene combinations, which allows modifiers that increase recombination to spread.

In the above studies (Peters and Lively 1999; Lythgoe 2000; Otto and Nuismer 2004), the case where selection is strong relative to the rates of sex and recombination was examined only through simulation (Pomiankowski and Bridle 2004). In the present article, we examine simpler models that allow us to focus entirely on the analysis of fluctuating epistasis as a factor favoring the evolution of sex and recombination. Specifically, we use fitness functions that lead allele frequencies to equilibrate at , after which point directional selection is absent. Although these models are not empirically justified, we consider them to represent extreme scenarios, where directional selection is absent and where we can get an analytical handle on the impact of fluctuations in epistasis, which are observed in more biologically realistic models. This approach was introduced by Nee (1989), who found that coevolutionary interactions provide an advantage to recombination that is independent of the period of the fluctuations. This contrasts with the results of Peters and Lively (1999, 2007), who found that host–parasite coevolution does not favor high rates of recombination much outside the Barton zone (*i.e*., fluctuating epistasis with a period between 4 and 10 generations).

To elucidate the mechanisms at work, we adopt a two-step approach. We first analyze a one-species model where fluctuations in epistasis are governed by the abiotic environment (as in Sasaki and Iwasa 1987). We then study a second model where fluctuations in epistasis are driven by host–parasite coevolution (as in Nee 1989). Together these two models help clarify the extent to which fluctuating epistasis—with or without coevolution—favors the evolution of sex and recombination. In both cases, we consider whether evolutionarily stable rates of recombination coincide with the rate of recombination that maximizes mean fitness (Sasaki and Iwasa 1987; Nee 1989).

Finally, we use these two models to analyze the evolution of sex and recombination in a metapopulation, accounting for migration among patches. Furthermore, we discuss the link between the evolution of recombination and local adaptation (a measure of mean fitness in these models). Throughout, we consider only deterministic models with large local and global population sizes.

## ONE-SPECIES MODEL

We first consider a single haploid and hermaphroditic organism with nonoverlapping generations living in an isolated and large population of constant size (no drift). Using a deterministic model, we follow haplotype frequencies through a life cycle consisting of a census, selection, and reproduction. We further assume random mating when sex occurs. Selection acts on two loci (*A* and *B*) with two alleles (*A*/*a*, *B*/*b*). The phenotype of an individual is set to 0 if it is either *AB* or *ab* and to 1 if it is *aB* or *Ab*. Phenotype 0 has fitness , while phenotype 1 has fitness (Figure 1a). It is assumed that the quantity varies sinusoidally over time,(1)where measures the amplitude of the fitness oscillations and is the speed of these oscillations, which varies between 0 (infinitely slow oscillations) and ( changes sign each generation). The corresponding period of the cycle is . This fitness regime was first introduced in the case of a single species by Sturtevant and Mather (1938) and was analyzed by Sasaki and Iwasa (1987).

To investigate the evolution of sex and recombination, we allow the probability of sex and/or the rates of recombination to depend on a third modifier locus (*M*) with two alleles (*M*/*m*), with gene order *MAB*. Thus, there are eight possible haplotypes (*MAB*, *MAb*, *MaB*, *Mab*, *mAB*, *mAb*, *maB*, *mab*, labeled from 1 to 8), whose frequencies among adults are *x _{j}*, for

*j*= 1–8. Whenever convenient,

*p*will denote the frequency of allele

_{j}*j*. The modifier locus is assumed to be neutral (but this assumption is relaxed in the discussion), and the fitness of genotype

*j*is thus , where if and otherwise. After selection, the haplotypic frequencies are for haplotype

*j*, where is the mean fitness of the population.

Following selection, the probability that two haploid individuals carrying modifier alleles *k* and *l* engage in sexual reproduction is σ_{kl}. Haploid individuals that do not mate reproduce asexually. Sexual mating is followed immediately by meiosis with recombination between loci *A* and *B* at rate *r _{kl}* and between loci

*M*and

*A*at rate

*R*. We let

_{kl}*c*denote the probability of a double crossover. Appropriate choices of

_{kl}*c*thus allow interference among chiasmata and alternative gene orders to be considered. In deriving the recursions for a haploid population, the probabilities of undergoing sex and the rates of recombination always enter as products. Therefore, we simplify the equations using the compound parametersBecause recombination affects the array of offspring produced only when it occurs between heterozygous loci, the compound parameters involving the

_{kl}*M*locus (ψ

_{kl}and χ

_{kl}) are relevant only in

*Mm*heterozygotes, so we may drop the

*kl*subscript. A major advantage of using these compound parameters is that in a haploid population, the same equations apply whether the modifier locus alters the probability of sex or the rates of recombination or both. The special case of a sexual form arising within an asexual population with no gene flow between them can also be modeled by setting ρ

_{mm}> 0 and ρ

_{Mm}= ρ

_{MM}= ψ = χ= 0, in which case only group selection acts on the frequency of sex (Felsenstein 1974). We used the recursion equations presented in Otto and Nuismer (2004) to follow the frequencies of each genotype after reproduction.

#### Change in modifier frequency:

To determine if higher rates of sex and recombination are favored we ask whether an allele *m* that increases the rate of sex and recombination, ρ_{kl}, rises in frequency. After selection and sexual reproduction the change in frequency of *m* exactly equals(2a)In the above equation, and refer to the linkage disequilibria between two loci (*i* and *j*) and between three loci (*i*, *j*, and *k*) at time *t* (see appendix a). We simplify the notation in Equation 2a by defining new variables to measure the departure of the selected allele frequencies from : and . If we assume that α and linkage disequilibria are small (of order ζ), Equation 2a becomes(2b)Equation 2 shows that the fate of the modifier depends on the associations between the modifier locus, *M*, and the loci *A* and *B*; that is, the frequency of the allele *m* evolves only through indirect selection (hitchhiking) with loci *A* and *B*. To proceed in the analysis, we thus need to understand the dynamics of these genetic associations.

#### A QLE analysis:

As a first attempt to predict the fate of the modifier, let us assume that the processes generating disequilibria (epistasis in this model) are weak relative to the processes reducing disequilibria (sex and recombination). In this case, the disequilibria between loci quickly reach their steady-state values predicted on the basis of current allele frequencies, the selection coefficients, and the rates of sex and recombination. This steady state is known as the “quasilinkage equilibrium” (QLE). Assuming a modifier that has only a small effect on sex and recombination, the dynamics of a modifier allele in a single population can be approximated at QLE by(3)Equation 3 was derived following the methods described in Barton (1995). The term *K* measures the efficacy of the modifier, which equals the average effect of the modifier, , divided by the probability that recombination breaks apart a three-locus haplotype , thereby breaking down associations among the modifier and selected loci. The term is the average rate of sex and recombination between loci *A* and *B*. The term *E* measures the amount of epistasis defined on a multiplicative scale: . Finally, *v* measures the contribution of linkage disequilibrium to the additive genetic variance in fitness and equals , where *a _{j}* is the total selective force acting on allele

*j*: . When

*v*is positive (negative), the linkage disequilibrium increases (decreases) the frequency of the currently most- and least-fit combinations of alleles, which accelerates (hinders) evolutionary change (Barton 1995).

The direction of selection of the modifier depends on the sum of two terms (Barton 1995; Lenormand and Otto 2000). The first term (proportional to ) measures the short-term effect of recombination: recombination is beneficial when it breaks apart unfavorable genetic associations (*i.e*., when disequilibria have opposite sign to the current value of epistasis). The second term (proportional to ) measures the long-term effect of recombination: recombination is beneficial when breaking down linkage disequilibria increases the additive genetic variance in fitness (this occurs when ).

Following Barton (1995), we can also determine the QLE value of the disequilibrium between loci *A* and *B* when epistasis fluctuates according to Equation 1 (appendix a),(4a)whereand(4b)At this QLE, is of smaller order than , and Equation 3 reduces to(5)with . If *m* is an allele increasing the frequency of sex and/or recombination, then , and Equation 5 is always negative, predicting that sex and recombination should always be selected against. Essentially, sex and recombination are not favored because epistasis is too strong relative to the strength of directional selection, and the main effect of recombination is to break apart good gene combinations. The short-term cost of sex and recombination thus outweighs any potential long-term benefits.

This prediction matches the results of simulations presented by Sasaki and Iwasa (1987) when the period is very large and recombination rates are initially high (see also our simulation results in Figure 2). When the period is short or recombination rates are low, however, some positive level of recombination can evolve. The discrepancy with the QLE analysis comes from the fact that fluctuations in epistasis that are rapid relative to the frequency of sex and recombination drag the linkage disequilibria away from their QLE values.

#### Recursion analysis:

Following the analysis of Barton (1995, Appendix 4) we now take into account fluctuations in linkage disequilibria without assuming that linkage disequilibrium is always and instantaneously at its steady-state value (the QLE assumption). This recursion analysis is greatly simplified by the peculiar dynamical behavior of this simplified model. As pointed out by Sasaki and Iwasa (1987), the selection regime used rapidly yields a situation where allele frequencies converge toward [*i.e*., and when ]. Sasaki and Iwasa (1987) proved the convergence in a continuous-time model, and numerical simulations of our discrete-time model confirm this convergence as long as some recombination occurs. Once the allele frequencies have reached , the dynamics of the system are very simple, and only linkage disequilibria vary through time. Furthermore, the dynamics of the modifier locus are simplified, because when directional selection is absent [, see (4b)]. In other words, the evolution of a modifier of recombination is governed only by the short-term effect of recombination, and the change in the frequency of allele *m* on the *M* locus exactly equals(6a)If we further assume that and linkage disequilibria are small (of order ζ) this yields (using 4b)(6b)

Thus the direction of selection on the modifier depends only on the product of epistasis and the three-locus disequilibrium. This three-locus disequilibrium depends on the two-locus disequilibrium between loci *A* and *B*. In appendix a, we assume that the effect of the modifier is weak () to derive an approximation for the three-locus disequilibrium,(7)where and . Using (7) in (6b) we get(8)Because is a rapidly decreasing function of τ in the presence of sex and recombination, Barton (1995) argued that only the first few terms in the above sum have a major impact on the evolution of sex and recombination.

For sex and recombination to be favored, must be positive, which requires knowledge of the dynamics of the two-locus linkage disequilibrium. Again assuming that and linkage disequilibria are small (of order ζ), the two-locus disequilibrium is governed by the recursion equation(9)(appendix a), where . The *Z* transform can be used to solve the recursion Equation 9, and assuming disequilibrium is initially absent, we obtain the general solution(10)When there is some sex and recombination (), Equation 10 converges toward(11a)which can be evaluated using the fitness function (1),(11b)where(11c)The term represents the lag between the fluctuations in epistasis and the two-locus linkage disequilibrium. This lag increases with the speed of the fluctuations, *k*, and it decreases with recombination.

Using (11a) and (7) in (6b) yields the change in frequency of a modifier allele:(12)Because if there is some sex and recombination, the sum over τ can be evaluated explicitly,(13)where we have ignored transient dynamics due to the initial conditions. Averaging over a cycle (denoted ), we get the per generation change in the frequency of the modifier allele:(14)The above expression can be used to predict the direction of selection on the modifier locus.

If epistasis fluctuates slowly (*i.e*., ), we get(15a)which is equivalent to the QLE prediction (5) averaged across a cycle. From the fact that (15a) is negative, we recover the conclusion obtained from the QLE analysis: sex and recombination are selected against when epistasis fluctuates slowly. Conversely, selection favors increased levels of sex and recombination if epistasis fluctuates rapidly (*i.e*., ):(15b)More generally, if a population engages in little sex and recombination relative to the speed of the cycles (*i.e*., ), increased levels of sex and recombination are favored:(15c)In summary, when the rate of genetic mixing is initially low or the period of the cycles is sufficiently short, higher rates of sex and recombination evolve. Otherwise, decreased levels of sex and recombination evolve.

Ultimately, the rates of sex and recombination will reach an evolutionarily stable (ES) level that is no longer susceptible to invasion by other modifier alleles; *i.e*., . We now determine the ES level of recombination in two cases: when the loci are equidistant and when the recombination rate between loci *M* and *A* is fixed {other cases can be investigated numerically by setting in Equation 14}. In both cases, we assume that the population is sexual with no interference in recombination among the three loci (*i.e*., ).

When the loci are equidistant (*i.e*., ) the ES level of sex and recombination equals(16a)Alternatively, when the recombination rate between *M* and *A* is fixed at , the recombination rate between loci *A* and *B* evolves to a level that satisfies(16b)The evolutionary stability of these ES values was checked by verifying that when . Given the restriction that recombination rates must be , we have the following expression for the ES recombination rate:(17)When is held fixed, the ES value of recombination between loci *A* and *B* (16b) increases when there is tighter linkage between the modifier and the first selected locus (*i.e*., when is smaller). This makes sense as the modifier can hitchhike for longer by association with favorable alleles if tightly linked to at least one of the selected loci. Furthermore, the ES value increases with the speed of the oscillations, *k*, and thus decreases with the period of the cycle. Indeed, the maximum level of recombination, 0.5, is the evolutionarily stable state for periods between 2 and 7 generations when the loci are equidistant [from (16a)] or when the modifier is loosely linked [from (16b)] and for periods between 2 and 9 generations when the modifier is tightly linked [from (16b)]. Thus, this analysis confirms the general claim that high recombination rates evolve in the Barton zone (with periods of 4–10 generations). Although we find that high rates of recombination can evolve with periods shorter than the Barton zone (including periods of 2 generations, *i.e*., ), an examination of Barton's 1995 model indicates that this is possible in haploid models (as considered here), but not in diploid models as considered by Barton (1995).

As previously pointed out by several authors (Charlesworth 1976; Maynard-Smith 1978; Barton 1995), selection for high rates of recombination requires rapid fluctuations in the sign of epistasis. Interestingly, the amplitude of the oscillations () does not affect the direction of selection on a modifier or the ES value, although does affect the strength of selection and thus matters when there are direct costs associated with higher recombination (see discussion).

#### Mean fitness:

It is worth comparing the ES recombination rate to the recombination rate that would maximize the long-term geometric average of the mean fitness within the population, as studied by Sasaki and Iwasa (1987). In this one-species model with allele frequencies at , the mean fitness is(18a)If selection is weak [*i.e*., the amplitude, , is small and the disequilibrium is approximately (11b)], the geometric average of is well approximated by the arithmetic average over one cycle, which is(18b)where again , and is the lag between fluctuations in epistasis and disequilibrium given by Equation 11c. Recombination affects the mean fitness by altering the amplitude of the fluctuations in linkage disequilibrium and the lag, σ. While increasing recombination decreases the amplitude by which linkage disequilibrium oscillates in response to fluctuations in epistasis (see Equation 11b), it also decreases the lag because disequilibrium responds more quickly to changes in epistasis. These two effects act on the arithmetic mean of fitness in opposite directions, and there is consequently a recombination rate that maximizes (18b):(19)According to (19), the optimal recombination rate increases with the speed of the cycles. For long periods (*k* small), Equation 19 is ∼, as found by Sasaki and Iwasa (1987) using a similar model. For short periods (*k* large), Equation 19 more nearly equals , as found by Sasaki and Iwasa (1987) using a square-wave model of selection with maximally strong selection.

If group selection were governing the evolution of sex and recombination, the rate of sex and recombination would evolve toward . Indeed, when the modifier is completely linked to the *A* locus ( = 0), the optimal rate of sex and recombination (Equation 19) coincides with the ES level (Equation 16b). Consequently, the solid curve describing the ES level in Figure 2 when = 0 coincides with the optimal level of sex and recombination. In all other cases, the ES level of sex and recombination is lower than the value that would maximize mean fitness; this mismatch occurs because recombination between the modifier and the selected loci uncouples the fate of the modifier from its effects on the mean fitness of offspring.

## COEVOLUTIONARY MODEL

In the one-species model analyzed above, fluctuations in epistasis are imposed externally (see Equation 1). In contrast, we now focus on a situation where the temporal variability in epistasis emerges from coevolutionary interactions with an antagonist species. We follow haplotype frequencies in two haploid hermaphroditic species (the host and the parasite) through a life cycle consisting of a census, selection, and reproduction, with nonoverlapping generations. The two organisms form two large populations of constant size (no drift). Within each population we further assume random mating (when sex occurs). The species interaction is mediated via two loci (*A* and *B*) with two alleles (*A*/*a*, *B*/*b*) in each species. Every generation, each host encounters one parasite, chosen randomly from the parasite population. In each species, the phenotype of an individual is 0 if it is either *AB* or *ab* and is 1 if it is *aB* or *Ab*. When the host phenotype matches that of the one parasite that it encounters, the parasite gains a fitness advantage (α_{p} > 0), while the host suffers a fitness cost (−1 ≤ α_{h} < 0). The assumption regarding the specificity of the interaction (Figure 1b) was first introduced by Nee (1989) as a simple two-locus model that captures two essential features of the Red Queen hypothesis: fitness interactions are antagonistic and recombination influences the outcome of the species interaction. We thus refer to this coevolutionary model as the Nee model.

As in the one-species model, we allow the probability of sex and/or the rates of recombination to depend on a third modifier locus (*M*) with two alleles (*M*/*m*) and with gene order *MAB*. Thus, in each species, there are eight possible haplotypes (*MAB*, *MAb*, *MaB*, *Mab*, *mAB*, *mAb*, *maB*, *mab*, labeled from 1 to 8), whose frequencies among adults in species *i* are *x _{i}*

_{,j}for

*j*= 1–8. Where convenient, we denote the frequency of allele

*j*by

*p*

_{i}_{,j}. We use the subscript

*i*=

*h*to refer to parameters and variables in the host species and

*i*=

*p*to refer to the parasite. The fitness of genotype

*j*of species

*i*is then , where if and otherwise, and where refers to the other species. After selection, the haplotypic frequencies are for haplotype

*j*in species

*i*, where is the mean fitness of species

*i*.

Following selection, the probability that two haploid individuals carrying modifier alleles *k* and *l* engage in sexual reproduction follows the rules defined in the single-species case. To simplify matters, we consider the evolution of sex and recombination in one species (the focal species) at a time, setting = 0 in the interacting species for *j* = 5–8. To determine if higher rates of sex and recombination can be favored we track changes in the frequency of allele *m* in the focal species *i*.

#### QLE analysis:

When the fluctuations in epistasis are slow relative to the rates of recombination (*i.e*., when selection is weak, see Equation 27 below), a QLE analysis can be used to predict the fate of the modifier allele from Equation 3. In this model, the speed of these fluctuations is not a parameter but depends on the coevolutionary dynamics and, in particular, on the strength of selection on the host and the parasite (see Equation 27 below). Under the assumption that and linkage disequilibria are small (of order ), we have(20a)with . The selection coefficients induced by the other species become(20b)where and . Consequently, as in the one-species model, the product can be neglected relative to the strength of epistasis, and (3) reduces to(21)Thus, if the modifier allele *m* increases recombination in species *i*, and (21) is always negative. Here, as in the one-species model and as in most models analyzed by Otto and Nuismer (2004), sex and recombination are not favored because epistasis is too strong relative to the strength of directional selection.

Like the single-species model analyzed above, this coevolutionary model is peculiar in that allele frequencies converge toward as long as there is some recombination. After this convergence, allele frequencies remain fixed, and only the fluctuations in linkage disequilibrium are needed to describe the dynamics of the selected loci. Equation 20 no longer applies once the allele frequencies have reached , however, as the terms equal zero. It is thus necessary to consider smaller-order terms in the QLE approximation to predict the dynamics of the modifier. Assuming that sex and recombination are sufficiently frequent relative to the strength of selection in species *i* that the focal species (but not necessarily the interacting species) has reached QLE, and assuming that linkage disequilibria in both species are small (of order ), we have(22a)(22b)(22c)If the interacting species is also at QLE, Equations 22 are satisfied for both species only if , indicating that disequilibria will not be maintained in this model when rates of sex and recombination are sufficiently high in both species.

When the rate of sex and recombination is high relative to selection in the focal species but not in the interacting species, the QLE analysis predicts that rates of sex and recombination should decrease over evolutionary time in the focal species (*i.e*., Equation 22c is negative for modifier alleles with positive ). As the rate of genetic mixing decreases, however, the assumption that the focal species will be at QLE becomes tenuous. Indeed, simulations reveal that sex and recombination do not evolve to zero but to some positive level, which increases with the strength of selection (see Figure 3). To explore cases where selection is strong relative to the rate of sex and recombination, we follow Barton (1995) and study the dynamics of linkage disequilibria, just as we did in the one-species model.