## Abstract

Disruption of interactions among ensembles of epistatic loci has been shown to contribute to reproductive isolation among various animal and plant species. Under the Bateson–Dobzhansky–Muller model, such interspecific incompatibility arises as a by-product of genetic divergence in each species, and the Orr–Turelli model indicates that the number of loci involved in incompatible interactions may “snowball” over time. We address the combined effect of multiple incompatibility loci on the rate of introgression at neutral marker loci across the genome. Our analysis extends previous work by accommodating sex specificity: differences between the sexes in the expression of incompatibility, in rates of crossing over between neutral markers and incompatibility loci, and in transmission of markers or incompatibility factors. We show that the evolutionary process at neutral markers in a genome subject to incompatibility selection is well approximated by a purely neutral process with migration rates appropriately scaled to reflect the influence of selection targeted to incompatibility factors. We confirm that in the absence of sex specificity and functional epistasis among incompatibility factors, the barrier to introgression induced by multiple incompatibility factors corresponds to the product of the barriers induced by the factors individually. A new finding is that barriers to introgression due to sex-specific incompatibility depart in general from multiplicativity. Our partitioning of variation in relative reproductive rate suggests that such departures derive from associations between sex and incompatibility and between sex and neutral markers. Concordant sex-specific incompatibility (for example, greater impairment of male hybrids or longer map lengths in females) induces lower barriers (higher rates of introgression) than expected under multiplicativity, and discordant sex-specific incompatibility induces higher barriers.

GENE flow may persist between incipient species for extended periods, even in the face of severe disruptions in the fertility, viability, or behavior of hybrids or their descendants. For example, introgression appears to persist between *Drosophila pseudoobscura* and *D. persimilis*, in spite of the rise, ∼850,000 years ago (Leman *et al.* 2005), of a second-chromosome inversion associated with multiple mechanisms of reproductive isolation. In these species, at least three inverted chromosomal regions appear to contribute to hybrid incompatibility (Noor *et al.* 2001), and in general, the number of incompatible interactions among loci is expected to “snowball” as divergence time increases (Orr 1995; Orr and Turelli 2001). Here we address the effects of multiple incompatibility loci on the rate of introgression at neutral marker loci throughout the genome. This study departs from previous work (*e.g.*, Barton and Bengtsson 1986) by addressing sex specificity in expression of disruptions associated with hybridization, in rates of crossing over between incompatibility factors and marker loci, and in linkage to sex chromosomes or other regions transmitted in a sex-specific manner.

### Nonuniform divergence among genomic regions

Among the most robust patterns that have emerged from genome-scale surveys is the nonuniformity across genomic regions of divergence among populations. Geography constitutes a major explanatory variable for global patterns of variation in human genomes (Rosenberg *et al.* 2002; Coop *et al.* 2009). Humans show striking locus-specific *F*_{ST}, the classical index of population structure (Akey *et al.* 2002; Holsinger and Weir 2009), with many regions showing extreme values beyond three empirical standard deviations from the mean for a chromosome; (Weir *et al.* 2005). This pattern is observed in a wide variety of nonhuman species as well (*e.g.*, Rieseberg *et al.* 1999; Machado and Hey 2003; Teeter *et al.* 2009).

The widely used IM program (isolation with migration, Hey and Nielsen 2004, 2007) uses observations of nucleotide variation at multiple, independent loci to infer introgression rates and other demographic parameters assumed to be common to all genomic regions. Hey and Nielsen (2004) applied their method separately to each of 14 markers in *D. pseudoobscura* and *D. persimilis*, obtaining estimates of introgression rates that varied over orders of magnitude: for example, a scaled rate of flow from *D. persimilis* to *D. pseudoobscura* of *M*_{2} = 0.09 for *per* (X-linked) and *M*_{2} = 4.25 for *Adh* (autosomal). Earlier work (Wang *et al.* 1997) also indicated substantial differences between loci in rates of gene exchange.

Genetic analyses across interspecific hybridization zones provided the first demonstrations of the differential permeability of reproductive barriers across the genome (Barton and Hewitt 1985). Incompatibility is expected to impede introgression throughout the genome, but more intensely in regions harboring incompatibility factors (Bengtsson 1985; Barton and Bengtsson 1986; Navarro and Barton 2003). A number of studies have reported lower rates of introgression in regions shown to contain factors contributing to interspecific incompatibility. Using markers located on all 17 chromosomes of sunflowers *Helianthus annuus* and *H. petiolaris*, Rieseberg *et al.* (1999) found significantly higher divergence between the species in 26 regions, of which 16 were associated with reduced pollen fertility in hybrids. A more recent study of these species found numerous amino acid differences fixed between the species near breakpoints of inverted regions, where quantitative trait loci for hybrid pollen sterility tend to cluster (Strasburg *et al.* 2009). Lower introgression rates in the vicinity of known incompatibility factors have been detected between house mice *Mus musculus musculus* and *M. m. domesticus* (Payseur *et al.* 2004; Payseur and Nachman 2005), among host races of pea aphids (Via 2009), and between *D. pseudoobscura* and *D. persimilis* (Kulathinal *et al.* 2009).

One interpretation of nonuniform divergence is that genomic regions that show extreme variation among demes have experienced recent selective sweeps restricted to certain geographical localities (reviewed by Schlötterer 2003; Beaumont and Balding 2004; Storz 2005). This view holds that demographic structure should affect the pattern of variation in a uniform manner, with locus specificity indicative of selection (*e.g.*, Akey *et al.* 2002). Beyond sweeps, distinct species or local subpopulations of the same species may undergo genetic divergence through various processes, including adaptation to local ecological conditions (*e.g.*, Charlesworth *et al.* 1997).

Here, we use “incompatibility selection” to describe selective regimes characterized by neutrality within a specific demographic unit and negative selection elsewhere. Under this process, an incompatibility locus is generally monomorphic within populations for selectively equivalent forms of the wild-type allele, except for occasional introductions of deleterious foreign alleles, which are fully functional in their home deme. While incompatibility selection may derive from disruption of deme-specific ensembles of interacting loci formed during past episodes of selective sweeps, the present genomic state is nontransient, maintained by a balance between negative selection and introgression.

### Barriers to neutral gene flow

Bengtsson (1985) addressed the effect of interspecific incompatibility on introgression of regions linked to incompatibility loci, characterizing the “gene flow factor” as the probability that a newly arrived gene at a neutral marker locus will become incorporated into the local gene pool. Barton and Bengtsson (1986) characterized reproductive barriers due to multiple, epistatic incompatibility factors. Navarro and Barton (2003) used this framework to study the rate of accumulation of incompatibility factors between species. In the absence of sex specificity of any kind, loci unlinked to incompatibility factors face identical barriers to introgression and the joint barrier induced by multiple incompatibility factors corresponds to the product of the barriers generated by the factors individually.

Here, we address the implications for neutral introgression of sex specificity. Sex specificity may include differences between the sexes in fertility or viability of hybrids and their descendants, in rates of crossing over between a neutral marker locus and incompatibility factors, or in transmission of the marker or incompatibility loci (including sex linkage or sex-limited transmission). For clarity, we restrict consideration to a pair of hybridizing species.

Our results depart markedly from expectations in the absence of sex specificity. Reproductive barriers generally show epistasis, even in the absence of functional interactions among incompatibility factors. This form of epistasis in introgression rate reflects associations between sex and incompatibility factors and between sex and neutral markers. Such associations tend to generate locus-specific introgression rates, even in the absence of physical linkage of marker loci to incompatibility factors. In general, concordance among incompatibility factors in their relative expression between the sexes (for example, more severe effects in hybrid males than in females or greater crossover rates in females) generates lower reproductive barriers (greater introgression) than the multiplicative expectation and discordance generates higher barriers. Submultiplicativity of barriers arising under concordance appears to reflect the greater efficiency of selection (Haldane 1957; Hill and Robertson 1966; Barton 1995) in eliminating incompatibility factors that more severely impair the same sex.

## Theoretical Framework

Our major results fall into two classes. First, we describe the derivation of relative reproductive rate, the expected contribution to future generations of a foreign marker gene introduced by a migrant relative to a marker gene in a resident. Related to the gene flow factor of Bengtsson (1985), relative reproductive rate is the central quantity in a coalescent-based approximation to the evolutionary process at neutral sites in a genome subject to incompatibility selection. Second, we address the barrier to introgression generated by multiple incompatibility factors that show no functional epistasis. This analysis presents a partitioning of variation in relative reproductive rate of neutral marker genes on all possible backgrounds.

For simplicity, we use terminology applicable to species with chromosomal sex determination in which males are heterogametic. In the context of Table 1, we indicate the modifications required to address cases with heterogametic females.

### Genealogical migration rate

#### Effects of selection on gene flow:

In the population genetics literature, the backward migration rate generally refers to the proportion of the local gene pool that derives from the gene pool of a different deme in the immediately preceding generation; the forward migration rate denotes the rate at which genes contribute in the immediately succeeding generation to a gene pool different from the one in which they presently reside. In the presence of selection, the forward and backward rates differ, even under a time-reversible migration process. Further, contribution of introgressed genes to future generations depends not only on the rate of mixing of gene pools but also on the reproductive success of their carriers (Karlin 1982).

Selection can change the fundamental structure of the coalescent process at a target of selection (Krone and Neuhauser 1997; Neuhauser and Krone 1997) and influences the coalescent process at neutral sites linked to targets of selection. Of central importance to the latter process is the waiting time between migration events traced backward along the line of descent of a randomly sampled gene. Modeling the distribution of waiting time as exponential, we refer to the parameter of that distribution as the *genealogical migration rate*. Neutral substitution proceeds at rates expected in the absence of selection (Birky and Walsh 1988), with neutral divergence between demes dependent on the genealogical migration rate. Our index of introgression differs from earlier definitions that refer to the frequencies of species-specific marker alleles (*e.g.*, Barton 1979; Gavrilets 1997; Kobayashi *et al.* 2008).

#### Relative reproductive rate:

We model the waiting time between migration events traced backward along the line of descent of a neutral marker gene as exponential, assigning its parameter as(1)for *g* the genealogical migration rate in the deme from which the gene currently resides, *m* the backward migration rate, and ω the relative reproductive rate, representing the expected contribution to future generations of the marker gene. Relative reproductive rate reflects the number and expression of foreign incompatibility alleles in the background of the focal marker gene over many generations into the future.

At each of *n* incompatibility loci, let 1 denote the foreign incompatibility allele and 0 the local wild-type allele (the form that functions well in the local genome). A gene at the neutral marker locus may reside on 2^{n}^{+1} possible backgrounds, defined by sex (female or male) and the states of the alleles at the *n* incompatibility loci (1 or 0).

Females bearing the 1 incompatibility allele in heterozygous form at locus *i* and no other locus have fitness σ_{f}_{,}* _{i}* relative to females bearing only 0 alleles; σ

_{m}_{,}

*denotes the relative fitness of a male carrying the 1 allele only at locus*

_{i}*i*. We assume no functional epistasis among the incompatibility factors with respect to fitness: the fitness of carriers of multiple foreign incompatibility alleles at the same or different loci corresponds to the product of the σ-values of those alleles.

As in earlier work (*e.g.*, Bengtsson 1985), we assume a monotonic decrease in the number of foreign incompatibility alleles in the line of descent of a neutral marker gene traced forward in time. This assumption reflects the virtual absence within the local population of those deleterious genes; we address the implications of within-population polymorphism at incompatibility loci in a separate work (Fusco and Uyenoyama 2011).

At the point of zygote formation, **v** represents the vector of number of descendants of the focal marker gene on the 2^{n}^{+1} backgrounds. After a single generation, the expected distribution of descendants becomes **vC**, for **C** a product of matrices representing selection (**S**) on zygotes followed by genetic transmission of the marker across the possible backgrounds (**T**):(2)To establish a common basis for transmission of the focal marker gene through male and female carriers, we scale the contributions of males by the reproductive value of males (Fisher 1930). Because males transmit gametes to the next generation only to the extent that they succeed in fertilizing eggs, males have reproductive value *f*/(1 − *f*), for *f* the proportion of females among reproductives, relative to unity for females.

In a generation in the remote future, the expected total number of descendant genes is , for **e** the vector with all elements equal to 1. Relative to a marker gene on a pure local background, the contribution of a marker gene introduced by a migrant corresponds to(3)for **v _{m}** denoting the distribution of backgrounds of the marker gene in the hybrid offspring of the migrant and

**v**the analogous vector for a marker gene in offspring of a resident in the same generation as the hybrid. Kobayashi

_{r}*et al.*(2008) proposed a similar measure, which they called the “neutral effective migration rate.” Although they formulated their definition in terms of the frequency of an allele introduced by migrants, they noted its relationship to the history of a randomly sampled lineage.

As the number of generations into the future (*t*) grows large, the spectral radius λ of **C** comes to dominate (Equation 3). For diagonalizable **C**, λ corresponds to a simple, positive root of the characteristic equation and the set of eigenvectors that span the space (*Appendix A* addresses more general cases). Transmission of the focal marker genes through *t* generations is determined fromfor **D** a diagonal matrix with the eigenvalues of **C** along the diagonal, **Q** a matrix of right eigenvectors written as columns, and **Q ^{−1}** a matrix of left eigenvectors written as rows. For sufficiently large

*t*, λ

*comes to dominate the nonzero entries of*

^{t}**D**

*, which implies(4)for*

^{t}**q**the right eigenvector associated with the dominant eigenvalue λ of

**C**. In keeping with the Fisherian notion, we term

**q**(Equation 4) the

*reproductive value vector*, the

*i*th element of which represents the expected ultimate contribution to future generations of a marker gene on the

*i*th background.

In the Mathematica notebook (matrix builder.nb) provided in Supporting Information, File S1 and File S2, the matrices reflect an ordering first of females bearing backgrounds containing the maximum number of foreign (Equation 1) incompatibility alleles through successively fewer down to zero foreign alleles (resident background) and then of males bearing these backgrounds. In the case of introgression at an autosomal marker induced by incompatibility factor 1 alone, for example, the reproductive value vector (Equation 4) corresponds toin which η_{f,1} denotes the relative contribution to future generations of the marker gene borne by a female carrier of the foreign incompatibility allele relative to that of a noncarrier female or male (η_{f,0} = η_{m,0} = 1), η_{m,1} is the analogous quantity for a male carrier, and the prime is the transpose. Relative reproductive rate (Equation 4) corresponds toin which **v _{m}** = (

*f*, 0, 1 −

*f*, 0) and

**v**= (0,

_{r}*f*, 0, 1 −

*f*) reflect that the focal migrant and the focal resident each produce daughters and sons in proportions

*f*and (1 −

*f*).

*Appendix B*explicitly presents the matrices and the reproductive value vector

**q**(Equation 4) for the case of an autosomal marker gene linked to a single autosomal incompatibility locus.

We use ω (Equation 4), defined in terms of contribution to future generations, to approximate the backward-perspectived genealogical migration rate *g* (Equation 1). Results of simulation studies indicate excellent agreement between the expected and observed distributions of number of migration events traced backward along a randomly sampled lineage (Figure 2 and *Appendix C*). The Mathematica notebook matrix builder.nb (File S1 and File S2) constructs transmission matrix **C** and produces the reproductive value vector **q** (Equation 4) under user-specified values for the number of incompatibility loci, genomic locations of the marker and incompatibility loci, and sex-specific selection coefficients and recombination parameters.

### Partitioning of variation in long-term contribution

We assume the absence of functional epistasis among incompatibility loci in expression of the deleterious effects of incompatibility: the fitness of individuals bearing multiple foreign incompatibility alleles is equal to the product of the fitnesses induced by the alleles separately. Even so, our results indicate pervasive departures of the overall reproductive barrier from the product of the barriers induced by incompatibility factors individually. Here, we introduce a decomposition of variation in relative reproductive rate (Equation 4) in terms of interactions among incompatibility factors and sex.

#### Epistasis:

Expression (4) defines relative reproductive rate as a linear combination of elements of the reproductive value vector **q**, which gives the contributions to future generations of a marker gene on all possible backgrounds.

We explore the basis of departures from multiplicativity of the overall relative reproductive rate ω (Equation 3) at a neutral marker locus induced by multiple incompatibility factors.

We view reproductive value vector **q** (Equation 4) as a multiway table of a Fisherian factorial design experiment. Binary factors representing sex and the allelic state of the incompatibility loci affect the response variable of contribution to future generations of the focal marker gene.

Our index of epistasis corresponds to an analog of multilocus disequilibrium as proposed by Bennett (1954), in which, for example, the two-way association corresponds to a covariance:In the case at hand, we interpret *E*[*A _{i}*] as the expected relative contribution to future generations of the focal marker gene held by an individual bearing the foreign incompatibility allele at locus

*i*and no other locus,withthe expected long-term contribution in the presence of foreign incompatibility alleles at all loci having an index in the set Ω and no other loci. Superscripts specify the context: for example, denotes the relative reproductive rate at an X-linked marker gene introduced by a female migrant and the rate in the case of a male migrant.

Two-way epistasis reflects the departure from multiplicativity of barriers to introgression:*Appendix E* shows that for an autosomal marker (superscript A), *k*-way epistasis among *k* autosomal incompatibility loci corresponds to(5)for Ω_{[}_{k}_{]} a set of *k* indexes representing incompatibility loci, Ω_{[}_{j}_{]} a subset of Ω_{[}_{k}_{]} comprising *j* indexes, Ω_{[}_{k}_{]}\Ω_{[}_{j}_{]} the set of indexes in Ω_{[}_{k}_{]} but not in Ω_{[}_{j}_{]}, and(6)the departure of relative reproductive rate from multiplicativity. This partitioning of *k*-way epistasis includes components for all subsets of loci, giving a total of terms. It has the intuitively appealing property of zero epistasis for sets that include any neutral locus *j* (*E*[*A _{j}*] =

*ω*= 1), reflecting cancellation between terms of the form

_{j}*E*[…

*A*…] and .

_{j}We apply this partitioning of epistasis to relative reproductive rate (Equation 4) for an arbitrary number of incompatibility factors, assuming no linkage among the factors or the marker. Relative reproductive rate at an autosomal marker corresponds to(7)reflecting the equal transmission of the marker through the female and male hybrid offspring of migrants of either sex. Expression (7) also applies to an X-linked marker introduced by a female migrant (); for a male migrant, , reflecting that only female offspring carry the marker.

For cases in which relative reproductive rate corresponds to Equation 7, *Appendix E* shows that the overall departure from multiplicativity (Equation 6) decomposes into indexes of interaction between the foreign allele at incompatibility locus *i* and sex:(8)Positive Δ* _{i}* signifies a positive effect of femaleness: neutral marker genes borne by females contribute to future generations at higher rates than those borne by males. Negative Δ

*signifies that a marker gene on background*

_{i}*i*carried by a male contributes to future generations at a higher rate than a gene on the same background in a female. Sex-specific departures from multiplicativity correspond to(9a)(9b)

*Appendix E*shows that

## Results

### Barriers due to single sex-specific incompatibility factors

A major objective is to explore the effects of sex specificity on barriers to neutral introgression. *Appendix D* presents in detail the case of X-linkage of both the marker locus and an incompatibility locus, illustrating the modifications to transmission matrix **T** (Equation 2) to accommodate sex specificity. Here, we address the effect on the relative reproductive rate ω (Equation 4) of the genomic locations of the marker locus and of a single incompatibility locus.

Table 1 presents reproductive values at a marker locus in a genome together with a single incompatibility locus, with the loci at various locations in the genome, including autosomal (A), X-linked (X), Y-linked (Y), and mitochondrial (mt). The reproductive value columns list the relative long-term contribution to future generations of female (η_{f}) and male (η_{m}) carriers of the foreign incompatibility factor, and the factor–sex association column gives the difference between them (η_{f,1} − η_{m,1}; compare Equation 8). The final columns show the overall relative reproductive rates (Equation 4) of marker alleles introduced by female (ω^{f}) and male migrants (ω^{m}). For cases in which offspring of both sexes can transmit the marker and carry the incompatibility factor (A or X), the relative reproductive rate (Equation 4) of a female migrant corresponds to the average of the reproductive values of its female and male offspring:(compare Equation 7). In the case of an autosomal marker and an X-linked incompatibility factor, a marker allele introduced by a male migrant has higher relative reproductive rate,reflecting that sons of male migrants are free of the incompatibility factor. In the case of either an autosomal or an X-linked incompatibility factor, the relative reproductive rate at an X-linked marker allele introduced by a male migrant corresponds to η_{f} alone, reflecting that only daughters of male migrants carry the marker.

We assume that males do not transmit mitochondria, with the implication that incompatibility due to foreign mitochondria (mt) presents no barrier to autosomal or X-linked markers borne by a male migrant . Similarly, incompatibility factors on the X chromosome or mitochondria present no barrier to Y-linked markers because the marker and the factor are never transmitted to the same offspring.

Birds and various other organisms exhibit ZW sex determination, with male homogamety (ZZ) and female heterogamety (ZW). Reproductive values under this system correspond to those given in Table 1, with Z substituted for X and W for Y. Because complete cosegregation of mitochondria and the W chromosome constitutes in essence complete linkage between these regions, the reproductive values for cases involving a marker or a factor on mitochondria can be obtained from Table 1 by substituting mt for W.

### Barriers due to multiple incompatibility factors

We describe conditions under which the relative reproductive rate ω (Equation 4) induced by multiple incompatibility factors corresponds to the product of the relative reproductive rates induced by the factors individually. Beyond this case (absence of sex specificity in expression of incompatibility, crossover rates, or genetic transmission), the joint barrier to introgression engendered by multiple incompatibility loci departs in general from the product of the barriers induced by the factors in isolation. Epistasis in relative reproductive rate as defined in this section and in *Appendix E* reflects such departures from multiplicativity.

#### Multiplicative barriers to introgression:

In the absence of sex specificity, the Kronecker structure of the transition matrix **C** (Equation B5) for an autosomal marker extends to multiple incompatibility loci with multiplicative effects on the survival or reproduction of carriers. Here, we show that the total barrier to introgression at a neutral autosomal marker locus corresponds to the product of the barriers generated by the incompatibility loci individually under (1) sex-independent incompatibility selection and recombination, (2) independent transmission of the factors conditional on transmission of the marker, and (3) sex-independent genetic transmission.

We consider *l* incompatibility loci, for which the foreign allele at locus *i* reduces the viability and fertility of its carriers by a factor of σ* _{i}* relative to noncarriers. Crossing over occurs between the focal marker gene and incompatibility locus

*i*at rate

*r*, independently of crossing over in other regions. This assumption implies that at most two incompatibility loci (flanking the marker locus) can reside on the chromosome bearing the marker locus, with all other incompatibility loci unlinked (

_{i}*r*

_{f,}

*=*

_{j}*r*

_{m,}

*= ).*

_{j}The selection matrix (Equation B1) generalizes tofor withTransmission of the autosomal marker and sex retains the form (Equation B2). Independent transmission of the factors conditional on transmission of the marker impliesfor with(compare Equation B3). The full transition matrix corresponds to(compare Equation B5). The eigenvalues of correspond to products of the eigenvalues of the component matrices and its eigenvectors to Kronecker products of the eigenvectors of those matrices. In particular, elements of the reproductive value **q** (Equation 4) have the form(11)for Ω any subset of indexes in {1, … , *l*} and η_{f,}* _{i}* = η

_{m,}

*the reproductive value of female and males carriers of the foreign incompatibility allele at locus*

_{i}*i*. Consequently, the overall relative reproductive rate (Equation 4) corresponds to the product of the reproductive rates across the loci individually (Equation B6):

Multiplicative barriers also arise in the case of markers that are transmitted without recombination through only one sex: those on mitochondria or in the nonrecombining male- (female-)specific region of the Y (W) chromosome, for example. For markers transmitted only through females, the selection matrix (Equation B1) reduces to , with **S _{i}** containing only the female selection parameter σ

_{f,}

*, and the recombination matrix contains the crossover rates for females alone. As the transition matrix reduces to , the reproductive rate vector*

_{i}**q**has elements and the overall relative reproductive rate (Equation 4) is given by Equation 13, with the selection and recombination parameters corresponding to the values for females. In the absence of linkage between the marker and the

*i*th factor (

*r*

_{f,}

*= ), the*

_{i}*i*th multiplicand of Equation 12 reduces to(compare Equation B6). Complete linkage (

*r*

_{f,}

*= 0) implies a zero overall relative reproductive rate (no introgression at the marker).*

_{i}In this section, we have assumed the complete absence of sex specificity. The absence of interaction between any incompatibility locus and sex (Δ* _{i}* = 0 ∀

*i*) and of all sex-specific departures from multiplicativity () (Equation 11) as well as of the overall departure (Equation 12) implies the absence of epistasis of all orders.

#### Nonmultiplicative barriers to introgression:

Violation of any of the postulates enumerated in the preceding section induces a departure from multiplicativity of the barriers to introgression generated by more than one incompatibility locus. In this sense, the postulates represent minimal conditions for multiplicativity. To explore the nature and implications of nonmultiplicative barriers, we address the relative reproductive rate at a marker locus (autosomal or X-linked) subject to incompatibility generated by two loci: both autosomal or one autosomal and one X-linked.

#### Epistasis between autosomal factors with sex-specific expression or recombination:

We explore the effects of sex specificity in expression of incompatibility or in crossover rates on the rate of introgression at an autosomal marker flanked by autosomal incompatibility loci 1 and 2.

Conditional on transmission of the marker, transmission from female parents of the two factors occurs independently of each other,(13)for *r*_{f,}* _{i}* the rate of crossing over in females between the marker and incompatibility locus

*i*. A similar transmission matrix holds for male parents, with

*r*

_{m,}

*, the crossover rate in males, substituted for*

_{i}*r*

_{f,}

*. In the absence of physical linkage,*

_{i}*r*

_{f,}

*=*

_{i}*r*

_{m,}

*= . The foreign incompatibility allele at locus*

_{i}*i*reduces the viability of its female (male) carriers by a factor of σ

_{f,}

*(σ*

_{i}_{m,}

*), with multiplicative effects in carriers bearing foreign alleles at both loci (*

_{i}*e.g.*, σ

_{f,12}= σ

_{f,1}σ

_{f,2}).

Migrants of either sex produce female and male hybrid offspring in frequencies *f* and 1 − *f*. The relative reproductive rate at the marker locus (Equation 4) corresponds to(14)for η_{f,12} and η_{m,12} the contribution to future generations of the marker gene in females and males that bear the foreign factor at both incompatibility loci.

Our index of epistasis (Equation 10) among incompatibility loci reduces in this case to(15)in which(16)(compare Equation B9) andIn this case, the sex-specific measures of epistasis are proportional to each other:

(17)These autosomal factors undergo independent transmission (Equation 13) and cause no functional epistasis with respect to fitness of carriers. Furthermore, each in isolation induces the relative reproductive rate given by Bengtsson (1985), with the average crossover and viability rates substituted for the non-sex-specific parameters (Equation B7). Even so, the joint barrier to introgression departs in general from the product of the barriers induced by each locus separately (Equation 15).

To explore the nature of epistasis due to sex specificity in the impairment of carriers of foreign incompatibility loci, we determined the relative reproductive rate (Equation 4) at an autosomal marker induced by two autosomal incompatibility factors in the absence of physical linkage (*r*_{f,}* _{i}* =

*r*

_{m,}

*= for*

_{i}*i*= 1, 2 in Equation 16). In Figure 1, the relatively flat plane corresponds to the expectation under multiplicative reproductive barriers, while the saddle-shaped surface shows the actual joint barrier, with heavy lines marking the only points of agreement between the actual and expected barriers. This example illustrates the general finding that incompatibility factors with concordant relative effects on the sexes (for example, both afflicting male carriers to a greater extent than female carriers) induce lower barriers to neutral introgression than the multiplicative expectation, and factors with discordant effects induce higher barriers.

To verify the accuracy of our expression for the genealogical migration rate *g* (Equation 1), we conducted a numerical simulation study using SFS CODE (Hernandez 2008), modified to identify the lineage involved in each migration event. We specified an autosomal marker gene and two autosomal incompatibility loci under multiplicative fitnesses and no linkage among any of the loci. We simulated 100,000 generations forward in time and then counted the number of migration events in a randomly sampled line of descent traced backward in time. We compared the counts observed in 15,000 independent replicate runs to a Poisson distribution with the parameter determined by Equation 1 under the population parameters specified in the simulation (Figure 2 legend). Figure 2 indicates an excellent fit of the simulated data to our prediction. It also provides another illustration that incompatibility factors with concordant effects on the sexes induce lower barriers than expected under multiplicativity (Figure 2, left) and factors with discordant effects induce higher barriers (Figure 2, right).

Sex-specific recombination rates can generate epistasis even in the absence of sex specificity in carrier fitness (σ_{f,}* _{i}* = σ

_{m,}

*= σ < 1). In this case, Equation 16 reduces to*

_{i}Rates of introgression at the marker exceed those expected under multiplicative barriers if crossing over between the marker and each of the flanking incompatibility factors occurs at higher rates in females than in males (Δ_{1}, Δ_{2} > 0) or if both rates are higher in males (Δ_{1}, Δ_{2} < 0).

#### X-linked marker with autosomal and X-linked incompatibility factors:

We address barriers generated by one A and one X incompatibility locus to introgression at an X-linked marker introduced by a female migrant (*Appendix F* gives the expressions for a male migrant). We assume no crossing over between the X-linked factor and the marker in (hemizygous) males (*r*_{m,}* _{X}* = 0).

Epistasis in relative reproductive rate (Equation 10) between the X-linked and autosomal incompatibility loci corresponds to(18)Table 1 gives the index of association between the autosomal factor and sex,(19)and between the X-linked factor and sex,(20)Within-sex interactions correspond to(21a) (21b)giving an overall two-way epistasis between the incompatibility factors of (22)in which

Because a male transmits its entire X chromosome (the marker together with any foreign incompatibility allele) to all daughters and no sons,(Table 1). Accordingly, Equation 20 indicates that the X-linked factor obstructs transmission of the X-linked marker through males more than through females unless the X-linked factor causes complete sterility or inviability in females (σ_{f,X} = 0), has no deleterious effects in males (σ_{m,X} = 1), or shows absolute linkage to the marker (*r*_{f,X} = 0). The nature of epistasis, sex specific (Equation 21) as well as overall (Equation 22), depends on the effect of the autosomal factor . Expression (19) indicates positive epistasis (greater introgression than expected under multiplicativity) if the autosomal factor impairs male carriers to a greater extent (σ_{f,A} > σ_{m,A}).

#### Autosomal marker with autosomal and X-linked incompatibility factors:

We now consider introgression at an autosomal marker in a genome containing one A incompatibility locus, possibly linked to the marker, and one X locus. We assume a female migrant (*Appendix F* provides the expressions for a male migrant).

The overall two-way epistasis (Equation 10) between the incompatibility factors corresponds to (23)for(24)(compare to Equations 16 and 19), for *D*_{f} an index of association between sex and the X-linked factor (Equation F4) and Γ a positive quantity (*Appendix F*). The joint barrier to introgression departs from multiplicativity only if the autosomal factor shows an interaction with sex , reflecting sex-specific expression of incompatibility (σ_{f,A} ≠ σ_{m,A}) or sex-specific rates of crossing over with the marker (*r*_{f,A} ≠ *r*_{m,A}). Whether the overall epistasis is positive or negative depends on the nature of the female-specific epistasis between the incompatibility factors ( ∝ *D*_{f} (Equation F4), which in turn depends on parameters of both the autosomal and X-linked factors.

Indexes *D*_{f} and *D*_{m} (Equation F4) function as indicators of the effect of sex on the barrier to transmission of the autosomal marker locus induced by the X-linked incompatibility factor: positive values indicate higher transmission of the marker through female than through male carriers of the incompatibility factor. In general, *D*_{m} ≤ 0, signifying that femaleness reduces transmission by carriers of the X-linked incompatibility factor, with no effect (*D*_{m} = 0) only if the factor does not in fact contribute to incompatibility (σ_{f,X} = σ_{m,X} = 1). This property may reflect that while female carriers pass the X-linked incompatibility factor to offspring of both sexes, male carriers pass it only to daughters.

Another indication of the intrinsic association of an X-linked incompatibility factor with sex is that it can affect the sign of the overall epistasis even if it exhibits no sex specificity in expression of incompatibility or crossing over (σ_{f,X} = σ_{m,X} and *r*_{f,X} = *r*_{m,X}). In this case, the autosomal marker has lower reproductive rates in female carriers of the X-linked incompatibility factor than in male carriers (*D*_{f}, *D*_{m} < 0).

## Discussion

We have explored the implications of sex-specific interspecific incompatibility for neutral introgression across the genome. We proposed a scaling of the backward migration rate (Equation 3) that accounts for sex-specific incompatibility selection and linkage and showed that the distribution of waiting times along a random lineage traced backward conforms well to the distribution determined by this scaling (Figure 2). Among the key qualitative findings are that (1) neutral introgression generally occurs at locus-specific rates, even in the absence of linkage of marker loci to incompatibility factors and (2) the joint barrier generated by multiple incompatibility factors is generally nonmultiplicative, even in the absence of functional epistasis among the incompatibility factors.

Here, we provide a qualitative discussion of these results, illustrate some implications for the interpretation of patterns of genetic variation in closely related Drosophila species, and suggest that sex specificity is a pervasive feature of interspecific hybridization in plants and animals.

### Locus specificity in reproductive barriers reflects associations with sex

Factors contributing to interspecific incompatibility impede introgression of neutral markers at a rate that depends on the level of linkage, selection intensity, and functional epistasis among incompatibility factors (Bengtsson 1985; Barton and Bengtsson 1986; Navarro and Barton 2003). In the absence of sex specificity, the locus-specific nature of barriers to introgression extends only to regions immediately adjacent to targets of selection. For example, Table 1 indicates that in the absence of linkage with a neutral marker (*r*_{f} = *r*_{m} = ), a single factor transmitted through both sexes that induces non-sex-specific incompatibility (σ_{f} = σ_{m} = σ ≤ 1) either imposes no barrier to introgression at the marker or reduces it by a factor of σ/(2 − σ) [compare the “gene flow factor” (Equation F6) of Bengtsson 1985]. Factors borne on the mitochondria also have this effect, while Y-linked factors impede introgression on autosomes by a factor of 1/(2 − σ). In contrast, sex-specific incompatibility induces locus-specific barriers even at unlinked marker loci.

Sex specificity in expression of incompatibility, transmission, or level of linkage to incompatibility factors impedes introgression at neutral markers to unequal extents across genomic regions, even in the absence of functional epistasis. Figure 3 illustrates, for a female or male migrant, relative reproductive rates (Equation 4) at neutral marker loci at various locations in the genome (color-coded bars) with one or multiple incompatibility loci at locations indicated on the abscissa, assuming free crossing over (*r* = ) between the marker and any incompatibility factor. Because a male migrant never transmits both its X and Y chromosomes to the same offspring, an incompatibility factor on one chromosome poses no barrier to introgression of the other chromosome. Accordingly, Figure 3 shows relative reproductive rates of 1 for (magenta bar for an incompatibility factor only on the X) and (cyan bar for an incompatibility factor only on the Y), regardless of sex specificity. In all other cases, the differences in heights of the bars for a given set of locations for incompatibility factors arise as a consequence of sex specificity in intensity of incompatibility selection or transmission of the marker or incompatibility loci.

Locus-specific reproductive barriers of this kind reflect associations between sex and incompatibility factors and between sex and marker loci (last section of *Results* and *Appendix E*). Because neutral markers borne on X chromosomes or on mitochondria, for example, descend preferentially or exclusively through females, they experience incompatibility primarily in a female context. If male hybrids experience greater impairment, female-associated markers face lower barriers to introgression than male-associated markers. A Y-linked incompatibility factor introduced by a male migrant imposes no barrier to introgression at an X-linked marker because only its daughters carry the marker, but it does impede introgression at an autosomal marker, which accompanies the Y-linked factor in half of the sons of a male carrier. In Table 1, the factor–sex association column shows a measure of interaction (Equation 8) between a single incompatibility factor and sex in determining relative reproductive rate ω (Equation 4).

Differences between the sexes in levels of recombination have consequences similar to those of sex-specific expression of incompatibility. Greater rates of crossing over in females than in males, for example, permit greater introgression of female-associated markers linked to incompatibility factors.

### Epistasis in relative reproductive rate

Our analysis of the joint barrier to neutral introgression induced by multiple incompatibility factors assumes the absence of functional epistasis: the fitness of individuals bearing multiple factors corresponds to the product of the fitnesses induced by the factors individually. Under equal impairment of male and female carriers of foreign incompatibility factors, transmission of incompatibility factors on autosomes, and equal rates of crossing over between the sexes, we confirm the multiplicativity of the barriers (Equation 12), as commonly assumed. These conditions appear to be minimal, as violation of any one causes departures from multiplicativity (*Nonmultiplicative barriers to introgression* section).

In general, discordance among incompatibility factors in the nature of association with sex tends to increase barriers to introgression. For example, Figure 1 depicts the relative reproductive rate of an autosomal marker in a genome containing two incompatibility loci with sex-specific expression (σ_{f,i} ≠ σ_{m,i}) in the absence of linkage between any pair of loci. It illustrates that the highest barriers (lowest ω-values, bluer regions) derive from a positive interaction (Equation 8) with femaleness in one factor (Δ* _{i}* > 0) and a negative interaction in another factor (Δ

*< 0). Positive epistasis arises under concordant associations (Δ*

_{j}*Δ*

_{i}*> 0), implying higher rates of gene flow (redder regions). For extreme manifestations of Haldane’s rule (σ*

_{j}_{f}= 1, σ

_{m}= 0 or σ

_{m}= 1, σ

_{f}= 0), the actual rate of introgression is nearly twofold greater than expected under multiplicative reproductive barriers (red corners in Figure 1).

Rapid elimination of the deleterious foreign incompatibility factors permits greater neutral introgression. The selection process is more efficient if the factors tend to occur together (Hill and Robertson 1966; Barton 1995): in the same sex, in the present context. Concordance of the sex-specific effects of multiple incompatibility factors (greater impairment of male carriers than of female carriers, for example) implies their association with the same sex.

### Interspecific incompatibility in Drosophila

Llopart *et al.* (2005) studied patterns of nucleotide divergence and polymorphism in sister species *D. yakuba* and *D. santomea* in 29 genomic regions, including sites on the X chromosome, the Y chromosome, autosomes, and mitochondria. Regions on the X and Y contribute to severe reductions in hybrid male fertility (Coyne *et al.* 2004) and quantitative trait loci contributing to both prezygotic and postzygotic isolation occur on the X chromosome and the autosomes (Moehring *et al.* 2006a,b). Llopart *et al.* (2005) found significantly lower ratios of shared to fixed polymorphisms between the species on the X chromosome than on autosomes, which they considered consistent with reduced introgression of X-linked sites due to the disproportionate contribution of the X to hybrid male sterility. To account for the difference in effective number of genes under X and autosomal linkage and its effects on levels of polymorphism, they also conducted coalescent simulations under IM-based (Hey and Nielsen 2004) estimates of demographic parameters. However, this approach indicated no significant reduction in introgression rate at X-linked relative to autosomal sites.

In confirmation of the expectations of Llopart *et al.* (2005), our model indicates that incompatibility due to a single X-linked factor permits greater introgression in autosomal than in X-linked regions,regardless of the intensity of incompatibility selection (σ_{f,X}, σ_{m,X}), the rate of crossing over on the X (*r*_{f,X}), or the sex of the initial migrant (Table 1). The blue (autosomal marker) and cyan (X-linked marker) bars in Figure 3 for the case of a single X-linked factor (X on the abscissa) illustrate this effect.

In contrast, incompatibility due to a single autosomal factor introduced by a male migrant permits greater introgression in X-linked regions than in autosomal regions unlinked to the factor (*r*_{f} = *r*_{m} = ):[for example, blue (A) and cyan (X) bars in Figure 3 over A for a male migrant]. For autosomal factors introduced by a female migrant, higher introgression in X-linked regionsholds for factors that impair male carriers more than female carriers (σ_{f} > σ_{m}).

Many aspects of incompatibility between *D. yakuba* and *D. santomea* show sex specificity, with multiple factors on the X chromosome, the Y chromosome, and autosomes contributing to hybrid male sterility (Coyne *et al.* 2004; Moehring *et al.* 2006a). Accordingly, we expect the overall barrier to introgression to show sex-related epistasis. To illustrate the implications, we compare the relative reproductive rates at X-linked and autosomal markers induced by incompatibility factors both on the X chromosome and on an autosome unlinked to the autosomal factor (*r*_{f,A} = *r*_{m,A} = ). For the case in which the X-linked factor causes complete sterility in hemizygous form (σ_{m,X} = 0), with maximal crossing over in females (*r*_{f,X} = ; *r*_{m,X} = 0 in hemizygous males), our model indicates uniformly higher introgression rates in X-linked than in autosomal regions for the case of a male migrant . For female migrants, with σ_{m,X} = 0,This expression implies equal barriers to introgression on the autosome and the X chromosome if either factor expresses dominant lethality or sterility (σ_{f,A}σ_{f,X}σ_{m,A} = 0). Otherwise, greater introgression is expected at X-linked than at autosomal sites if the effects of the autosomal factor conform to Haldane’s rule (σ_{f,A} > σ_{m,A}).

Contrary to the expectation of Llopart *et al.* (2005), these results indicate that sex specificity in crossing over and in postzygotic incompatibility can induce a *greater* barrier to introgression in autosomal regions unlinked to any incompatibility factor than in regions on the X that are loosely linked (*r*_{f} = ) to an incompatibility factor. The presence of an incompatibility factor on the Y, as reported by Coyne *et al.* (2004), would reinforce this trend by inhibiting introgression of autosomes but not the X chromosome because foreign X-linked markers and the foreign Y never occur in the same genome.

Even under parameter values for which (*e.g.*, Figure 3), positive epistasis due to concordant sex-specific effects among factors (for example, greater impairment of male hybrids or higher rates of crossing over in females) may cause the joint barrier to exceed the multiplicative expectation considerably (Figure 1). In such cases, the introgression rate in unlinked autosomal regions may only slightly exceed the rate on the X, consistent with the nonsignificant differences reported by Llopart *et al.* (2005).

Unlike the expectation in the absence of sex specificity, Figure 3 illustrates marked differences in introgression rates among markers freely recombining with incompatibility loci. Our analysis of locus-specific introgression rates may contribute toward a basis for inferring the existence and location of incompatibility factors from the pattern of neutral variation throughout the genome. In particular, the ensemble response of marker loci in the four genomic regions shown in Figure 3 differs among models for the location of incompatibility loci and also between the sexes of the migrants within a given model.

### Sex-specific transmission or expression of incompatibility factors

As sex specificity in expression of interspecific incompatibility or in crossover rates has been widely observed, our findings raise the possibility of pervasive locus-specific neutral divergence among species.

Lenormand and Dutheil (2005) have reviewed studies documenting differences in map length between males and females in both plants and animals, including humans (Li *et al.* 1998). In Drosophila, the premiere model organism for the experimental investigation of interspecific incompatibility, crossing over is suppressed altogether in males (Morgan 1914).

Assessment of factors contributing to postzygotic interspecific incompatibility that have been identified to the level of operons confirms the traditional view that speciation arises as a by-product of divergence of all manner of genes (reviewed in Orr *et al.* 2006; Araripe *et al.* 2010). While the homeobox gene *Ods* induces sterility in male hybrids between *D. simulans* and *D. mauritiana* (Ting *et al.* 1998), the oncogene *Xmrk* contributes to tumor formation in backcross Xiphophorus hybrids of both sexes (Schartl 2008), suggesting that sex-specific incompatibility is common but not universal.

Many of the iconic traits associated with intrinsic postzygotic isolation (Coyne and Orr 2004) affect reproduction, often showing sex-limited expression or different expression in male and female hybrids. Haldane’s (1922) rule holds in a number of animals, including Drosophila, in which hybrid males tend to suffer much more severe postzygotic incompatibility than do hybrid females (Coyne and Orr 2004, Chap. 8). Direct experiments have demonstrated that introgression of chromosomal segments from *D. mauritiana* into a *D. simulans* background (True *et al.* 1996; Tao *et al.* 2003) or into a *D. sechellia* background (Masly and Presgraves 2007) induces male sterility at rates severalfold higher than those of female sterility or inviability in both sexes.

Numerous studies have described pervasive sex specificity in gene regulation and expression (Michalak and Noor 2003; Ranz *et al.* 2003; Civetta and Singh 2006). Table 1 of Ranz *et al.* (2003) indicates that of the 4776 coding sequences surveyed, >61% showed sex-biased expression in *D. melanogaster*. Further, they found that genes expressed at higher levels in males showed significantly greater divergence in expression levels between *D. melanogaster* and *D. simulans*.

Together with Haldane’s (1922) rule, the disproportionately large effect of the X chromosome constitutes a major rule of speciation (Coyne and Orr 1989). Experiments introgressing small segments between Drosophila genomes indicate that a higher proportion of (hemizygous) introgressions on the X chromosome induce hybrid male sterility than do homozygous autosomal introgressions (True *et al.* 1996; Tao *et al.* 2003; Masly and Presgraves 2007). A number of workers have provided discriminating assessments of leading proposals for evolutionary mechanisms that could contribute to the large-X effect (True *et al.* 1996; Tao *et al.* 2003; Coyne and Orr 2004; Presgraves 2008). Whether this major trend reflects faster fixation of advantageous variants on hemizygous X chromosomes than on autosomes (Charlesworth *et al.* 1987; Kirkpatrick and Hall 2004), accelerated divergence of genes affecting sex or reproduction (Civetta and Singh 1998), segregation distortion of sex chromosomes (Tao *et al.* 2001), or other processes, the large-X effect itself suggests that a substantial proportion of factors contributing to interspecific incompatibility may generate locus-specific barriers to neutral introgression of the kind studied here.

### Local adaptation within structured populations of conspecifics

Incompatibility selection may arise not only as a consequence of interspecific hybridization, but also through divergence due to local adaptation of subpopulations of the same species. Table 1 indicates that autosomal incompatibility factors reduce introgression of mitochondrial markers by a factor of σ_{f}/(2 − σ_{f}) and of Y-linked markers by a factor of σ_{m}/(2 − σ_{m}). This finding suggests that sex-specific differences in expression of locally adapted alleles might contribute to differences in divergence at mitochondrial and Y-linked markers (*e.g.*, Seielstad *et al.* 1998; Oota *et al.* 2001; Ségurel *et al.* 2008), even if the propensity to migrate and mating success of migrants are independent of sex.

Polymorphisms for incompatibility factors within demes might be expected under incompatibility derived from local adaptation within species. In a separate work, we explore the effects of such characteristics on rates of introgression of neutral markers throughout the genome (Fusco and Uyenoyama 2011).

## Acknowledgments

We gratefully acknowledge the constructive comments of Associate Editor Naoyuki Takahata and the anonymous reviewers and support for the National Evolutionary Synthesis Center (NESCent) working group on genomic introgression. Public Health Service grant GM 37841 (to M.K.U.) provided partial funding for this research.

## Appendix

### Appendix A: Relative Reproductive Rate

Because **C**, describing genetic transmission and selection (Equation 2), is nonnegative, its spectral radius λ corresponds to a nonnegative characteristic value with modulus exceeded by no other eigenvalue and the eigenvector associated with λ is nonnegative (Gantmacher 1959, Chap. XIII, section 3). It has a Schur decomposition of the formfor **U** a triangular matrix with the eigenvalues of the transformation along the diagonal (see, for example, Golub and Van Loan 1996). For convenience, we choose **Q** such that the eigenvalues appear along the diagonal in order of their moduli, with λ the first element. For *t* sufficiently large, λ* ^{t}* dominatesimplying(compare Equation 4), for

**q**the column or the sum of the columns of

_{0}**Q**that correspond to λ.

### Appendix B: Single Autosomal Incompatibility Factor With an Autosomal Marker

To clarify the connection to earlier work, we address the case of an autosomal marker gene linked to a single autosomal incompatibility locus, for which a marker allele can reside on 4 = 2^{2} possible backgrounds.

Beginning with the distribution of backgrounds at the point of formation of the hybrid or resident offspring (**v _{m}** or

**v**in Equation 4), selection first reduces the frequency of carriers of the foreign incompatibility factor, with female carriers surviving and reproducing at rate σ

_{r}_{f}and males at rate σ

_{m}relative to individuals bearing no foreign incompatibility allele. Further, we scale the contributions of males by the Fisherian male reproductive value

*f*/(1 −

*f*). The selection matrix corresponds to(B1)in which the ordering of the elements corresponds to females bearing background 1 and 0 at the incompatibility locus and males bearing these backgrounds. Carriers of the focal marker gene of either sex transmit it to female and male offspring at rate ,(B2)and transmission at the incompatibility locus given transmission of the marker gene is represented by(B3)in which (1 −

*r*

_{f}) represents the probability that a female bearing the foreign incompatibility factor transmits it together with the focal marker gene and (1 −

*r*

_{m}) the analogous quantity for male carriers. Joint transmission at the marker and incompatibility loci follows (B4)(B4)for ⊗ the Kronecker product.

In the absence of sex specificity in rates of crossing over (*r*_{f} = *r*_{m} = *r*) and in expression of incompatibility (σ_{f} = σ_{m} = σ), the transition matrix corresponds to(B5)The rightmost matrix, representing incompatibility selection and the conditional transmission of the incompatibility factor, has a dominant eigenvalue of unity, the right eigenvector corresponding to which isThe remaining matrix product, representing scaling of male contributions and transmission of the focal marker gene, has dominant eigenvalue *f*, with right eigenvectorAs the Kronecker product of these matrices, **C** has dominant eigenvalue *f* (the product of the eigenvalues of the matrices) with right eigenvectorRelative reproductive rate (Equation 4) corresponds to (B6)(B6)This expression, which has been derived numerous times (*e.g.*, Equation A3 in Navarro and Barton 2003), corresponds to the “gene flow factor” of Bengtsson (1985) and “barrier strength” of Barton and Bengtsson (1986).

Under sex specificity in selection (Equation B1) or transmission (Equation B3), the relative reproductive rates of female and male carriers correspond toimplying a relative reproductive rate at the marker locus of (B7)(B7)for(B8)(compare Equation B6).

In general, sex of the initial migrant and the incompatibility factor show epistasis with respect to the long-term contribution of the focal marker gene to future generations (compare Equation 8): (B9)(B9)No epistasis (η_{f} = η_{m}) arises in the absence of sex specificity either in expression of incompatibility (σ_{f} = σ_{m}) or in recombination (*r*_{f} = *r*_{m}). For a given total magnitude of selection (σ_{f} + σ_{m}), (η_{f} − η_{m}) increases with the excess viability or fertility of female over male carriers (σ_{f} – σ_{m}). Similarly, for a given total magnitude of recombination (*r*_{f} + *r*_{m}), (η_{f} – η_{m}) increases with the excess crossing over in female over male carriers (*r*_{f} – *r*_{m}).

### Appendix C: Validation of the Neutral Approximation

Among our main theses is that the evolutionary process at neutral marker loci in a genome containing incompatibility factors is well approximated by a purely neutral process with a forward migration rate equal to our backward migration rate (Equation 1). This section presents confirmation of this proposal, on the basis of results of numerical simulations generated by SFS CODE (Hernandez 2008), modified to track migration events along lineages at a neutral marker locus.

Our neutral approximation holds that the number of migration events along a lineage follows a Poisson distribution with the parameter equal to the product of the number of generations observed and the per-generation backward migration rate (Equation 1). This construction implies that the waiting time between migration events along a lineage has an exponential distribution with the parameter equal to the backward migration rate. We assumed multiplicative effects of foreign incompatibility alleles, whether segregating at the same locus or at different loci.

We conducted forward simulations over *t* = 50,000 generations, specifying a two-deme model with equal population sizes (*N*_{1} = *N*_{2} = 1000 genes or 500 diploid individuals) and symmetric forward migration rates (*N*_{1}*m*_{12} = *N*_{2}*m*_{21} = 0.1). We assumed that the foreign incompatibility allele at each of two autosomal incompatibility loci reduces the viability of its carrier by a factor of σ = 0.65 and that migrants compete for mates on an equal basis with residents. In the absence of linkage among any of the two incompatibility loci and an autosomal marker locus, these assumptions imply a relative reproductive rate (Equation 4) of ω = 0.4815 and a genealogical migration rate (Equation 1) of *g* = 4.8148 × 10^{−5} at the neutral marker. A total of 36,233 migration events over 15,000 independent replicates showed an excellent fit (*χ*^{2} = 3.946, 9 d.f.) to expectations under a Poisson distribution with parameter *gt* = 2.4074.

Because relative reproductive rate (Equation 3) represents a limit for large numbers of generations and negligible frequencies of foreign incompatibility alleles, one might expect discrepancies to arise in cases with forward migration rates sufficiently high to permit the migration of an individual carrying a foreign incompatibility allele back into the other species. To explore the limits of our model, we repeated the simulation experiment under forward migration rates an order of magnitude higher (*N*_{1}*m*_{12} = *N*_{2}*m*_{21} = 1.0). We found that 7.5% of migration events traced backward along a lineage involved backgrounds that differed from the pure resident background, compared to 0.087% for the lower forward migration rate. The distribution of the number of migration events along a random lineage departs significantly from a Poisson distribution (*χ*^{2} = 90.4898, 32 d.f.): the observed mean (24.451) exceeds the expectation (*b _{i}t* = 24.074) and the kurtosis value indicates excess mass in the tails. To provide a quantitative summary, we fit the observations to a Neyman’s type A distribution, a compound distribution that assumes that the Poisson parameter is itself a Poisson-distributed random variable,for ϕ an index of clumping (see Johnson and Kotz 1969, Chap. 9). The observations show an excellent fit (

*χ*

^{2}= 19.898, 24 d.f.) to a Neyman’s type A distribution with mean set equal to the observed mean and a least-squares estimate of the clumping parameter ϕ of 0.0141, obtained by minimizing the squared deviation between the observed variance up to the ninth moment and their expectations, using the probability generating function given in Johnson and Kotz (1969).

### Appendix D: X-Linkage

We address the case of X-linkage of both the marker locus and a single incompatibility locus in an organism exhibiting X–Y sex determination with no crossing over on the X chromosome in hemizygous males.

As in the autosomal case, female carriers transmit the focal marker gene symmetrically to female and male offspring:In contrast, male carriers transmit the marker gene to all daughters, without recombination, and to no sons, with the last two rows of **T** corresponding to(compare Equation B4).

For a female migrant, the relative reproductive rate (Equation 4) becomesin whichThese expressions imply epistasis (Equation 8) between sex and the X-linked incompatibility locus proportional to(compare Equation B9). This expression indicates a positive interaction between femaleness and the foreign incompatibility factor under incomplete linkage (*r*_{f} > 0), regardless of whether male or female carriers suffer greater effects of incompatibility [sign of (σ_{f} − σ_{m})]. This association may reflect higher recombination rates in females (*r*_{f} > *r*_{m} = 0).

For a male migrant, only female descendants receive the focal marker allele, implying a relative reproductive rate of

### Appendix E: Epistasis Among Incompatibility Factors

Here we describe a measure of epistasis among incompatibility factors (compare Bennett 1954), applied to the relative reproductive rate (Equation 4) for an arbitrary number of incompatibility factors.

To establish Equation 5, we expand(E1)for Ω_{[}_{k}_{]} (*k* ≥ 2) a set of *k* indexes representing *k* incompatibility loci, Ω_{[}_{j}_{]} a set of *j* indexes in Ω_{[}_{k}_{]} (*j* ≤ *k*), Ω_{[}_{k}_{]}\Ω_{[}_{j}_{]} denoting the set of indexes in Ω_{[}_{k}_{]} but not in Ω_{[}_{j}_{]}, and representing the departure of relative reproductive rate from multiplicativity (Equation 6).

For relative reproductive rate corresponding to Equation 7, we address the relationship between the overall departure from multiplicativity (Equation 6) and the sex-specific measures (Equation 9). Using that η_{f,}* _{i}* = ω

*+ Δ*

_{i}*and η*

_{i}_{m,}

*= ω*

_{i}*− Δ*

_{i}*for Δ*

_{i}*= (η*

_{i}_{f,}

*− η*

_{i}_{m,}

*)/2 (Equation 8), we have(E2)for representing terms of the form Δ*

_{i}*. Note that*

_{i}In the expansion of Equation E2, terms involving an odd number of the Δ* _{i}*’s cancel out, while all the terms with an even number appear twice, implyingfor Ω

_{[2}

_{i}_{]}representing all subsets of Ω

_{[}

_{j}_{]}that contain 2

*i*elements. We rewrite Equation E1 as

We find thatFor *k* odd, this sum corresponds toand for *k* even,Substitution of these expressions into Equation E3 produces Equation 10.

### Appendix F: Epistasis Between an X-Linked and an Autosomal Factor

#### X-linked marker (male migrant)

Because only female offspring of a male migrant receive the X-linked marker,with overall epistasis equal to the female-specific epistasis,for given in Equation 21a. As in the case of the female migrant, the sign of epistasis depends on (Equation 19).

#### Autosomal marker

For a *female migrant*, epistasis (Equation 10) between the incompatibility loci corresponds to(F)for given by Equation 26 and(F2)(Table 1). Within-sex interactions correspond to(F3a) (F3b)(F3b)in which (F4)(F4)giving an overall two-way epistasis between the incompatibility factors of Equation 23. Comparison of Equations 17 and F3 suggests that *D*_{f} and *D*_{m} assume the role of Δ_{X}, an indicator of the nature of the interaction between sex and the X-linked incompatibility factor.

As indicated in the text following Equation 24, *D*_{m} is always nonpositive and zero only if the X-linked factor does not contribute to incompatibility selection (σ_{f,X} = σ_{m,X} = 1). In contrast, *D*_{f} is positive for (F5)(F5)Sufficient conditions for positive epistasis (higher introgression) include that both factors cause greater impairment to female carriers (σ_{f,A} < σ_{m,A} and σ_{f,X} < σ_{m,X}) or that both cause sufficiently greater impairment to males, with positive Δ_{A} and Equation F5 satisfied. In the absence of sex specificity in expression of incompatibility due to the X-linked factor (σ_{f,X} = σ_{m,X}), *D*_{f} is nonpositive, implying either no epistasis orindicating negative epistasis (less introgression) if the map distance between the autosomal marker and the autosomal factor is greater in females.

For a *male migrant*, all female offspring and none of the male offspring of the migrant carry the X-linked factor, while offspring of both sexes carry the autosomal factor:

Epistasis between the loci with respect to relative reproductive rate is given bywith given by Equation 24, the within-female departure from multiplicativity by Equation F3a, and

As in the case of a female migrant (Equation 23), epistasis arises only if the autosomal factor is sex specific . Nonzero (η_{f,X} − 1) implies that the X-linked factor contributes to incompatibility (σ_{f,X} < 1 or σ_{m,X} < 1). For a male migrant, the overall value for epistasis between the autosomal and X-linked incompatibility factors isfor *D*_{m} and Γ given in Equation F4. As *D*_{m} is negative if the X-linked factor contributes to incompatibility (σ_{f,X} < 1 or σ_{m,X} < 1), this expression indicates that the autosomal and X-linked factors show positive epistasis (higher introgression rates than expected under multiplicativity) if the autosomal factor permits greater reproductive rates through male than through female carriers and negative epistasis if female carriers show higher reproductive rates.

## Footnotes

*Communicating editor: N. Takahata*

- Received May 14, 2011.
- Accepted June 19, 2011.

- Copyright © 2011 by the Genetics Society of America

Available freely online through the author-supported open access option.