## Abstract

*Q*_{ST} measures the differentiation of quantitative traits between populations. It is often compared to *F*_{ST}, which measures population differentiation at neutral marker loci due to drift, migration, and mutation. When *Q*_{ST} is different from *F*_{ST}, it is usually taken as evidence that selection has either restrained or accelerated the differentiation of the quantitative trait relative to neutral markers. However, a number of other factors such as inbreeding, dominance, and epistasis may also affect the *Q*_{ST} − *F*_{ST} contrast. In this study, we examine the effects of dominance, selection, and inbreeding on *Q*_{ST} − *F*_{ST}. We compare *Q*_{ST} with *F*_{ST} at selected and neutral loci for populations at equilibrium between selection, drift, mutation, and migration using both analytic and simulation approaches. Interestingly, when divergent selection is acting on a locus, inbreeding and dominance generally inflate *Q*_{ST} relative to *F*_{ST} when they are both measured at the quantitative locus at equilibrium. As a consequence, dominance is unlikely to hide the signature of divergent selection on the *Q*_{ST} − *F*_{ST} contrast. However, although in theory dominance and inbreeding affect the expectation for *Q*_{ST} − *F*_{ST}, of most concern is the very large variance in both *Q*_{ST} and *F*_{ST}, suggesting that we should be cautious in attributing small differences between *Q*_{ST} and *F*_{ST} to selection.

TWO measures of population differentiation are commonly used to assess the genetic structure of populations. The first of these is *F*_{ST}, which measures the differentiation of neutral markers between populations. *F*_{ST} is generally calculated using information on allele frequencies at neutral loci such as microsatellites and is therefore an estimate of the amount of differentiation that has occurred due to drift acting on the (finite) populations. *F*_{ST} is well characterized, with a vast literature on how factors such as mutation, migration, population size, population history and population subdivision affect it (Whitlock and McCauley 1999) and how best to calculate and estimate it (Weir and Cockerham 1984; Weir and Hill 2002). In addition, numerous studies have calculated *F*_{ST} for natural populations to investigate, for instance, effective population size, migration rates, and divergence times as well as examine population differentiation itself.

The second widely used measure of population differentiation is that of quantitative traits, termed *Q*_{ST}. Because knowledge of the underlying genetic loci (of which there may be dozens) is generally unknown, quantitative traits are measured on individuals and *Q*_{ST} is calculated in terms of variation in those traits between and within populations. Assuming an additive model, Wright (1951) derived expressions for the neutral expectation of variance between and within populations, which Lande (1992) noted may be translated into an *F*_{ST} equivalent to measure the differentiation of a quantitative trait. The origin of the term *Q*_{ST} is attributed to an article the following year (Spitze 1993).

In addition to the factors affecting neutral markers, quantitative traits may be under selection. In the absence of selection, the differentiation of an additive trait is equal to the differentiation of neutral markers (that is, *Q*_{ST} = *F*_{ST}), as both sets of loci are affected equally by drift (Lynch and Spitze 1994; Latta 1998). However, if selection has accelerated the divergence of quantitative traits between the populations (for instance, different phenotypes are selected for in different environments, leading to local adaptation of populations), *Q*_{ST} will exceed *F*_{ST}. Alternatively, *Q*_{ST} will be less than *F*_{ST} if quantitative traits are restrained by balancing selection (for instance, if natural selection favors the same phenotype in different populations). The comparison of *Q*_{ST} and *F*_{ST}, when measured for the same populations (Crnokrak and Merila 2002; Whitlock 2008), can therefore be used to infer the type and strength of selection on the quantitative trait. Many dozens of studies have now used the *Q*_{ST} − *F*_{ST} contrast to predict the effect of selection (see references in Leinonen *et al.* 2008), and the conclusion from two meta-analyses is that *Q*_{ST} generally exceeds *F*_{ST} (Merila and Crnokrak 2001; Leinonen *et al.* 2008), supporting a general role for divergent selection driving the differentiation of quantitative traits.

There are two problems when comparing *Q*_{ST} and *F*_{ST} values to elucidate the role of selection. The first is related to how *F*_{ST} and particularly *Q*_{ST} are measured and the intrinsic biases in calculating them using samples of natural, finite populations, despite a number of sophisticated methods available to calculate both (O'Hara and Merila 2005). For example, Whitlock (2008, p. 1894) asserts that *Q*_{ST} is “difficult to estimate accurately and precisely,” while O'Hara and Merila (2005, p. 1337) conclude from simulations that “the precision of the *Q*_{ST} estimates—irrespective of the estimation method used—is very low.” Thus, when comparing *Q*_{ST} and *F*_{ST}, it is not particularly clear how to best assess the significance of an observed difference or correct for biases in their estimates.

The second area of concern when comparing *Q*_{ST} and *F*_{ST} is in the nature of the quantitative trait itself. In contrast to neutral loci, alleles at quantitative trait loci may exhibit dominance, may interact with other loci via epistasis or extended linkage disequilibrium, or may respond differently than neutral alleles to the effects of mutation. A reasonable research effort has recently been directed toward how these factors affect *Q*_{ST} and *Q*_{ST} − *F*_{ST}, particularly investigating the changes to *Q*_{ST} − *F*_{ST} when *F*_{ST} is calculated using allele frequencies at the quantitative locus itself. Models of epistasis suggest that *Q*_{ST} will be depressed by interactions between loci (Whitlock 1999; Lopez-Fanjul *et al.* 2003), masking the effect of divergent selection on the trait. Additionally, because multiple loci contribute to a quantitative trait and allele effects sum across loci, mean values of a quantitative trait may not be different between populations even if divergence at individual loci is high. Conversely, low levels of divergence at the allelic level may have significant impacts on the quantitative trait means (Latta 1998; Le Corre and Kremer 2003). Thus, the signatures of divergent or uniform selection may be masked if allele frequencies at the quantitative trait are measured (or modeled), rather than the trait values themselves.

Using an island model and a pure drift model respectively, Goudet and colleagues (Goudet and Buchi 2006; Goudet and Martin 2007) and Lopez-Fanjul and colleagues (Lopez-Fanjul *et al.* 2003, 2007) have extensively investigated the impact of dominance on *Q*_{ST} − *F*_{ST} for a neutral trait. For the island model at equilibrium between migration and drift, *Q*_{ST} tends to be less than *F*_{ST} calculated at the same locus (Goudet and Buchi 2006; Goudet and Martin 2007), while if populations diverge via drift from an infinite population, *Q*_{ST} tends to be larger than *F*_{ST} when the recessive allele is at high frequency (Lopez-Fanjul *et al.* 2007). These results led Goudet and Buchi (2006) to suggest that dominance may mask the effects of divergent selection when *Q*_{ST} is compared to *F*_{ST} calculated from neutral markers. To date, however, there has been no exploration of the combined effect of selection, dominance, and inbreeding on *Q*_{ST} and *Q*_{ST} − *F*_{ST}. In this article, we examine *Q*_{ST} values at equilibrium to address whether divergent selection on quantitative traits may indeed be hidden by the effects of dominance. *F*_{ST} is calculated both at the selected locus and at neutral loci unlinked to the locus under selection. The numeric results from the analytic approach are checked and extended by simulations, in which multiple loci under divergent and balancing selection are also investigated. The following sections are arranged into (i) the derivation of and results from recursion equations used to calculate *F*_{ST} and *Q*_{ST}, (ii) a description of and results from an individual-based simulation model, and finally (iii) a general discussion. Parameters and terms for the following *F*_{ST} and *Q*_{ST} models are summarized in Table 1.

## THE ANALYTIC MODELS

We consider the simple case of two populations, *X* and *Y*, linked by migration. Diploid individuals within a population outcross or self to form the next generation, and some of these offspring migrate so that a proportion *m _{i}* of individuals in population

*i*(

*i*=

*X*,

*Y*) are migrants. We assume that migrants are indistinguishable from residents in reproduction, generations are discrete, and population sizes and structure are constant through time.

Let *N _{i}* be the number of individuals in population

*i*(

*i*=

*X*,

*Y*) and let the probability of selfing in both populations be

*f*while the probability of parents outcrossing is (1 −

*f*). Selfing and outcrossing are assumed to be independent, and so the number of selfed and outbred offspring per parent is independently binomially distributed with means

*f*and (1 −

*f*), respectively.

#### Population differentiation at neutral markers (*F*_{ST}):

To derive an explicit expression for *F*_{ST}, we construct recursion equations for coefficients describing the similarity between and within individuals in a subdivided population. These recursion equations may then be used to solve for *F*_{ST} at equilibrium. Full details of the derivation are shown in appendix a. If we make the assumption that migration rates and population sizes are equal in populations *X* and *Y* (*m _{X}* =

*m*=

_{Y}*m*and

*N*=

_{X}*N*=

_{Y}*N*), the explicit expression for

*F*

_{ST}is(1)where the effective population size

*N*

_{e}is defined in appendix a.

If migration rates are small, *F*_{ST} is expected to be(Maruyama and Tachida 1992; Wang 1997). Interestingly, using overestimates the value of *F*_{ST} found in (1). This is likely to be the result of the small number of populations we are considering. *F*_{ST} increases with increasing selfing rate and rises sharply with decreasing migration rates and population sizes.

#### Population differentiation at quantitative traits (*Q*_{ST}):

Following previous work (Goudet and Buchi 2006), we consider a single locus with two alleles, *A* and *B*, at frequencies *p _{i}* and 1 −

*p*, respectively, in population

_{i}*i*(

*i*=

*X*,

*Y*). We set the genotypic value

*G*of an individual to be −1 (

*G*), 1 (

_{AA}*G*), and

_{BB}*d*(

*G*) for genotypes

_{AB}*AA*,

*BB*, and

*AB*, respectively; we assume

*d*> 0 so that allele

*B*is dominant to allele

*A*. Genotype frequencies

*r*of

*AA*,

*AB*, and

*BB*in population

*i*(

*i*=

*X*,

*Y*) with selfing rate

*f*are shown in Table 2.

Following Heywood (2005), we assume that selection acts via a linear fitness function, so that the fitness of an individual in population *i* with genotypic value *G _{jk}* is(2)where

*s*is the population-specific selection coefficient. The mean fitness of population

_{i}*i*isand the relative fitness of an individual becomesWe assume that selection dominates drift , and so drift is not explicitly modeled in our equations.

Following selection and reproduction, the allele frequencies in the next generation (*t*) areFinally, after migration, the frequency of allele *A* in the next generation for population *X* becomes(3)A similar expression for *p _{Y}*

_{(t)}can be found by switching

*X*and

*Y*subscripts in the above equation.

To find the equilibrium frequencies, we solve *p _{X}*

_{(t)}=

*p*

_{X}_{(t−1)}and

*p*

_{Y}_{(t)}=

*p*

_{Y}_{(t−1)}. If the parameters are known, the equilibrium values of

*p*and

_{X}*p*may be easily found. However, it is not possible to find a general expression for

_{Y}*p*and

_{X}*p*at equilibrium. Even solving for a reduced set of equations, when we know some of the parameters of the model, proves to be difficult. For a given set of parameter values, the expressions for

_{Y}*p*(3) and

_{X}*p*can therefore be iterated over generations until there is no change in allele frequencies and the populations reach equilibrium. A polymorphic equilibrium with both alleles present in both populations requires that selection coefficients act in opposite directions in the two populations, that there is migration between the populations, and additionally that . For example, gives

_{Y}*p*= 0.3823 and

_{X}*p*= 0.6181, while and lead to

_{Y}*p*=

_{X}*p*= 1 as the high migration rate from population

_{Y}*Y*to

*X*causes the positive selection in population

*Y*to dominate both populations. Interestingly, if

*f*= 0,

*d*= 0,

*m*=

_{X}*m*, and

_{Y}*s*= −

_{X}*s*, then

_{Y}*p*+

_{X}*p*= 1, and both alleles are maintained in populations

_{Y}*X*and

*Y*[that is, ].

The expression for *Q*_{ST} including selfing rate *f* is(4)(Bonnin *et al.* 1996), where *V*_{B} is the variance between populations and *V*_{A} is the additive variance within populations. For a single quantitative locus with two alleles, Goudet and Buchi (2006) derived a general expression for the trait variance among *n* populations. For our case of two populations and genotypic values of −1, *d*, and 1 (see Table 2), this expression becomes(5)Similarly, for two populations the expression for the single-locus additive variance *V*_{A} within the populations (Templeton 1987; Lynch and Walsh 1998; Goudet and Buchi 2006) becomes(6)As demonstrated by Goudet and Buchi (2006), the expression for *Q*_{ST} reduces to the definition of *F*_{ST} (Wright 1951) at the locus when there is no dominance,(7)where for the case of two populations *X* and *Y*and

The value of *Q*_{ST} at equilibrium may be found by substituting equilibrium allele frequencies and known population parameters into our expression for *Q*_{ST} (Equation 4). For example, solving *p _{X}*

_{(t)}=

*p*

_{X}_{(t−1)}and

*p*

_{Y}_{(t)}=

*p*

_{Y}_{(t−1)}for gives equilibrium allele frequencies and

*Q*

_{ST}= 0.9570, while gives equilibrium allele frequencies and

*Q*

_{ST}= 0.0555.

#### Effects of dominance, inbreeding, and selection on *Q*_{ST}:

When calculated using equilibrium allele frequencies, *Q*_{ST} is a strictly increasing function of inbreeding, dominance, and the strength of divergent selection and decreases with increasing migration between populations. Note that equilibrium allele frequencies are themselves functions of the inbreeding, dominance, and selection coefficients. Under divergent selection, dominance has the greatest impact on *Q*_{ST} when inbreeding is low. For example, Figure 1 demonstrates that when , equilibrium *Q*_{ST} increases dramatically with increasing inbreeding and also increases with increasing dominance at the locus. This nonadditive interaction between dominance and inbreeding was also noted by Goudet and Buchi (2006) for a neutral locus.

Very small changes in the selection coefficients have a much larger impact on *Q*_{ST} than do equivalent changes to the dominance coefficient or selfing rates. This suggests that even a low level of divergent selection will overwhelm the effects of dominance or inbreeding on *Q*_{ST}. For example, we investigated the balance between increasing the dominance coefficient and decreasing the selection coefficient to maintain the same value of *Q*_{ST}. Assuming a migration rate of , a population size of *N* = 1000, and an inbreeding coefficient , the initial values for the selection coefficients (*s _{X}* = −

*s*) were chosen so that [calculated from (4) and (1), respectively] when

_{Y}*d*= 0, that is, when selection is approximately the same order of magnitude as drift. For ,

*s*= (0.0016, 0.0015, 0.0013) to maintain

_{X}*Q*

_{ST}(=

*F*

_{STN}) = 0.1169. So an increase in the dominance coefficient from to 1 is offset by a decrease in the strength of selection of only 0.0002. Therefore, although we have seen that dominance and inbreeding indeed influence equilibrium

*Q*

_{ST}values, divergent selection is a much more significant force in determining the level of quantitative trait differentiation.

*Q*_{ST} − *F*_{STQ} contrast:

For a single locus with two alleles, the value of *Q*_{ST} − *F*_{ST} is found by substituting equilibrium allele frequencies into Equations 4–6 for *Q*_{ST} and Equation 7 for *F*_{ST}. The full expression for the difference is shown in appendix b. For a neutral quantitative locus, *Q*_{ST} = *F*_{STQ} (*F*_{ST} for the *q*uantitative locus) only when there is no dominance (Goudet and Buchi 2006). If alleles at the locus do not act additively, the difference between *Q*_{ST} and *F*_{STQ} may be positive or negative, depending on the allele frequencies. For example, in the one-locus, two-allele model with dominance, if the recessive allele is rare in both populations, then *Q*_{ST} will be less than *F*_{STQ}, and vice versa if the recessive allele is at high frequency in both populations (Goudet and Buchi 2006). In general, the expected value for the difference between *Q*_{ST} and *F*_{STQ} is dependent on assumptions made about the distribution of neutral allele frequencies for a quantitative trait and the mode of population subdivision (see discussion in Goudet and Buchi 2006, Goudet and Martin 2007, and Lopez-Fanjul *et al.* 2007 and references therein).

When a dominant quantitative locus is under divergent selection, however, the range of possible allele frequencies in the two populations will be more restricted than in the neutral case. In particular, we no longer expect the frequency of the recessive allele to be positively correlated between populations, as the allele favored in one population is selected against in the other. Therefore, in contrast to the neutral case, it is unlikely that the same allele will be rare in both populations.

To test the impact of dominance with divergent selection on the difference between *Q*_{ST} and *F*_{STQ}, we iterated the recursion equations for *p _{X}* (Equation 3) and

*p*to find equilibrium allele frequencies for 10,000 random combinations of biologically feasible parameter values [ while ] and calculated

_{Y}*Q*

_{ST}and

*F*

_{STQ}at equilibrium. The distribution of

*Q*

_{ST}−

*F*

_{STQ}for the 10,000 random combinations is shown in Figure 2. We additionally restricted parameters to biologically realistic values [, , and ], which affected the magnitude of

*Q*

_{ST}−

*F*

_{STQ}, but the total frequency of

*Q*

_{ST}−

*F*

_{STQ}values less than zero was similar (data not shown). Significantly, these results demonstrate that for a range of feasible migration, selection, inbreeding, and dominance coefficients

*Q*

_{ST}will most likely exceed

*F*

_{STQ}when both are calculated using equilibrium allele frequencies at the locus under selection. Therefore, we conclude that dominance is unlikely to hide the signature of divergent selection on the

*Q*

_{ST}−

*F*

_{ST}contrast, and in fact the inflation of

*Q*

_{ST}relative to

*F*

_{ST}due to dominance will make it easier to detect divergent selection on the quantitative trait.

To assess the impact of individual factors, we calculated the correlation of *d*, *f*, *s _{X}*,

*s*,

_{Y}*m*, and

_{X}*m*with

_{Y}*Q*

_{ST}−

*F*

_{STQ}and with for the 10,000 random combinations of parameter values described above. Interestingly, there is a significant (

*P*< 0.001) positive correlation between

*d*and both

*Q*

_{ST}−

*F*

_{STQ}and (correlations 0.3449 and 0.5804), a significant negative correlation between

*Q*

_{ST}−

*F*

_{STQ}and

*f*,

*m*, and

_{X}*m*(correlations −0.3216, −0.2806, and −0.3211), and a significant negative correlation between and

_{Y}*f*(correlation −0.4750). These correlations indicate that, regardless of the strength of selection and the rates of migration and selfing, as dominance increases the difference between

*Q*

_{ST}and

*F*

_{STQ}also increases. In addition, with other factors constant, increasing the selfing rate or the migration rates decreases the difference between

*Q*

_{ST}and

*F*

_{STQ}. For loci under divergent selection, then, we expect the largest difference between

*Q*

_{ST}and

*F*

_{STQ}when selfing rates are small and the dominance effect is large. This result agrees with the conclusion for neutral loci from Goudet and Buchi (2006): the effect of dominance becomes smaller as inbreeding increases.

We also iterated allele frequencies to equilibrium for a range of dominance and inbreeding coefficients given . Figure 3A shows that, as expected from the correlation described above, there is a positive relationship between and the dominance coefficient. Interestingly, however, for the given parameter values, increases as the selfing rate approaches 0.3 and then decreases (Figure 3B). Both the numerator and the denominator in the expression for *Q*_{ST} − *F*_{STQ} involve *f*^{4} terms (appendix b), so it is unsurprising to see such a nonlinear effect on the difference. When we plot *Q*_{ST} − *F*_{STQ} instead of , the relationship with *d* and with *f* looks very similar to those seen in Figure 3, A and B, respectively. These graphs indicate that in fact it may not be a low but a moderate level of inbreeding, coupled with strong dominance, that is likely to give the largest positive *Q*_{ST} − *F*_{STQ} difference.

#### Nonequilibrium *Q*_{ST} − *F*_{STQ}:

Populations may not be at equilibrium when they are sampled, for example, when populations are adapting to a new selection pressure. Therefore, we used our recursion equations (3) to investigate the change in allele frequencies and in *Q*_{ST} − *F*_{STQ} as two populations adapt to a new selection pressure and stabilize toward equilibrium values. We found that in general, allele frequencies and *Q*_{ST} moved initially very rapidly toward equilibrium values, but that the rate of change slowed considerably when populations approach the equilibrium values.

The initial allele frequencies determine how the value of *Q*_{ST} − *F*_{STQ} changes over time. For example, if , , and , then our expected equilibrium values for *p _{X}*,

*p*, and

_{Y}*Q*

_{ST}−

*F*

_{STQ}are 0.2351, 0.8937, and 0.0249, respectively. The average equilibrium frequency of allele

*A*across populations is 0.5644. Figure 4 demonstrates the change in the value of

*Q*

_{ST}−

*F*

_{STQ}over time when the initial frequencies are (bottom line) and (top line). These graphs are typical of the change in

*Q*

_{ST}−

*F*

_{STQ}when initial (bottom line) and initial (top line). If initial allele frequencies are approximately equal and are less than the average frequency of allele

*A*at equilibrium,

*Q*

_{ST}−

*F*

_{STQ}will initially decrease and become more negative and then increase toward the equilibrium

*Q*

_{ST}−

*F*

_{STQ}. However, if initial allele frequencies are greater than the average equilibrium frequency,

*Q*

_{ST}−

*F*

_{STQ}will always be positive.

Interestingly, these results suggest that when divergent selection pressure is first applied, the value of *Q*_{ST} − *F*_{STQ} can become progressively more negative. In addition, a negative value of *Q*_{ST} − *F*_{STQ} may persist for a substantial number of generations before a positive value is reached. The speed of the transition from negative to positive *Q*_{ST} − *F*_{STQ} is predominantly determined by the strength of selection. As expected, however, *Q*_{ST} is a strictly increasing function of time, so when comparing *Q*_{ST} to *F*_{ST} measured at neutral loci, the value of *Q*_{ST} − *F*_{STN} will always be positive and will increase as *Q*_{ST} approaches its equilibrium value.

## THE SIMULATION MODEL

Our numeric results from the analytic approach suggest that dominance is likely to enhance the difference between *Q*_{ST} and *F*_{ST} for a quantitative trait under divergent selection, and therefore selection will be more easily detected. To test this prediction, and to incorporate the effect of drift, we simulated the genotypes of diploid individuals in two populations linked by migration. For a number *l* of unlinked loci, genotypes are initially randomly assigned to individuals in each population by assuming that all *a* alleles at a locus are equifrequent. A number of loci contribute to the quantitative trait values and are therefore selected, while the remainder act as neutral marker loci. Individuals randomly mate or self to produce offspring of the next generation. Offspring migrate with probability *m* to the other population. These offspring then join the parent pool for the following generation, and the process of mating and migration iterates through generations. This design closely follows the analytic model, with mating followed by migration, discrete generations, and stable population sizes over time.

We ran our simulation program for a number of different scenarios to investigate the impact both divergent and balancing selection have on the *Q*_{ST} − *F*_{ST} contrast. For each of five simulation sets (A–E), we considered six designs combining high and low levels of inbreeding and low, medium, and high dominance coefficients. In addition, we assume equal migration rates and population sizes (*m _{X}* =

*m*=

_{Y}*m*= and

*N*=

_{X}*N*=

_{Y}*N*= 1000). For all loci, mutation from one allele to any other occurs at equal, relatively high, probability μ to avoid fixation of alleles due to drift. Under each of the six designs, simulations were run for 2000 generations and replicated 100 times, and

*Q*

_{ST}and

*F*

_{ST}values were averaged over replicates. In the following sections, we first describe the methods used to calculate

*Q*

_{ST}and

*F*

_{ST}in our simulations and then present the results from the five simulation sets (A–E).

#### Calculating *F*_{ST} and *Q*_{ST} in simulations:

##### Calculating F_{ST}:

The simulation calculates *F*_{ST} separately at neutral loci (*F*_{STN}) and at the loci under selection (*F*_{STQ}) to allow comparison of the *Q*_{ST} − *F*_{ST} contrast both at selected and at neutral loci. Using allele frequencies in each population following migration, *F*_{ST} at each generation is calculated as(8)(Wright 1951), where(9)(10)and[recall that *i* indexes the population (*i* = *X*, *Y*), *l* is the number of *l*oci, and *a* is the number of *a*lleles at each locus]. *p _{ijk}* is the allele frequency of allele

*k*at locus

*j*in population

*i*and

*p*is the average allele frequency of allele

_{jk}*k*at locus

*j*across populations

*X*and

*Y*. Note the small change of notation from our one-locus, two-allele model (Table 2); we number allele frequencies at locus

*j*in population

*i*as

*p*

_{ij}_{1},

*p*

_{ij}_{2},

*p*

_{ij}_{3}, … , for alleles

*A*

_{1},

*A*

_{2},

*A*

_{3}, … , instead of frequencies

*p*and (1 −

*p*) for alleles

*A*and

*B*.

##### Calculating Q_{ST}:

For *Q*_{ST}, a small number of diallelic loci are assumed to contribute to the quantitative trait, and genotypic values are assigned by summing over loci. For each of *l* loci, marginal genotypic values for *AA*, *AB*, and *BB* are −1/*l*, *d*/*l*, and 1/*l*, where *d* is the level of dominance. We scale by the number of loci *l* so that the minimum and maximum genotypic values (−1 and 1) remain constant. On the basis of the genotypic value of an individual, selection is then applied to transform genotypic values into relative fitness. Individuals with higher fitness are more likely to be selected as parents for the next generation. If only one locus is under selection, we may use a linear fitness function as in the analytic approach (2) and model either divergent (*s _{i}* of different sign) or uniform (

*s*of the same sign) selection. For one or more loci, we also model optimal selection acting on the total genotypic value over loci. Following Turelli (1984) and Le Corre and Kremer (2003), we define the absolute fitness of an individual as(11)where

_{i}*G*

_{i}_{OPT}is the local genotypic optimum in population

*i*and γ > 0 is the strength of selection. As γ increases, the strength of selection decreases. Uniform and divergent selection are modeled by setting similar or different values for the local genotypic optima. Note that for the optimal selection regime, more than two populations and more than two alleles at the loci under selection may easily be included in the simulation design, but we restrict our simulations to the two-population, two-allele case for simplicity and comparison with the analytic approach.

After a large number of generations, the within- and between-population variances needed to calculate *Q*_{ST} approach asymptotic values. Variances are calculated either from population allele frequencies or from observed genotypic values of individuals in each population. For example, using trait values, *V*_{A} may be estimated as twice the covariance between offspring and midparent genotypic values or four times the covariance of half sibs [provided the level of inbreeding is not large and the dominance variance is small (Cockerham and Weir 1984)]. Similarly *V*_{B} may be calculated from the difference between the trait means of the populations. However, for the two-allele case we can use the exact expressions for *V*_{A} and *V*_{B} calculated from allele frequencies [(6) and (5), respectively]. For more than one locus, this expression becomes(12)where we sum the between- and within-population variances for each locus across the *l* loci before calculating *Q*_{ST}.

We compared the expectations for quantitative trait differentiation when *Q*_{ST} is calculated from (i) allele frequency information [using (5) and (6)] or (ii) phenotypic measures and covariances between parents and offspring as described above. We ran 1000 replicates of our simulation model for the simple case of no inbreeding and no dominance, with selection coefficients and migration rates chosen randomly from their biologically feasible ranges [ and ]. The number of individuals in each population was set to 1000, and the simulation was run for 2000 generations. As expected when *d* = 0, *Q*_{ST} and *F*_{ST} values were identical when both are calculated using allele frequencies. The means of *Q*_{ST} calculated from allele frequencies (0.4444) and *Q*_{ST} calculated from phenotypic values (0.4461) were very similar, as were their variances (0.1376 and 0.1375, respectively). However, calculating *Q*_{ST} from phenotypic values introduces additional sampling error to the value. For example, as there is no dominance, the correlation between *F*_{ST} and *Q*_{ST} calculated from allele frequencies is 1, while the correlation of *F*_{ST} and *Q*_{ST} calculated from phenotypes was 0.9985. Similarly, if we run the simulation under the same conditions but additionally select dominance and inbreeding coefficients randomly from their biologically feasible ranges and 3000 replicates], the correlation between *F*_{ST} and *Q*_{ST} calculated from allele frequencies is 0.9929, while the correlation between *F*_{ST} and *Q*_{ST} calculated from phenotypes is 0.9501. Therefore, using phenotypic values to calculate *Q*_{ST} would make the contrast between *F*_{ST} and *Q*_{ST} more variable and any difference between them less likely to detect. Results of the following simulations are therefore presented with values of *Q*_{ST} calculated from allele frequencies.

#### Simulation set A—differentiation via divergent linear selection:

To verify the predictions of our analytic model, we first ran the simulation with one locus under divergent selection using the linear fitness function described in Table 2. Selection strength was set at . Table 3 compares the expected analytic values (exp), for *Q*_{ST}, *F*_{STQ} (*F*_{ST} at the selected *q*uantitative locus), and *F*_{STN} (*F*_{ST} for nine *n*eutral loci) with the average results from our simulations (sim) for each design. Exp values are calculated using Equations 1 (*F*_{STN} exp), 4 (*Q*_{ST} exp), and 7 (*F*_{STQ} exp) after iterating recursion equations to equilibrium. Sim values are based on final allele frequencies from the simulations and calculated using Equations 8 (*F*_{STN} sim, *F*_{STQ} sim) and 12 (*Q*_{ST} sim).

In general, the simulated values agree well with those from our analytic models. There are some surprisingly large differences between the expected and the simulated values for *F*_{STN} for high rates of selfing. This appears to be due to a large proportion of replicates fixing different alleles in the two populations across many of the neutral loci (and hence *F*_{STN} ≈ 1). Therefore, despite explicitly including drift in our expression for *F*_{ST} (Equation 1), the combination of high rates of selfing and low migration has reduced *N*_{e} further than we predicted. Setting a higher mutation rate for neutral loci for these particular designs would certainly have prevented such loss of alleles and this elevation in *F*_{STN} simulated values.

##### Q_{ST} − F_{STQ}:

The average simulation results for *Q*_{ST} − *F*_{STQ} are very similar to the expected values, and, as expected, all averages are very close to zero at low dominance (*d* = ). Figure 5A demonstrates the close correlation between average expected (solid lines) and simulated values (dashed lines) at the three levels of dominance when the selfing rate is low. However, the variation in the value of *Q*_{ST} − *F*_{STQ} between replicates in many cases exceeds its expectation (data not shown). In addition, we can see that in many cases a high proportion of individual replicates may have negative *Q*_{ST} − *F*_{STQ} values. For example, for the medium dominance, low-inbreeding design we expect *Q*_{ST} − *F*_{STQ} = 0.0202, and although the average *Q*_{ST} − *F*_{STQ} from simulations is close to expectation, nearly a quarter of replicates have *Q*_{ST} − *F*_{STQ} < 0.

To investigate the impact of the large variation of *Q*_{ST} and *F*_{ST} further, we ran a set of 1000 simulations. Dominance, inbreeding and selection coefficients, and migration rates were chosen randomly from their biologically feasible ranges and population sizes fixed (*N _{X}* =

*N*= 1000). Figure 6 demonstrates that the distribution of

_{Y}*Q*

_{ST}−

*F*

_{STQ}differs significantly from the distribution generated from our analytic model (Figure 2). Thus, although we generally expect

*Q*

_{ST}>

*F*

_{STQ}on the basis of our results from the recursion equations, the simulations indicate that in real populations the effect of sampling (

*i.e*., drift) may hide any elevation of

*Q*

_{ST}due to dominance. This result agrees with the discussion in Goudet and Buchi (2006) and Goudet and Martin (2007) regarding the large variance of

*Q*

_{ST}and

*F*

_{STQ}for a neutral quantitative locus.

As predicted from Figure 2 and the analytic expectations, Table 3 and Figure 5A indicate that there is a positive relationship between *Q*_{ST} − *F*_{STQ} and the dominance coefficient: within each of the two levels of selfing there is a general trend for increased difference between *Q*_{ST} and *F*_{STQ} as dominance coefficients increase. This supports our earlier conclusion that dominance increases the likelihood that divergent selection will be detected.

Higher inbreeding levels tend to reduce the *Q*_{ST} − *F*_{STQ} contrast in both the simulated and the expected analytic values (Figure 5B; plotted for the medium dominance level *d* = ), although it is difficult to compare these results to Figure 3B because only two selfing rates are modeled. There is no significant difference between the *Q*_{ST} − *F*_{STQ} contrasts for high, medium, and low mutation rates (data not shown).

##### Q_{ST} − F_{STN}:

Table 3 also shows the value of *Q*_{ST} − *F*_{STN} for each design, averaged over the 100 replicates. These contrasts are large due to the strong selection coefficients used in the simulations and hence the high *Q*_{ST} values compared to neutral marker *F*_{ST}. In a meta-analysis of 55 empirical studies comparing *Q*_{ST} and *F*_{STN}, the average difference between *Q*_{ST} and *F*_{STN} was 0.12 (Leinonen *et al.* 2008), suggesting that on average selection is weak in reality. However, 6 of 62 studies (of which 4 of 55 were included in the meta-analysis) reported *Q*_{ST} > 0.8, and of these 3 were also associated with relatively low *F*_{STN} values (*F*_{STN} < 0.25); therefore such high levels of divergence are not impossible.

Despite the very large average values of *Q*_{ST} − *F*_{STN}, the large variance in values for *Q*_{ST} and *F*_{STN} causes the *Q*_{ST} − *F*_{STN} contrast for simulated values to be negative in some replicates. In particular, designs with a high selfing rate have a reasonable proportion (up to 4%) of negative *Q*_{ST} − *F*_{STN} values. Interestingly, in a separate set of simulations with a much higher migration rate (*m* = ), the proportion of negative *Q*_{ST} − *F*_{STN} values was much larger. In cases where high migration was paired with high selfing rates, even the average *Q*_{ST} − *F*_{STN} over 100 replicates was negative (data not shown). Although this appeared to be a consequence of a large proportion of replicates fixing alleles at all neutral loci (*F*_{STN} ≈ 1; see above), these results indicate that high migration and drift may in fact cause *Q*_{ST} < *F*_{STN} even when the quantitative trait is under divergent selection.

We now assess the effect a different selection regime has on the *Q*_{ST} − *F*_{ST} contrast for the quantitative and neutral loci in simulation sets B–E. As in simulation set A above, we use a combination of low and high inbreeding coefficients and low, medium, and high dominance as the basis for our six simulation designs.

#### Simulation set B—differentiation via divergent optimal selection:

Simulation set B assessed whether divergent optimal selection at one locus gives similar results to the simulations using divergent linear selection above (set A). Recalling our optimal selection fitness function (Equation 11), we set divergent local optima for populations *X* and *Y* (*G _{X}*

_{OPT}= −

*G*

_{Y}_{OPT}= ) with relatively strong selection strength (γ = 5). Again each of the six designs was replicated 100 times and

*F*

_{STQ},

*F*

_{STN}, and

*Q*

_{ST}were calculated from final allele frequencies using Equations 8 and 12. Average values for

*Q*

_{ST}were ∼0.06 larger than corresponding simulation values under divergent linear selection (Table 3), suggesting that the strength of selection was slightly higher. All average

*Q*

_{ST}−

*F*

_{STQ}and

*Q*

_{ST}−

*F*

_{STN}values are positive (data not shown). In some cases a reasonable proportion of

*Q*

_{ST}−

*F*

_{STQ}replicates are negative but this fails to affect the average

*Q*

_{ST}−

*F*

_{STQ}values for each design. These results indicate that divergent optimal selection elevates

*Q*

_{ST}still further above

*F*

_{STQ}than does divergent linear selection, making it even more likely that selection will be detected. These simulations further support the conclusion that a dominant locus under divergent selection is likely to increase the value of

*Q*

_{ST}relative to

*F*

_{STQ}.

#### Simulation sets C and D—differentiation via balancing optimal selection:

Simulation sets C and D represent balancing selection acting on one and two loci, respectively. For both sets, the local optima were set at zero and were the same for populations *X* and *Y* (*G _{i}*

_{OPT}= 0), with strong selection toward this optimum (γ = 5).

##### One selected locus:

Table 4 shows the results of 100 replicates for each design for set C (one locus). Values are based on final allele frequencies from the simulations and calculated using Equations 8 (*F*_{STN}, *F*_{STQ}) and 12 (*Q*_{ST}). Under balancing selection, we expect *Q*_{ST} − *F*_{STN} < 0 because the differentiation of quantitative traits will be restrained relative to neutral markers. Table 4 shows that indeed in all cases average *Q*_{ST} − *F*_{STN} values are less than zero.

When allele frequencies at the quantitative trait locus are similar in populations *X* and *Y*, we expect *Q*_{ST} − *F*_{STQ} to be approximately equal to zero. Provided allele frequencies are not identical, a low frequency of the recessive allele in both populations gives *Q*_{ST} − *F*_{STQ} values slightly less than zero, while a higher frequency gives *Q*_{ST} − *F*_{STQ} values slightly greater than zero. This difference between *Q*_{ST} and *F*_{STQ} across different frequencies of the recessive allele in populations *X* and *Y* is demonstrated in Figure 7, for and (also see similar contour plots in Figure 2, A and B, in Goudet and Buchi 2006). The negative or close to zero values of *Q*_{ST} − *F*_{STQ} under balancing selection (Table 4) therefore suggest a trend for low frequencies of the recessive allele in both populations. Negative *Q*_{ST} − *F*_{STQ} values are more likely at medium and high dominance coefficients than for low dominance coefficients. This leads to the exciting conclusion that, as with divergent selection, balancing selection is more likely to be detected when alleles at the locus are not additive. *Q*_{ST} values are depressed relative to *F*_{ST} at the quantitative locus, and when compared to *F*_{ST} for neutral loci, the signal of balancing selection (*Q*_{ST} − *F*_{STN} < 0) becomes stronger.

##### Two selected loci:

We also simulated 100 replicates of the six designs where two loci contribute to the quantitative trait and eight loci are neutral (set D, data not shown). The *F*_{STN} values agree well with those from previous sets, as expected for these neutral loci. However, *Q*_{ST} and *F*_{STQ} values are remarkably high (*Q*_{ST} ≈ 0.4) compared to values for one selected locus (Table 4; *Q*_{ST} ≈ 0.1). This high level of differentiation with low migration is explained by a reasonable proportion of replicates fixing different alleles in populations *X* and *Y* (see also Goldstein and Holsinger 1992 and Latta 1998). For example, for two loci with two alleles *A* and *B* at frequencies *p* and *q*, the same mean (μ) for populations *X* and *Y* can be achieved when allele frequencies are similar in both populations (such as and , giving and ) or different alleles are fixed in both populations (such as while , giving and ) (where *p _{ij}* is the frequency of allele

*A*at locus

*j*in population

*i*).

The elevation of *Q*_{ST} levels has a large impact on the average *Q*_{ST} − *F*_{STN} values across these designs (data not shown). Under balancing selection we expect *Q*_{ST} − *F*_{STN} < 0, but in designs with low levels of inbreeding the average contrast is much greater than zero. This indicates that *Q*_{ST} calculated using allele frequencies at the quantitative loci may be very large despite the mean values of the trait of interest being similar between populations (Goldstein and Holsinger 1992; Latta 1998). Clearly because *Q*_{ST} is usually measured on trait values rather than calculated from allele frequencies across loci (as the quantitative loci of interest are unlikely to be known), this is a rather academic conclusion. This result does suggest, however, that knowledge of allele frequencies at loci underlying a quantitative trait will not necessarily help determine whether the trait has been under divergent or balancing selection.

#### Simulation set E—neutral differentiation:

Finally, we replicated the results of Goudet and Buchi (2006) by simulating purely neutral differentiation for both the quantitative and the neutral loci, for two loci coding the quantitative trait and eight neutral loci (set E, data not shown). We set the optimum for both populations *X* and *Y* equal to zero, but additionally reduced the strength of selection (γ = 5 × 10^{9}) so that the fitness of every genotype was effectively equal. Because the quantitative trait is neutral, we expect for each design that *Q*_{ST} ≈ *F*_{STQ} ≈ *F*_{STN}. Our simulations suggest that average *Q*_{ST} − *F*_{STQ} values are slightly less than zero, although the proportions of *Q*_{ST} − *F*_{STQ} greater than and less than zero within replicates of each design are approximately equal. This result agrees with that of Lopez-Fanjul *et al.* (2003, 2007), Goudet and Buchi (2006), and Goudet and Martin (2007)—when integrating over the surface of all feasible allele frequencies in populations *X* and *Y*, the average *Q*_{ST} − *F*_{STQ} value tends to be negative when *d* ≠ 0, despite the positive and negative areas of parameter space being approximately equal (Figure 7). Thus, when a trait is evolving neutrally, dominance may deflate the value of *Q*_{ST} relative to *F*_{ST} and therefore may lead us to conclude that balancing selection is restricting the differentiation of the quantitative trait.

Finally of note, the values of *F*_{STN} are reassuringly similar across sets A–E, reflecting pure drift in all cases. Note that the neutral loci are unlinked to the quantitative loci and their differentiation is dependent only upon the effective size of the population. In particular, *F*_{STN} values are consistently high when rates of selfing are high, reflecting local fixation of alleles due to the small effective population size of these designs.

## DISCUSSION

There has been growing interest in the evolution of quantitative traits in natural populations, particularly in those traits that have important fitness consequences. Empirical evidence from natural populations suggests that quantitative trait divergence frequently exceeds the divergence seen at neutral markers (Merila and Crnokrak 2001; Leinonen *et al.* 2008). In addition to the large variances expected for *Q*_{ST} and *F*_{ST}, and the difficulty in testing any difference between them (O'Hara and Merila 2005; Whitlock 2008), recent modeling has further suggested that the observation of *Q*_{ST} > *F*_{ST} may be caused by factors intrinsic to the quantitative loci themselves, such as dominance and epistatic interactions between loci (Latta 1998; Whitlock 1999; Le Corre and Kremer 2003; Lopez-Fanjul *et al.* 2003; Goudet and Buchi 2006).

In this article, we constructed both an analytic and a simulation model to describe the differentiation of dominant quantitative traits and neutral markers between two populations linked by migration. We included dominance, selfing, and a divergent linear selection function into recursive equations to find equilibrium allele frequencies for the two populations and used these allele frequencies to calculate *Q*_{ST} and *F*_{ST} at the quantitative locus. In addition, we derived expressions and recursive equations to calculate *F*_{ST} at equilibrium for neutral loci. Finally, we constructed an individual-based simulation model to investigate the effects of divergent and balancing selection and drift on the *Q*_{ST} − *F*_{ST} contrast.

These models have given important insight into the behavior of *Q*_{ST} and *F*_{ST} in the presence of dominance and inbreeding. By using our recursion equations to find expected equilibrium values for *Q*_{ST}, we have demonstrated that when a quantitative trait is under divergent selection, the expectation for *Q*_{ST} − *F*_{STQ} (*F*_{ST} calculated at the quantitative locus using equilibrium allele frequencies) is generally greater than zero when the quantitative locus is dominant. *Q*_{ST} is likely to be elevated relative to the divergence calculated from allele frequencies at the quantitative trait (*i.e.*, *F*_{STQ}), and when comparing *Q*_{ST} to *F*_{ST} calculated from neutral loci (*F*_{STN}), this elevation in *Q*_{ST} means that it is more likely that divergent selection will be detected. Results from simulations also indicate that for balancing selection, where we expect *Q*_{ST} < *F*_{ST}, the presence of dominance may deflate *Q*_{ST} relative to *F*_{STQ} and also lead to a larger negative value for *Q*_{ST} − *F*_{STN}. Despite the impact that dominance, and to a certain extent inbreeding, has on enhancing the *Q*_{ST} − *F*_{STQ} contrast, this should be seen in context: selection has a very much stronger impact on *Q*_{ST} values than dominance or inbreeding.

#### Effect of dominance:

The reason that dominance generally enhances the difference between *Q*_{ST} and *F*_{ST} for the selected locus can be seen clearly when we plot the increase of expected *Q*_{ST} and *F*_{STQ} values against increasing dominance coefficients for a “typical” set of parameters (Figure 8). Although *Q*_{ST} and *F*_{ST} have the same value when there is no dominance (*d* = 0), when the populations are under divergent selection, *Q*_{ST} increases more rapidly than *F*_{ST} with increasing dominance coefficients. In contrast to divergent selection, simulation results for balancing selection suggest that the growth in *F*_{ST} values across dominance coefficients may be greater than that for *Q*_{ST} (Table 4), explaining why the presence of dominance enhances the difference between *Q*_{ST} and *F*_{ST} for both divergent and balancing selection. We conclude that dominance is very unlikely to mask the effect of selection on the signatures of divergent and balancing selection (*Q*_{ST} > *F*_{ST} and *Q*_{ST} < *F*_{ST}, respectively).

There are in addition a number of interesting findings from the sets of simulations. For our simulation results for divergent selection using a linear fitness function (Table 3), the observed difference between *Q*_{ST} and *F*_{STQ} for *d* > 0 was frequently negative due to the very large variance in both *Q*_{ST} and *F*_{STQ} values, despite the relatively large population sizes modeled (*N _{X}* =

*N*= 1000). In addition, even with strong selection acting on the quantitative trait, the value of

_{Y}*Q*

_{ST}was less than

*F*

_{STN}for neutral markers in some cases. Nevertheless, at high levels of dominance it was more probable that

*Q*

_{ST}−

*F*

_{STQ}and

*Q*

_{ST}−

*F*

_{STN}values were positive, suggesting that divergent selection is more likely to be detected in dominant than in additive traits. Interestingly, in simulations with a different divergent selection regime (where genotypic fitnesses are determined by an optimal fitness function), dominance had an even stronger effect on enhancing the difference between

*Q*

_{ST}and

*F*

_{STQ}. Under balancing selection,

*Q*

_{ST}seemed to be less than

*F*

_{STN}in the presence of dominance, suggesting that, as with divergent selection, balancing selection is more likely to be detected in dominant than in additive traits.

The results for one locus under balancing selection are consistent with expectation when comparing quantitative and neutral loci (*Q*_{ST} − *F*_{STN} < 0). However, when two loci were selected, the observed *Q*_{ST} values at low levels of migration and selfing were extremely high compared to the *F*_{STN} values. This was a consequence of calculating *Q*_{ST} using allele frequencies at these loci rather than the overall trait values; although the mean trait values were similar between populations, allele frequencies could be highly differentiated. Therefore, when interpreting sequence differentiation at loci thought to contribute to quantitative traits, it is important to bear in mind that large sequence differences between two populations may not be reflected in any significant trait differentiation if they are offset by differences in other coding loci throughout the genome.

Finally, as explored by Lopez-Fanjul *et al.* (2003, 2007), Goudet and Buchi (2006), and Goudet and Martin (2007), neutral traits show generally very little difference in *Q*_{ST} and *F*_{STQ} values, with similar proportions of *Q*_{ST} − *F*_{STQ} greater than and less than zero across designs.

#### Extensions to more general cases:

##### Multiple alleles:

In all of our simulations described above, we assumed that the quantitative trait was coded by one or more diallelic loci. To assess whether our conclusions are restricted to the case of two alleles, we ran an additional set of simulations. Inbreeding coefficients, migration rates, and mutation rates were fixed at 1/*N* with population size *N _{X}* =

*N*=

_{Y}*N*= 1000. To model divergent selection, selection coefficients were chosen from a uniform distribution between 0 and 0.1 (population

*X*) and between −0.1 and 0 (population

*Y*). The simulations were run for 2000 generations with 200 replicates. One set of simulations was run for a single-locus, two-allele model, where the dominance coefficient

*d*was chosen at random from (0, 1), with genotypic values defined as in Table 2. The second set of simulations was run for a single-locus, four-allele model (alleles

*A*,

_{i}*i*= 1–4). Genotypic values of homozygotes were defined aswhere α

_{i}is the additive effect of allele

*A*and is drawn from a normal distribution of mean 0 and standard deviation of . Genotypic values of heterozygotes were defined aswhere

_{i}*d*is the dominance term. Each of the six unique dominance terms was drawn from a normal distribution with mean 0 and standard deviation defined bysuch that ∼95% of heterozygotes fall within the range between the two homozygote genotypic values.

_{ij}*F*

_{ST}at the quantitative locus (

*F*

_{STQ}) was calculated using (8). We calculated

*Q*

_{ST}using Equation 4, where

*V*

_{A}is defined as twice the covariance between parent and offspring genotypic values (averaged over populations), and

*V*

_{B}is the variance in mean genotypic values between populations.

The results from the two sets of simulations strongly support our earlier conclusion that under divergent selection, dominance causes a positive difference between *Q*_{ST} and *F*_{STQ}. The mean difference between *Q*_{ST} and *F*_{STQ} was significantly greater than zero whether two or four alleles contribute to the quantitative trait (*P* < 0.001), and in both cases only a small proportion of replicates had *Q*_{ST} − *F*_{STQ} values less than zero. Dominance is therefore likely to enhance the difference between *Q*_{ST} and *F*_{STQ}, whether the quantitative trait locus has two or more alleles, when the quantitative trait is under selection.

##### Multiple populations:

An additional restriction of our simulations was that we investigated only two populations linked by migration. The variance in *Q*_{ST} and *F*_{ST} is certainly affected by the number of populations studied (see Whitlock 2008 and references therein) and will affect the likelihood that we can detect a difference between them. We therefore performed simulations for two and four populations under divergent selection, to assess the change in variance in *Q*_{ST}, *F*_{STQ}, and *Q*_{ST} − *F*_{STQ}. *Q*_{ST} and *F*_{ST} are calculated by extending Equations 8 and 12 to the case of four populations.

We performed two sets of simulations, the first with low migration and inbreeding, where many parameters were fixed (, *d* = ). The second set of simulations had a larger proportion of parameters that are allowed to vary, with and *f*, *d*, and *m* chosen at random from their biologically feasible ranges. For both sets, half of the selection coefficients were chosen at random from (−1, 0), and half chosen at random from (0, 1). For two and four populations, each of the simulation sets was run for 2000 generations with 500 individuals in each population, and 1000 replicates were performed.

For both sets of simulations, the variances in *Q*_{ST}, *F*_{STQ}, and *Q*_{ST} − *F*_{STQ} decreased substantially for four populations compared to two populations. For the first set of simulations, the variances for *Q*_{ST}, *F*_{STQ}, and *Q*_{ST} − *F*_{STQ} were 0.0097, 0.0099, and 0.0017, respectively, for two populations and 0.0022, 0.0021, and 0.0007 for four populations. Results from the second set of simulations showed a similar trend. These findings suggest that increasing the number of populations will not only decrease the variance in *Q*_{ST} and *F*_{STQ}, but also decrease the variance in the difference between the two. When comparing *Q*_{ST} to *F*_{ST} from neutral marker loci, the increased precision in *Q*_{ST} will make it more likely that we will detect divergent selection on the quantitative trait. The variances in *Q*_{ST} and *F*_{ST}, and hence the likelihood that we will observe a difference between them, are influenced by many other factors, for example, sample size and how *Q*_{ST} and *F*_{ST} are estimated (Whitlock 2008). These factors need to be taken into account when we determine the significance of any difference (or indeed, lack of difference) between *Q*_{ST} and *F*_{ST} for empirical data.

We have demonstrated that under divergent selection, dominance is likely to increase the value of *Q*_{ST}, relative to the value expected for a purely additive trait. Our simulations also suggest that balancing selection will decrease the value of *Q*_{ST}. For neutral loci, the value of *Q*_{ST} under dominance may be increased or decreased relative to *Q*_{ST} expected for an additive trait. Under divergent, balancing, and no selection, we expect *Q*_{ST} greater than, less than, and approximately equal to *F*_{ST}, respectively. Thus, the presence of dominance will enhance the difference between *Q*_{ST} and *F*_{ST} when the quantitative locus is selected and will make it more likely that we will be able to detect these “signatures of selection” in natural populations. For neutral loci, dominance does lead to a difference between *Q*_{ST} and *F*_{ST}. The difference is, however, small, making a false detection of selection unlikely.

A perhaps unsurprising result from both analytic and simulation models was the very large impact of migration rates on *Q*_{ST} and *F*_{ST} values. Even with strong directional selection, high levels of migration can severely restrict the differentiation between populations, such that the difference between *Q*_{ST} and *F*_{ST} measured at neutral markers becomes so low as to be indistinguishable from a neutral model (data not shown). Thus, while we cannot have confidence in the *Q*_{ST} − *F*_{ST} contrast giving us any reasonable indication of selection when selection is weak and population sizes are small, we similarly are unlikely to detect consistent signatures of selection when migration is high. One major conclusion from these simulations has been previously well stated by Goudet and Martin (2007, p. 1373): compared to dominance (or indeed other factors), “the large variance of *Q*_{ST} is certainly more worrisome for the prospect of identifying traits under selection.”

## APPENDIX A

#### Population differentiation at neutral markers (*F*_{ST}):

Recall that *N _{i}* is the number of individuals in population

*i*(

*i*=

*X*,

*Y*) and that we let the probability of selfing in both populations be

*f*while the probability of parents outcrossing is (1 −

*f*). Selfing and outcrossing are assumed to be independent, and so the numbers of selfed and outbred offspring per parent are binomially distributed with means

*f*and (1 −

*f*), respectively.

We calculate a number of coefficients describing the similarity between and within individuals in a subdivided population, which we use to calculate *F*_{ST}. The inbreeding coefficient *F* describes the probability that two alleles at a given locus within an individual are identical by descent (IBD), while θ, the coancestry coefficient, is the probability that two alleles chosen at random from two individuals within a population are IBD. Both *F* and θ can be calculated separately for the two populations, and overall values found by multiplying each population parameter by its proportional population size. Finally, α is the probability that two alleles chosen at random from different populations are IBD.

#### Recursions for probabilities of identity by descent:

Following the approach of Wang (1997), we derive the full set of recursion equations for *F*, θ, and α. Individuals within a population outcross or self to give offspring in the next generation, and then a number of these offspring migrate so that population *i* has a proportion *m _{i}* of migrants and (1 −

*m*) of nonmigrants. We first derive the expressions for the probability of IBD following reproduction and then following migration. Definitions of terms used in the following derivations may be found in Table 1.

_{i}*F*, θ, and α after reproduction:

##### F:

Offspring arise from selfing and outcrossing, with expected proportions *f* and (1 − *f*). The inbreeding coefficients of offspring from selfing and outbreeding are and θ_{i}, respectively. The average *F* in the next generation is thereforefor population *i*.

*θ*:

When calculating θ between two individuals *I*_{1} and *I*_{2}, we need to consider a number of situations:

Both

*I*_{1}and*I*_{2}are the result of selfing, with probability*f*^{2}. Assuming all parents have an equal chance of producing selfed offspring, the probability that*I*_{1}and*I*_{2}have the same or different parents is 1/*N*and (*N*− 1)/*N*, respectively. The probability of IBD between*I*_{1}and*I*_{2}with the same parent is and with different parents is θ.*I*_{1}is the result of selfing and*I*_{2}is not (or vice versa), with probability 2*f*(1 −*f*).*I*_{1}and*I*_{2}share a parent with probability 2/*N*, and the probability of IBD between a selfed and an outbred sib is .*I*_{1}and*I*_{2}share no parents with probability , and the probability of IBD between unrelated individuals is θ.Both

*I*_{1}and*I*_{2}are the result of outcrossing, with probability (1 −*f*)^{2}. The probability that*I*_{1}and*I*_{2}have the same parent pair is 2/*N*(*N*− 1), and the probability of IBD between these full sibs is .*I*_{1}and*I*_{2}share one parent with probability 1/*N*and do not share the next with probability , and can do so four ways, giving an overall probability of . The probability of IBD between half sibs is . Finally, all parents between and within*I*_{1}and*I*_{2}are different with probability , and the probability of IBD between unrelated individuals is θ.

Multiplying probabilities, summing each of the subcases, and then summing over the three mating types, the overall probability of IBD between two individuals in population *i* after reproduction is

##### α:

Reproduction does not change the probability of IBD between populations, so

*F*, θ, and α after migration:

##### F:

After migration, the probability that alleles within an individual are IBD for population *X* is a weighted average of and :A similar expression for population *Y* is obtained by swapping the *X* and *Y* subscripts.

##### θ:

After migration, the probability that two alleles from different individuals within *X* (or *Y*) are IBD is a weighted average of and . For two individuals taken at random from population *X*, the probability that they are both nonmigrants is , that they are both migrants is , and that one is migrant and one nonmigrant is . Then the average probability of IBD isA similar expression for population *Y* is obtained by swapping the *X* and *Y* subscripts.

##### α:

After migration, the probability that two alleles, one from *X* and one from *Y*, are IBD is again a weighted average of and : .

By substituting values of and into the equations for *F _{X}*

_{(t)},

*F*

_{Y}_{(t)}, θ

_{X(t)}, θ

_{Y(t)}, and α

_{(t)}above, we derive the full set of recursive equations for the probabilities of identity by descent. These recursion equations may be written in matrix notation aswhere(A1)and

*M*=

_{i}*m*is the number of migrant individuals in population

_{i}N_{i}*i*(

*i*=

*X*,

*Y*). Interestingly, these equations differ from those of Wang (1997) and Vitalis and Couvet (2001), who assume exchange of genes rather than individuals during migration, and hence migration has a different interpretation in the

*F*, θ, and α terms.

If migration and population sizes are the same in the two populations (*m _{X}* =

*m*=

_{Y}*m*and

*N*=

_{X}*N*=

_{Y}*N*), then

*F*=

_{X}*F*=

_{Y}*F*, θ

_{X}= θ

_{Y}= θ, and the recursions reduce to three equations, where(A2)and

*M*=

*mN*.

While the values of *F*, θ, and α increase monotonically over time, their instantaneous rates of increase [, , and ] converge to the same value if there is some migration between populations (Wang 1997). Their asymptote is equal to half the inverse of the asymptotic effective population size; that is,(A3)The expression for *N*_{e} may be found from the transition matrix **T**. Following Wang (1997),(A4)where λ is the leading eigenvalue of **T**. Although finding λ is straightforward when the parameters of the model (*f*, *m _{X}*,

*m*,

_{X}*N*, and

_{X}*N*) are known, there are five eigenvalues for the 5 × 5 matrix

_{Y}**T**and it becomes difficult to write a general expression for the leading eigenvalue. For example, Mathematica 4.0 (Wolfram Research 1999) finds the five eigenvalues in terms of root objects that may be evaluated only numerically. For given values of

*N*,

_{X}*N*,

_{X}*m*,

_{X}*m*, and

_{Y}*f*, we may find a numeric value for

*N*

_{e}either by using the leading eigenvalue of

**T**or by iterating the recursion

**S**

_{(t)}=

**TS**

_{(t−1)}+

**C**until the changes in Δ

*F*, Δθ, and Δα become small.

When we make the simplification that migration rates and population sizes are equal for the two populations, and **T** is reduced to a 3 × 3 matrix [see (A2) above], we can solve for the leading eigenvalue λ of **T**. By substituting λ into (A4), we can derive an explicit expression for *N*_{e},whereandand | | is the modulus of the expression.

##### F_{ST} at equilibrium:

Using our recursive equations, we may also determine values for Wright's *F*-statistics, including *F*_{ST}. The *F*-statistics are defined as(Wright 1969), where *F*, θ, and α are the probabilities of IBD described above. For given parameter values of *f*, *m _{i}*, and

*N*, therefore, we can use recurrent Equation A1 to obtain equilibrium values for

_{i}*F*

_{ST},

*F*

_{IS}, and

*F*

_{IT}. In the simple case of

*m*=

_{X}*m*=

_{Y}*m*and

*N*=

_{X}*N*=

_{Y}*N*, we obtainandSolving these equations for

*F*

_{ST}using (A3) yields(A5)where

*N*

_{e}is defined above.

## APPENDIX B: EXPRESSION FOR *Q*_{ST} – *F*_{ST} FOR A SINGLE QUANTITATIVE LOCUS WITH TWO ALLELES

Note that *p _{i}* is the frequency of allele

*A*in population

*i*(

*i*=

*X*,

*Y*).where and .

## Acknowledgments

The authors thank the anonymous reviewers for their extremely helpful suggestions to improve the manuscript.

## Footnotes

Communicating editor: J. Wakeley

- Received October 24, 2008.
- Accepted November 1, 2008.

- Copyright © 2009 by the Genetics Society of America