The Effect of Neutral Nonadditive Gene Action on the Quantitative Index of Population Divergence
Carlos López-Fanjul, Almudena Fernández, Miguel A. Toro


For neutral additive genes, the quantitative index of population divergence (QST) is equivalent to Wright’s fixation index (FST). Thus, divergent or convergent selection is usually invoked, respectively, as a cause of the observed increase (QST > FST) or decrease (QST < FST) of QST from its neutral expectation (QST = FST). However, neutral nonadditive gene action can mimic the additive expectations under selection. We have studied theoretically the effect of consecutive population bottlenecks on the difference FST - QST for two neutral biallelic epistatic loci, covering all types of marginal gene action. With simple dominance, QST < FST for only low to moderate frequencies of the recessive alleles; otherwise, QST > FST. Additional epistasis extends the condition QST < FST to a broader range of frequencies. Irrespective of the type of nonadditive action, QST < FST generally implies an increase of both the within-line additive variance after bottlenecks over its ancestral value (VA) and the between-line variance over its additive expectation (2FSTVA). Thus, both the redistribution of the genetic variance after bottlenecks and the FST - QST value are governed largely by the marginal properties of single loci. The results indicate that the use of the FST - QST criterion to investigate the relative importance of drift and selection in population differentiation should be restricted to pure additive traits.

ASSESSING the relative contributions of natural selection and genetic drift to population differentiation for quantitative traits is an important issue, in both evolutionary and conservation genetics (Wright 1978; Endler 1986). In the absence of selection, inbreeding affects all genes to the same average degree, and the effect of the breeding structure on population divergence can be described by Wright’s among-population fixation index FST. In parallel, a dimensionless measure of the quantitative genetic variance among populations (termed QST by Spitze 1993) can be defined as QST = Vb/(Vb + 2Vw), where Vb and Vw are, respectively, the between- and the additive within-population components of the genetic variance for the trait considered. For neutral genes, with additive action between and within loci, it is expected that Vb = 2FSTVA and Vw = (1 - FST)VA, where VA is the ancestral additive genetic variance (Wright 1951). In this situation, QST is the neutral quantitative analog of FST. This result holds quite generally, regardless of the model of population structure (Whitlock 1999). The computation of the expected divergence of population means due to drift requires the estimation of generally unknown parameters, such as the rate of mutation, the time since divergence, and the effective population size (Lande 1977). However, it can be used as the null expectation that can be compared to the corresponding QST estimate for a quantitative trait, assumed to be additive. Thus, divergent or convergent selection may be invoked, respectively, as a cause of the observed increase (QST > FST) or decrease (QST < FST) of QST from its neutral expectation (QST = FST).

Experimentally, this approach has been used in many studies (see Merilä and Crnokrak 2001 and McKay and Latta 2002 for reviews). In most instances, the errors in the estimation of FST and QST were large, resulting in nonsignificant pairwise comparisons of these estimates. Nevertheless, a meta-analysis carried out by Merilä and Crnokrak (2001) indicated that QST was generally larger than FST, and this result was interpreted in the sense that a considerable part of the observed population divergence for quantitative traits should be attributed to differential selection pressures imposed by local environmental conditions.

Notwithstanding, the correspondence between QST and FST depends crucially on the assumption of pure additive gene action. This may not be an important restriction to the study of morphological traits, typically showing substantial additive genetic variation and little or no inbreeding depression, but will markedly affect that of life-history traits, usually exhibiting larger levels of nonadditive variance and, correspondingly, higher inbreeding depression (Crnokrak and Roff 1995; DeRose and Roff 1999). In the absence of selection, it has been shown theoretically that inbreeding can change the magnitude of the contribution of dominant and/or epistatic loci to the values of Vb and Vw, relative to their additive expectations (Robertson 1952; Goodnight 1988; Willis and Orr 1993; Cheverud and Routman 1996; López-Fanjul et al. 1999, 2000, 2002). In parallel, increases of the additive variance with inbreeding have been reported for viability in Drosophila melanogaster (López-Fanjul and Villaverde 1989; Garcíaet al. 1994) and Tribolium castaneum (Fernándezet al. 1995) and for morphological and behavioral traits in the housefly (reviewed by Meffert 2000). Thus, nonadditive gene action can potentially modify the expected additive relationship between FST and QST.

In this article, we have investigated theoretically the effect of successive population bottlenecks on the difference FST - QST for two-locus neutral epistatic systems, covering all possible types of marginal gene action at the single-locus level (excluding overdominance). Our approach follows that of Robertson (1952), where the expected values of the derived within-line additive variance and the between-line variance, after consecutive bottlenecks of size N, are obtained from the expressions giving the corresponding ancestral values in an infinite population at equilibrium and the moments of the allele frequency distribution in populations of size N with binomial sampling. Explicit equations in terms of the genetic effects and allele frequencies derived by López-Fanjul et al. (2002) have been used, allowing the specification of the necessary conditions to observe a departure of QST from the pertinent FST value.


We consider the model developed by López-Fanjul et al. (2002), where the variation is due to segregation at two neutral independent loci (i = 1, 2) at Hardy-Weinberg equilibrium. At each locus there are two alleles, with frequencies pi and qi (=1 - pi). Both loci have equal homozygous effect (s/2), showing any degree of dominance in the absence of epistasis (hi = 0, ½, or 1 for complete recessive, additive, or complete dominance action, respectively). This basic gene action can be viewed as that shown by single loci segregating against a fixed genetic background. Epistasis has been imposed on that basic system, and it is represented by a factor k affecting the genotypic value of the double homozygote for the negative allele at each locus (k < 2, k > 2, or k = 2 for diminishing, reinforcing, or no epistasis, respectively). A full specification of the genotypic values is given in Table 1. At the ith locus, the marginal average effect of gene substitution (αi), the marginal genotypic value of the heterozygote (δi, expressed as deviation from the midhomozygote value), and the marginal degree of dominance (γi) are given by αi=s[hi+(12hi)qi+(k2)qiqj2], (1) δi=s[(12hi)+(k2)qj2]2, (2) γi=hi[1+(k2)qj2]. (3) Thus, epistasis (k = 2) modifies the basic properties of single loci, as αi, δi, and γi become dependent on the allelic frequencies at the other locus (qj); i.e., they are contingent on the genetic background. For a given k value, the basic (hi) and the marginal (γi) degrees of dominance become closer to each other as qj decreases. On the other hand, γi approaches zero (complete recessivity) as k and qj increase.

View this table:

Genotypic values for the neutral two-locus epistatic system

In an infinitely large panmictic population, expressions for the mean (ancestral mean M) and the additive component of the genetic variance (ancestral additive variance VA) can be obtained from Table 1, as M=12sΣhiqisΣ(12hi)qi2+(k2)q12q22, (4) VA=Σαi2Hi, (5) where Hi is the ancestral heterozygosity at the ith locus (Hi = 2piqi). These expressions are polynomial functions of pim(i=1,2;m=14) and their expected values at equilibrium, after t consecutive bottlenecks of N randomly sampled parents each (derived mean Mt and additive variance VAt ), can readily be deduced by substituting pim in Equations 4 and 5 by the corresponding exact mth moment of the allelic frequency distribution with binomial sampling, given by Crow and Kimura (1970, p. 335). In parallel, the between-line variance V(Mt) after t consecutive bottlenecks can be derived by taking variances in Equation 4, the resulting expression being also a function of the first four moments of the allelic frequency distribution at each locus. As these moments can also be written in terms of the inbreeding coefficient after t generations (Ft), expressions for Mt , VAt , and V(Mt) also apply when bottleneck sizes are not constant from generation to generation. Those expressions (given by López-Fanjulet al. 2002) are analytically unmanageable, but numerical solutions can be calculated for any combination of allele frequencies, as well as the corresponding value of the quantitative index of population divergence Qt=V(Mt)(V(Mt)+2VAt) (see next section). In the following, the notations Ft, Qt, V(Mt), and VAt are kept for the two-locus system and the subdivided population studied, but FST, QST, Vb, and Vw are used, respectively, with reference to the whole set of loci affecting a metric trait and the relevant population structure.

It can be shown (López-Fanjulet al. 2002) that the change in mean after t bottlenecks is always negative for basic recessive gene action (complete or incomplete, 0 ≤ hi < ½) and reinforcing epistasis (k > 2) or no epistasis (k = 2). Nevertheless, diminishing epistasis (k < 2) and/or basic dominance (incomplete or complete, ½ < hi ≤ 1) result in an unrealistic enhancement of the mean with inbreeding and, therefore, they are not considered further.

For pure additive action it is expected that Qt = Ft, as VAt=(1Ft)VA and V(Mt) = 2FtVA (Wright 1951). For additive-by-additive epistasis, Qt < Ft for Ft < 1 (Whitlock 1999) as VAt=(1Ft)VA+4Ft(1Ft)VAA and V(Mt)=2FtVA+4Ft2VAA , where VAA is the additiveby-additive variance component (Goodnight 1988). With dominance (with or without epistasis) equations giving VAt and V(Mt) cannot be written in terms of summary statistics, as in the previous cases, but as complex functions of the allele frequencies and effects at each locus and the pertinent epistatic factor (Robertson 1952; Willis and Orr 1993; López-Fanjul et al. 1999, 2000, 2002). However, for nonepistatic complete recessive action (hi = 0, k = 2), some further insight on the redistribution of the genetic variance induced by bottlenecks and, consequently, on the relationship between Qt and Ft, can be achieved as follows.

In general, from Equation 5, the expected additive variance after bottlenecks can be given as VAt=Σcov(αi2,Hi)+ΣE(Hi)E(αi2)=Σcov(αi2,Hi)+(1Ft)ΣHi(V(αi)+αit2) (6) (López-Fanjulet al. 2002), where E(Hi) = (1 - Ft)Hi, and αit and Vi) are, respectively, the derived marginal average effect of gene substitution at the ith locus after t bottlenecks and its variance, which can be deduced by taking expectations or variances, respectively, in Equation 1. Equation 6 shows that any excess of VAt over its ancestral value can be assigned to the spatial and temporal changes in αi (represented by Vi) and αit , respectively), which are both induced by drift, as well as to the covariance term, which depends on the ancestral properties of dominant loci [cov(αi2,Hi)=0 for pure additive or additive-by-additive gene action]. Of course, dominance may be basic (0 ≤ hi < ½) or marginal (hi = ½, k > 2). For nonepistatic complete recessives (hi = 0, k = 2), αit=αi , Vi) = Fts2Hi/2, and cov(αi2,Hi)>0 . Thus, VAt=Σcov(αi2,Hi)+(1Ft)(VA+2FtVD), (7) where VD is the dominance component of the ancestral genetic variance, VD=δi2Hi2=s2Hi24 (from Equation 2). Equation 7 shows that VAt always exceeds its additive expectation, i.e., VAt>(1Ft)VA . Furthermore, as cov(αi2,Hi)>0 , the condition VAt>VA can be given as VA < 2(1 - Ft)VD, implying qi < (1 - Ft)/(2 - Ft), i.e., qi < ½. These results also apply for incomplete recessives (0 < hi < ½).

In parallel, the between-line variance can be written as V(Mt)=μ4FtVAFt2VD, where μ4=s2[E(qi4)qi4]>0 . Thus, V(Mt) equals only its additive expectation (2FtVA) for μ4=3FtVA+Ft2VD . Moreover, Qt will be larger than Ft if μ4<3FtVA+5Ft2VD and smaller otherwise, these conditions being slightly more restrictive than those for V(Mt) ≠ 2FtVA, unless Ft is large [e.g., V(Mt) > 2FtVA for μ4>3FtVA+Ft2VD ].

Summarizing, for neutral loci and nonadditive gene action (dominant and/or epistatic), Qt will generally depart from Ft, except in the particular case of V(Mt)=2FtVAt(1Ft) .


Three representative cases were studied, with additive (hi = ½) or recessive (hi = 0) basic gene action at both loci (s = 0.1) and strong reinforcing epistasis (k = 6) or with recessive nonepistatic action (hi = 0, k = 2). For each case, surfaces were represented (Figure 1), giving the values of the following contrasts after one bottleneck (N = 2, F1 = 0.25) for all possible combinations of allele frequencies at both loci: (1) the ratio of derived to ancestral additive components of variance VA1VA , (2) the ratio of the between-line variance to its expected value for additive gene action V(M1)/2F1VA, and (3) the difference F1 - Q1 between the inbreeding coefficient and the quantitative index of population divergence. With different basic gene action at each locus and epistasis, intermediate results were obtained (not shown).

For complete recessive nonepistatic action, Q1 < F1 for only low to moderate frequencies of the recessive allele at both loci (or for the recessive allele fixed in one locus and segregating at low frequency in the other); otherwise, Q1 > F1. The absolute value of the difference F1 - Q1 increased as the corresponding allele frequencies became more extreme. With additional epistasis, the condition Q1 < F1 holds for a much broader range of allele frequencies, and Q1 > F1 for only high frequencies of the recessive allele at both loci (or for the dominant allele fixed in one locus and the recessive one segregating at high frequency in the other). This situation is similar to that obtained with basic additive action and epistasis but, for low frequencies of both negative alleles, the excess of F1 over Q1 was, comparatively, much reduced. As shown by Equation 3, this can be ascribed to the marginal degrees of dominance (γi) becoming closer to the basic ones (hi = ½) as the frequencies of both negative alleles diminish. However, that excess was preserved when the negative allele is fixed in one locus and segregates at low frequency in the other as, in this case, the marginal degree of dominance of this second locus approaches zero (i.e., the locus becomes increasingly recessive). These results apply to populations subjected to a single bottleneck of any size, albeit the absolute value of F1 - Q1 decreased as the size of the bottleneck increased. With basic recessive action, increasing values of the epistatic factor k did not affect much the absolute value of the contrast (not shown) as, in this case, the basic and marginal degrees of dominance are the same. However, with epistasis and basic additive action, that absolute value was positively correlated with k, as the marginal degree of dominance tends to zero for increasing k values. Of course, for basic additive action without epistasis F1 = Q1.

Figure 1.

—Ratio of derived to ancestral additive components of variance Formula (log scale), ratio of the between-line variance to its additive expectation V(M1)/2F1VA (log scale), and difference F1 - Q1 between the inbreeding coefficient and the quantitative index of population divergence, after one bottleneck (N = 2, F1 = ½), for (1) nonepistatic complete recessive action at both loci (hi = 0, k = 2), (2) basic complete recessive action at both loci and reinforcing epistasis (hi = 0, k = 6), and (3) basic additive action at both loci and reinforcing epistasis (hi = ½, k = 6). Darker zones correspond to ratios smaller than one or to negative F1 - Q1 values.

As shown in Figure 1, Q1 < F1 holds approximately for the whole range of allele frequencies implying both V(M1) > 2F1VA and VA1>VA , irrespective of the type of basic gene action, and the reverse was also true. These conditions also hold after consecutive bottlenecks, but the absolute value of Ft - Qt initially increases with the number of bottlenecks until a maximum is reached for Ft close to 0.5 and then subsequently decreases to zero (not shown). These changes have also been described by Robertson (1952) for nonepistatic recessives at low frequency. Summarizing, with epistasis, Qt < Ft for all combinations of allele frequencies at both loci, excepting for high frequencies of the negative (recessive) alleles. Without epistasis, however, Qt < Ft for only low to moderate frequencies of those alleles.


We have shown that the Qt value generated by neutral dominant and/or epistatic loci, after t consecutive population bottlenecks, will always be larger or smaller than its additive expectation Ft, with the trivial exception determined by those particular combinations of allele frequencies fixing the boundary lines between the positive and negative regions of the Ft - Qt surface. Therefore, the use of the Ft - Qt difference as a criterion to investigate the relative importance of genetic drift and natural selection in population differentiation is restricted to pure additive traits, as nonadditive action at neutral loci can mimic the expectations for additive loci under divergent (Qt > Ft) or convergent selection (Qt < Ft). Moreover, for nonneutral nonadditive loci, selection will also affect (positively or negatively) the Ft - Qt value and this additional effect could even change the expected sign of that difference under neutrality.

For nonadditive gene action, previous theoretical work concerned with the divergence between Ft and Qt was restricted to the neutral additive-by-additive model, where Qt < Ft for Ft < 1 (Whitlock 1999). However, the effect of dominance (with or without epistasis) can qualitatively alter that result, as Qt may be smaller or larger than Ft, depending on the relevant allele frequencies. With simple dominance, Qt < Ft for only low to moderate frequencies of the recessive alleles, but additional epistasis extends that condition to higher values of the frequencies of the negative (recessive) alleles at one of the loci involved (but not at both). Irrespective of the type of gene action considered, we have also shown that Qt < Ft generally implies an increase of the within-line additive variance after bottlenecks (VAt>VA) , as well as an excess of the between-line variance over its additive expectation (V(Mt) > 2FtVA). Thus, both the redistribution of the genetic variance after bottlenecks and the value of Ft - Qt are governed largely by the marginal properties of single loci, epistasis acting only as a modulating factor (López-Fanjul et al. 1999, 2000, 2002).

So far, this discussion has been limited to investigating the consequences of population bottlenecks on the Ft - Qt difference generated by two-locus nonadditive neutral systems. An extension of these results to the whole set of loci determining the additive variance of a quantitative trait will, in principle, require a complete specification of their genotypic effects and allele frequencies, as the contribution of loci with the same type of gene action to the total FST - QST value can even be of different sign, depending on their respective allele frequencies. Generalizations into multilocus systems can be made only if individual loci show the same type of gene action and segregate with similar frequencies. Only in this situation do our theoretical results provide a framework within which some experimental data can be interpreted. The following discussion is restricted to D. melanogaster and T. castaneum, where detailed genetic information on relevant traits is available.

At one extreme of the spectrum, we have traits such as abdominal bristle number or wing size and shape characteristics of the wing. In natural populations of Drosophila, very little or no inbreeding depression has been detected for those characters and their between- and within-line additive variances after bottlenecks very closely approached the expectations under the pure additive model (López-Fanjulet al. 1989; Whitlock and Fowler 1999). In parallel, spontaneous mutations affecting bristle number, wing length, and wing width occur at a low rate and have a relatively large average homozygous effect (García-Doradoet al. 1999). Furthermore, those mutations with an effect smaller than one-half phenotypic standard deviation of the pertinent trait were predominantly additive and quasi-neutral (Santiagoet al. 1992; López and López-Fanjul 1993; Merchanteet al. 1995). Thus, a large fraction of the corresponding genetic variance in natural populations will be contributed by a small number of quasi-neutral additive loci segregating at intermediate frequencies (Robertson 1967; Gallego and López-Fanjul 1983). At the other end of the spectrum, we consider viability in natural populations of Drosophila and Tribolium. After bottlenecks, viability showed strong inbreeding depression and its within-line additive variance significantly increased above the ancestral value (López-Fanjul and Villaverde 1989; Garcíaet al. 1994; Fernández et al. 1995). Therefore, much of the genetic variance of the trait should be due to partially (or totally) recessive deleterious alleles segregating at low frequencies. In Drosophila, most newly arisen mutations affecting viability had, on the average, a relatively large homozygous disadvantage and their gene action was close to recessive (García-Doradoet al. 1999; García-Dorado and Caballero 2000; Chavarríaset al. 2001). Using mutational information, Wang et al. (1998) have been able to show that the inbreeding depression and the increase in the additive variance of viability following bottlenecks can be ascribed mainly to a rise in the frequency of lethals and partially recessive mutations of large deleterious effects.

View this table:

FST and QST estimates after experimental bottlenecks

Table 2 shows the between- and the additive within-line components of the genetic variance after one or three consecutive bottlenecks (N = 2) and the corresponding FST and QST values, for seven morphological traits in Drosophila (wing area, five angles whose vertices are defined by the intersections of the veins of the wing, and abdominal bristle number) and viability in Drosophila and Tribolium. As expected, FST and QST were very close for all morphological traits (average FST - QST =-0.018, range -0.03-0) and, for viability, FST was considerably larger than QST in all cases (average FST - QST = 0.14, range 0.08-0.30). It must be stressed that, in the experiments reviewed, all lines have been maintained under the same environmental conditions and have been subjected to a small number of bottlenecks (typically one). Thus, the effect of selection can be assumed to be small and the contrasting behavior of the FST - QST value for morphological traits and viability can be ascribed essentially to the changes induced by a known bout of random drift in sets of loci differing in their predominant type of gene action. In other words, the results in Table 2 can be taken as an experimental check of the validity of our theoretical predictions.

Incomplete information on the genetic properties of the traits studied makes the interpretation of the FST - QST difference more problematic. Estimates of FST (from molecular markers) and QST (for different quantitative traits) have been reported for sets of populations in a variety of plant and animal species (reviewed by Merilä and Crnokrak 2001 and McKay and Latta 2002). For most traits QST was larger than FST, this result being interpreted as a consequence of differential population adaptation to local conditions. The most common experimental design included populations from different geographic origins, each of them subdivided into families (typically full-sib families) and a number of individuals assayed per family, all of them maintained under the same environmental conditions. Here we are not concerned with the validity of inferring the population structure from the marker’s information, but we consider only those potential biases resulting from the treatment of quantitative data (detailed by Merilä and Crnokrak 2001). Following standard ANOVA procedures, the total variance for each trait was partitioned into sources arising from variation between populations, between families, within populations, and within families. In general, the genetic architecture of the traits assayed was unknown and, therefore, the estimates of the additive within-line variance obtained from full-sib analysis could be biased upward, as they may include fractions of the dominance and epistatic components of variance, as well as twice the environmental component of variance common to family members (including maternal effects). Biases due to nonadditive gene action will be more likely for life-history traits, which generally show higher levels of dominance variance than morphological traits show. Furthermore, estimates of the between-population variance could also be inflated to an unknown amount, due to common environmental effects (maternal and nonmaternal) that have not been experimentally removed. Even in a pure neutral additive situation, with biased estimates of the between- and within-line variances due to common environment only, Vb = 2FST(1 + a)VA, Vw = (1 - FST)(1 + b)VA, and QST = (1 + a)FST/[(1 + a)FST + (1 + b)(1 - FST)]. Therefore, the neutral expectation QST = FST will hold only when the magnitude of both biases is the same (a = b), and a > b or a < b will, respectively, mimic the expectation under divergent (QST > FST) or convergent selection (QST < FST). In parallel, as shown by our theoretical analysis, nonadditive action of neutral quantitative loci could also distort the expected additive relationship between FST and QST, even if the estimates of QST are free from environmental biases.

Summarizing, the sign of the difference FST - QST will be indicative of selection for only those traits whose genetic variance is mostly (or totally) generated by segregation at pure additive loci. Although these traits are commonly assumed to be quasi-neutral, the FST - QST criterion may be useful to establish the validity of this hypothesis.


We thank Michael Whitlock and Kevin Fowler for kindly allowing us to use their unpublished estimates of variance components for Drosophila wing traits. This study was supported by grant PB98-0814-C03-01 from the Ministerio de Educación y Cultura and RZ01-028-C2-1 from Instituto Nacional de Investigaciones Agrarias.


  • Communicating editor: Z-B. Zeng

  • Received July 17, 2002.
  • Accepted April 10, 2003.


View Abstract