Abstract
Without restricting the evolutionary forces that may be present, the theory of fixation indices, or F-statistics, in an arbitrarily subdivided population is developed systematically in terms of allelic and genotypic frequencies. The fixation indices for each homozygous genotype are expressed in terms of the fixation indices for the heterozygous genotypes. Therefore, together with the allelic frequencies, the latter suffice to describe population structure. Possible random fluctuations in the allelic frequencies (which may be caused, e.g., by finiteness of the subpopulations) are incorporated so that the fixation indices are parameters, rather than random variables, and these parameters are expressed in terms of ratios of evolutionary expectations of heterozygosities. The interpretation of some measures of population differentiation is also discussed. In particular, FST is an appropriate index of gene-frequency differentiation if and only if the genetic diversity is low.
WRIGHT's fixation indices, or F-statistics, are the parameters most widely used to describe population structure. Wright (1969, pp. 294–295; 1978, pp. 80–89; and refs. therein) defined the fixation indices as correlations between uniting gametes. His treatment is restricted to neutral diallelic loci; it is somewhat artificial (because numerical values are assigned to gametes) and not entirely clear.
Cockerham (1969, 1973; see also Weir and Cockerham 1984; Cockerham and Weir 1986) based his study of population structure on the analysis of the variance and covariances of indicator variables for allelic state, and he related his parameters to fixation indices and measures of identity by descent. Although Cockerham's analysis is more lucid and general than Wright's, it is disturbing that negative variance components may occur if mates are less closely related than the average within subpopulations (i.e., if Wright's FIS < 0).
Nei (1973; 1977; 1986; 1987, pp. 159–166; see also Nei and Chesser 1983) presented a third approach, formulated entirely in terms of the allelic and genotypic frequencies in the population. He expressed the fixation indices in terms of ratios of heterozygosities. His treatment is biologically the most direct, and it clearly requires no restrictions on the action of the evolutionary forces.
Allelic and genotypic frequencies may fluctuate because of finite subpopulation numbers or random variation in evolutionary forces. Even in this case, Wright's and Cockerham's measures of population structure are still parameters because they are defined in terms of expectations or probabilities. Nei's indices, however, become random variables through their dependence on the allelic and genotypic frequencies in the population. Therefore, his indices are more difficult to relate to theoretical investigations of population structure (Nagylaki 1989 and refs. therein; Nagylakiet al. 1993), which are usually formulated in terms of covariances of allelic frequencies or probabilities of identity in allelic state or of identity by descent.
Here, we shall combine some of the desirable properties of the treatments of Cockerham and Nei. In the next section, we shall develop Nei's approach fully and systematically for deterministic genotypic frequencies. Then we shall extend our analysis to randomly varying allelic frequencies. In the final section, we shall discuss some of our results and the interpretation of some measures of population differentiation.
DETERMINISTIC GENOTYPIC FREQUENCIES
After defining the fixation indices, we shall present the constraints they satisfy, express the indices for each homozygote in terms of the indices for heterozygotes, derive the generalization of Wright's hierarchical relationship among the indices, and evaluate the complement of each index as a ratio of heterozygosities.
The population is subdivided into an arbitrary number of subpopulations. Let wk denote the proportion of the population in subpopulation k, so that
∑kwk=1.
(1)
We consider a single autosomal locus with r alleles Ai. The frequencies of the allele Ai and the ordered genotype AiAj in subpopulation k are pi,k and Pij,k, respectively. Thus, Pij,k = Pji,k for every i and j, and the frequencies of the unordered genotypes AiAi and AiAj in subpopulation k are Pii,k and 2Pij,k for i ≠ j, respectively. Then we have
pi,k=∑jPij,k.
(2)
The frequencies of the allele Ai and the genotype AiAj in the entire population are
p‒i=∑kwkpi,k,P‒ij=∑kwkPij,k,
(3)
where the bar indicates averaging over subpopulations.
We do not restrict the action of the evolutionary forces, except that they must be deterministic. This implies, in particular, that every subpopulation must be (in principle) infinite.
We now define Nei's (1977) genotype-specific fixation indices. The subscripts I, S, and T refer to individuals, subpopulations, and the total population, respectively. The parameters FIS,ij,k and FIT,ij designate standardized measures of the deviation from Hardy-Weinberg proportions of genotype AiAj in subpopulation k and in the entire population, respectively; FST,ij signifies a standardized measure of the covariance of the frequencies of the alleles Ai and Aj:
Pii,k=pi,k2+FIS,ii,kpi,k(1−pi,k),
(4a)
Pij,k=(1−FIS,ij,k)pi,kpj,k,i≠j;
(4b)
P‒ii=p‒i2+FIT,iip‒i(1−p‒i),
(5a)
P‒ij=(1−FIT,ij)p‒ip‒j,i≠j;
(5b)
pi2¯=p‒i2+FST,iip‒i(1−p‒i),
(6a)
pipj¯=(1−FST,ij)p‒ip‒j,i≠j.
(6b)
If every subpopulation is panmictic, then (4) implies that FIS,ij,k = 0 for every i, j, and k. In this case, P‒ij=pipj¯ , so comparing (5) with (6) informs us that FIT,ij = FST,ij for every i and j.
The panmictic indices are the complements of the fixation indices:
HIS,ij,k=1−FIS,ij,k,
(7a)
HIT,ij=1−FIT,ij,
(7b)
HST,ij=1−FST,ij.
(7c)
The fixation indices satisfy some simple constraints. From (4b), (5b), and (6b) we see immediately
FIS,ij,k,FIT,ij,FST,ij≤1,i≠j.
(8)
These fixation indices can be negative. Since 0 ≤ Pii,k ≤ pi,k and 0≤P‒ii≤p‒i , from (4a) and (5a) we conclude
−pi,k1−pi,k≤FIS,ii,k≤1,−p‒i1−p‒i≤FIT,ii≤1,
(9)
which is misprinted in Chakraborty (1993). Rewriting (6) as
FST,ii=Var(pi)p‒i(1−p‒i),
(10a)
FST,ij=−Cov(pi,pj)p‒ip‒ji≠j,
(10b)
and noting that
Var(pi)≤p‒i(1−p‒i)
(11a)
and
[Cov(pi,pj)]2≤Var(pi)Var(pj)≤p‒i(1−p‒i)p‒j(1−p‒j),
(11b)
from (10) we infer
0≤FST,ii≤1
(12a)
(Chakraborty 1993) and
∣FST,ij∣≤[(1−p‒i)(1−p‒j)p‒ip‒j]1∕2,i≠j.
(12b)
Now we express the fixation indices for each homozygote in terms of the heterozygote indices, which therefore suffice for the analysis of population structure. Substituting (4) into (2) leads to
(1−pi,k)FIS,ii,k=∑j;j≠ipj,kFIS,ij,k,
(13)
which can be rewritten more compactly but less instructively as
FIS,ii,k=∑jpj,kFIS,ij,k.
Inserting (5) into the average of (2) yields (National Research Council 1996, Appendix 4A)
(1−p‒i)FIT,ii=∑j;j≠ipjFIT,ij.
(14)
Finally, substituting (6) into the equation
∑jpipj¯=p‒i,
we find that FST,ij also satisfies (14):
(1−p‒i)FST,ii=∑j;j≠ip‒jFST,ij.
(15)
Thus, in each subpopulation, the ½r(r + 1) − 1 independent genotypic frequencies can be replaced by the r − 1 independent allelic frequencies and the ½r(r − 1) heterozygote fixation indices FIS,ij,k (i ≠ j). An analogous reparametrization holds for the mean genotypic frequencies in (5) and the covariances [see (10)] in (6).
Note that if FIS,ij,k=F∼IS,k , independent of i and j, for every i and j such that i ≠ j, then (13) appropriately implies that FIS,ii,k=F∼IS,k for every i. Similar results hold for FIT,ij and FST,ij.
Next, we derive the generalization of Wright's (1943) relationship among the fixation indices. First, guided by (4), we define the weighted average of FIS,ij,k over subpopulations (Nei 1977; Wright 1978, pp. 80–81):
F‒IS,ii=1p‒i−p¯i2∑kwkpi,k(1−pi,k)FIS,ii,k,
(16a)
F‒IS,ij=1pipj¯∑kwkpi,kpj,kFIS,ij,k,i≠j.
(16b)
Inserting (8) into (16b) and (9) into (16a) demonstrates that F‒IS,ij≤1 for every i and j. Since the averages (16) are properly normalized (i.e., the sum of the weights is 1), from (7a) we have
H¯IS,ij=1−F‒IS,ij.
(17)
Note carefully that the weighting in (16) differs from that in (3).
Solving (4) for FIS,ij,k, substituting into (16), and recalling (3), we deduce (Nei 1977)
F‒IS,ii=P‒ii−pi2¯p‒i−pi2¯,
(18a)
F‒IS,ij=pipj¯−P‒ijpipj¯,i≠j.
(18b)
We insert (13) into (16a) and invoke (16b) to express every average homozygote index in terms of the average heterozygote indices:
F‒IS,ii=1p‒i−pi2¯∑j;j≠1pipj¯F‒IS,ij.
(19)
Now we can prove that
HIT,ij=H¯IS,ijHST,ij
(20)
for every genotype Ai Aj. From (18) we obtain
P‒ii=pi2¯+F‒IS,ii(p‒i−pi2¯),
(21a)
P‒ij=(1−F‒IS,ij)pipj¯,i≠j.
(21b)
For i = j, we equate (21a) to (5a), solve for FIT,ii, and invoke (6a), (7b), (7c), and (17) to establish (20). For i ≠ j, we equate (21b) to (5b), employ (7b) and (17), solve for HIT,ij, and deduce (20) from (6b) and (7c).
Finally, we express each panmictic index as a ratio of heterozygosities, or gene diversities. Let fI,k and f‒I denote the actual homozygosities in subpopulation k and
in the entire population, respectively; the corresponding heterozygosities are hI,k and h‒I :
fI,k=∑iPii,k,f‒I=∑iP‒ii=∑kwkfI,k,
(22a)
hI,k=1−fI,k=∑i,j;i≠jPij,k,
(22b)
h‒I=1−f‒I=∑i,j;i≠jP‒ij=∑kwkhI,k.
(22c)
If subpopulation k were panmictic, its homozygosity would be fS,k; if every subpopulation were panmictic, the homozygosity in the entire population would be f‒s . The corresponding heterozygosities are hS,k and h‒s . Thus,
fS,k=∑ipi,k2,f‒S=∑ipi2¯=∑kwkfS,k,
(23a)
hS,k=1−fS,k=∑i,j;i≠jpi,kpj,k=∑ipi,k(1−pi,k),
(23b)
h‒S=1−f‒S=∑i,j;i≠jpipj¯=∑i(p‒i−pi2¯)=∑kwkhS,k.
(23c)
Therefore, fS,k is the probability that two genes chosen at random from subpopulation k are the same allele; the probability that two genes chosen at random from the same subpopulation are the same allele is f‒S . The corresponding probabilities that the two genes are different alleles are hS,k and h‒S .
If the entire population were panmictic, its homozygosity and heterozygosity would become fT and hT, respectively:
fT=∑ip‒i2,
(24a)
hT=1−fT=∑i,j;i≠jp‒ip‒j=∑ip‒i(1−p‒i).
(24b)
Therefore, fT is the probability that two genes chosen at random from the entire population are the same allele; the probability that they are different alleles is hT. From (23a) and (24a) we see at once that f‒S≥fT , whence h‒S≤hT .
We shall indicate averages over genotypes by an asterisk. Consider first FIS,ij,k. Multiplying (13) by pi,k and summing over i yields the equivalent homozygote and heterozygote averages
FIS,k∗=1hS,k∑ipi,k(1−pi,k)FIS,ii,k
(25a)
=1hS,k∑i,j:i≠jpi,kpj,kFIS,ij,k,
(25b)
which are properly normalized because of (23b). Inserting (4b) into (25b) and invoking (22b) and (23b) leads to
HIS,k∗=1−FIS,k∗=hI,k∕hS,k
(26)
in every subpopulation k. Therefore, FIS,k∗ can be negative, but FIS,k∗≤1 for every k.
Recalling (23c), we define the averages of FIS,k∗ over subpopulations as
F‒IS∗=1h‒S∑kwkhS,kFIS,k∗.
(27)
Substituting (26) into (27) and employing (22c) and (23c) yields
H¯IS∗=1−F‒IS∗=h‒I∕h‒S.
(28)
This simple result, in which the numerator and denominator in (26) are averaged separately, follows from the weightings in (25) and (27). Note that F‒IS∗ can be negative, but F‒IS∗≤1 .
By substituting (25) into (27) and appealing to (16), we can also express F‒IS∗ as an average over homozygotes or heterozygotes:
F‒IS∗=1h‒S∑i(p‒i−pi2¯)F‒IS,ii
(29a)
=1h‒S∑i,j:i≠jpipj¯F‒IS,ij,
(29b)
which are properly normalized by (23c).
Now we turn to FIT,ij. Multiplying (14) by p‒i and summing over i gives the equivalent homozygote and heterozygote averages
FIT∗=1hT∑ip‒i(1−p‒i)FIT,ii
(30a)
=1hT∑i,j:i≠jp‒ip‒jFIT,ij,
(30b)
whose normalization is justified by (24). Inserting (5b) into (30b) and utilizing (22c) and (24b), we obtain
HIT∗=1−FIT∗=h‒I∕hT.
(31)
Therefore, FIT∗≤1 , but FIT∗ can be negative.
For FST,ij, from (15) we get
FST∗=1hT∑ip‒i(1−p‒i)FST,ii
(32a)
=1hT∑ij:i≠jp‒ip‒jFST,ij.
(32b)
Substituting (6b) into (32b) and using (23c) and (24b), we find
HST∗=1−FST∗=h‒S∕hT.
(33)
Since hT≥h‒S≥0 , we have 0≤FST∗≤1 .
From (28), (31), and (33) we infer at once the hierarchical formula
HIT∗=H¯IS∗HST∗.
(34)
Nei (1977) derived (28), (31), (33), and (34) for homozygotes. Our treatment establishes these results also for heterozygotes. Observe from (34) that when (20) is averaged over genotypes, the factors on the right-hand side are averaged separately. This occurs because the weightings in (30) and (32) differ from those in (29).
In the above analysis, we posited a discretely subdivided population. However, if we restrict our attention to FIT,ij, this assumption becomes unnecessary. Indeed, the definitions (5), (22c), and (24) involve only allelic and genotypic frequencies in the entire population. Therefore, (14), (30), and (31) hold for arbitrary population structure.
STOCHASTIC ALLELIC FREQUENCIES
Here, we shall extend the analysis in the last section to randomly varying allelic frequencies, which may reflect finite subpopulation numbers or random variation in evolutionary forces. In this case, it is obvious that Nei's (1977) definitions (4), (5), and (6) lead to fixation indices that are random variables. Indeed, since (26), (28), (31), and (33) are ratios of random heterozygosities, even their expectations are difficult to evaluate and to relate to theoretical studies of population structure, which are usually formulated in terms of covariances of allelic frequencies or probabilities of identity in allelic state or of identity by descent. The fixation indices we shall define are parameters.
We shall examine only the allelic frequencies. These are of greatest evolutionary interest and suffice for most theoretical investigations of population structure, which are usually restricted to panmictic subpopulations. To account for random variation, we imagine that the population T, which comprises the subpopulations S, is replicated infinitely many times to form the metapopulation U. Each of these replicates is an independent realization of the evolutionary process, so U is an infinite collection of such realizations. We do not assume that the subpopulations S are panmictic.
The arrangement of this section is the same as that of the preceding one.
The allelic frequencies pi,k are now random variables. As in the last section, a bar indicates averages over subpopulations S within the population T:
p‒i=∑kwkpi,k.
(35)
Of course, p‒i is now a random variable. For typographical simplicity, we use an angle bracket to signify averages over evolutionary realizations (or sample paths). Thus, 〈pi,k〉 is averaged over T within U, and the grand mean of the frequency of Ai is
πi≡E(pi)=〈p‒i〉.
(36)
Analogy with (21), (5), and (6) suggests the definitions
〈pi2¯〉=〈p‒i2〉+FST,ii〈p‒i(1−p‒i)〉
(37a)
〈pipj¯〉=(1−FST,ij)〈p‒ip‒j〉,i≠j;
(37b)
〈pi2¯〉=πi2+FSU,iiπi(1−πi),
(38a)
〈pipj¯〉=(1−FSU,ij)πiπj,i≠j;
(38b)
〈p‒i2〉=πi2+FTU,iiπi(1−πj),
(39a)
〈p‒ip‒j〉=(1−FTU,ij)πiπj,i≠j.
(39b)
As in (7), the panmictic indices are the complements of the above fixation indices.
Solving (37) to (39) for the fixation indices yields
FST,ii=〈Var(pi∣T)〉〈p‒i(1−p‒i)〉,
(40a)
FST,ij=−〈Cov(pi,pj∣T)〉〈p‒ip‒j〉,i≠j;
(40b)
FSU,ii=Var(pi)πi(1−πi),
(41a)
FSU,ij=−Cov(pi,pj)πiπj,i≠j;
(41b)
FTU,ii=Var(p‒i)πi(1−πi),
(42a)
FTU,ij=−Cov(p‒i,p‒j)πiπj,i≠j.
(42b)
A glance at (37b), (38b), and (39b) immediately reveals the constraints
FST,ij,FSU,ij,FTU,ij≤1,i≠j.
(43)
These fixation indices can be negative. Reasoning as in (11), from (40a), (41a), and (42a) we deduce
0≤FST,ii,FSU,ii,FTU,ii≤1.
(44)
Bounds corresponding to (12b) are easy to derive, but are too complicated to be illuminating.
We can easily derive the remaining results in this section ab initio, but we can obtain them more quickly by the following transformation. In (21), (5), and (6), we drop the bar from F‒IS,ij ; make the substitutions I → S, S → T, and T → U; replace the bars by angle brackets; and finally substitute Pij→pipj¯ and pi→p‒i . This transformation yields pipj¯→〈pipj¯〉 and p‒i→πi . Then (21), (5), and (6) become (37), (38), and (39), respectively.
To express the fixation indices for each homozygote in terms of the heterozygote indices, we apply our transformation to (19), (14), and (15), which become, respectively,
FST,ii=1〈p‒i(1−p‒i)〉∑j:j≠i〈p‒ip‒j〉FST,ij,
(45a)
FSU,ii=11−πi∑j:j≠iπjFSU,ij,
(45b)
FTU,ii=11−πi∑j:j≠iπjFTU,ij.
(45c)
The generalization (20) of Wright's relationship among the fixation indices becomes
HSU,ij=HST,ijHTU,ij
(46)
for every i and j.
Finally, we express each panmictic index as a ratio of expected heterozygosities. If every subpopulation S were panmictic, the expected homozygosity and heterozygosity in the entire population T would be f‒S and h‒S , respectively. Thus, in this case, f‒S and h‒S are the homozygosity and heterozygosity in the metapopulation U:
f‒S=∑i〈pi2¯〉,
(47a)
h‒S=1−f‒S=∑i,j:i≠j〈pipj¯〉=∑i〈p‒i−pi2¯〉.
(47b)
If the entire population T were panmictic, these expectations would become
fT=∑i〈p‒i2〉,
(48a)
hT=1−fT=∑i,j:i≠j〈p‒ip‒j〉=∑i〈p‒i(1−p‒i)〉.
(48b)
If the metapopulation U were panmictic, its homozygosity and heterozygosity would be
fU=∑iπi2,
(49a)
hU=1−fU=∑i,j:i≠jπiπj=∑iπi(1−πi).
(49b)
Note that the definitions (47), (48), and (49) follow from the transformation of (22), (23), and (24), respectively.
From (47a), (48a), and (49a) we obtain easily f‒S ≥ fT ≥ fU, which implies that hU ≤ hT ≤ h‒S .
To average FST,ij over homozygotes or heterozygotes, we transform (29):
FST∗=1hT∑i〈p‒i(1−p‒i)〉FST,ii
(50a)
=1hT∑i,j:i≠j〈p‒ip‒j〉FST,ij,
(50b)
for which (28) yields
HST∗=1−FST∗=h‒S∕hT.
(51)
For FSU,ij, from (30) and (31) we obtain
FSU∗=1hU∑iπi(1−πi)FSU,ii
(52a)
=1hU∑i,j:i≠jπiπjFSU,ij.
(52b)
HSU∗=1−FSU∗=h‒S∕hU.
(53)
For FTU,ij, from (32) and (33) we get
FTU∗=1hU∑iπi(1−πj)FTU,ii
(54a)
=1hU∑i,j:i≠jπiπjFTU,ij,
(54b)
HTU∗=1−FTU∗=hT∕hU.
(55)
Since h‒S ≥ hT ≥ hU ≥ 0, the results (51), (53), and (55) inform us that
0≤FST∗,FSU∗FTU∗≤1,
which also follows easily from (44), (50a), (52a), and (54a).
From (51), (53), and (55) we establish immediately the hierarchical result
HSU∗=HST∗HTU∗,
(56)
in accordance with (34).
The panmictic index HST∗ is a measure of variation between subpopulations. Our development justifies the use of (51) for this parameter in theoretical investigations (see, e.g., Takahata 1983; Crow and Aoki 1984; Takahata and Nei 1984; Slatkin and Barton 1989; Slatkin 1991, 1993), and the ratio (51) of expected heterozygosities may also be preferable for data analysis to the expectation of the ratio of random heterozygosities (Nei and Chakravarti 1977; Neiet al. 1977). Substituting (47) and (48) into (51) produces the explicit formula
HST∗=1−∑i〈pi2¯〉1−∑i〈p‒i2〉.
(57)
DISCUSSION
Without restricting the evolutionary forces that may be present, we have developed systematically the theory of fixation indices in an arbitrarily subdivided population. Our indices are parameters, rather than random variables. To estimate the pattern and strength of evolutionary forces (such as migration) from the above theory, a model must be specified and used to derive formulas for the fixation indices, as in examples 3 and 4 at the end of this section.
The formulas (26), (28), (31), (33), (51), (53), and (55) for the panmictic indices all have the same simple form: if B is a finer level of subdivision than C, then
HBC=hB∕hC,
(58)
where hX designates the expected heterozygosity with random mating within subdivisions at level X. Then not only are the hierarchical relations (34) and (56) obvious, but so is their extension to further nested subdivision (Wright 1969, p. 295). Thus, if R, S, T, and U signify increasingly coarse subdivision, we have
HRU=HRSHSTHTU.
(59)
We proceed to discuss the interpretation of some measures of population differentiation. According to (10a) and (12a), the fixation index FST,ii is a standardized measure of the intersubpopulation variance of the frequency pi of the allele Ai. By (10b), the corresponding covariance measure for the frequencies of Ai and Aj is FST,ij. If every subpopulation is panmictic, then FIT,ij = FST,ij for every i and j, and therefore (5) shows that the parameters FST,ij yield the genotypic frequencies in the entire population.
Now consider in more depth the interpretation of the homozygote or heterozygote average index FST∗ , defined by (32) and evaluated in (33). Wright (1978, p. 82) noted and exemplified that FST∗ measures “the amount of differentiation among subpopulations, relative to the limiting amount under complete fixation” and that FST∗ is “not a measure of degree of differentiation in the sense implied in the extreme case by absence of any common allele. It measures differentiation within the total array in the sense of the extent to which the process of fixation has gone toward completion.” These is an appropriate measure observations suggest that FST∗ of differentiation in a population with low genetic diversity, but that it may be misleading in a highly diverse population. Below, we develop this idea more precisely and illustrate it by four examples.
Since nucleotide diversities are generally low, therefore FST∗ is usually a suitable measure of differentiation at the nucleotide or codon level.
We separate the cases of high and low genetic diversity and use the criteria of Kimura and Maruyama (1971); see also Nagylaki (1983, 1985, 1986).
Our index of genetic diversity is the effective number of alleles (Kimura and Crow 1964; Maruyama 1970)
ne=1∕fT,
(60)
where fT is given by (24a) or (48a). In an infinite, panmictic population with l alleles, it is trivial to prove that ne ≤ l, with equality if and only if all the alleles are equally frequent (Nagylaki 1992, pp. 29–30). Diversity is high if ne ⪢ 1 and low if ne ≈ 1.
For high diversity, our measure of gene-frequency differentiation is fT∕f‒S . We shall say that differentiation is strong if fT≪f‒S (defined as fT∕f‒S≪1 and weak if fT≈f‒S (recall that fT≤f‒S ).
For low diversity, the ratio fT∕f‒S is insensitive to differentiation because fT≈f‒S≈1 . A more sensitive measure is h‒S∕hT : strong and weak differentiation correspond to h‒S≪hT and h‒S≈hT , respectively.
Now consider
FST∗=hT−h‒ShT=f‒S−fT1−fT.
(61)
For low diversity, our criteria are, indeed, equivalent to FST∗≈1 if differentiation is strong and to FST∗≈1 if it is weak. For high diversity, however, FST∗≈f‒S−fT , so if fT≪f‒S≪1 , then differentiation is strong yet FST∗≪1 ; thus, strong differentiation does not imply that FST∗≈1 . Weak differentiation does imply that FST∗≪1 .
Example 1: Suppose that there are K subpopulations, of which L (0 < L < K) are fixed for A1 and K − L for A2. Then (23c) and (24b) give h‒S=0 and hT > 0, whence (33) yields FST∗=1 . This indicates that every subpopulation is fixed, and not all for the same allele. Since there are only two alleles, however, complete differentiation between subpopulations (in the sense of having no common alleles) is possible only for two subpopulations.
Example 2: By contrast, consider n subpopulations of the same size, without common alleles, each with homozygosity fS. Then fT = 1/nfS, so from (33) we obtain
FST∗=(n−1)fSn−fS.
(62)
Thus, FST∗<1 unless fS = 1, even though the subpopulations are fully differentiated. Furthermore, FST∗≈1 if fS ≈ 1, whereas FST∗≪1 if fS ⪡ 1. The second possibility is misleading unless carefully interpreted. For high diversity, fS ⪡ n (which must always hold if n ⪢ 1), so FST∗≪1 for small n, and this result can occur for any n. If diversity is low, then fS ≈ 1 and n must be small, which correctly implies that FST∗≈1 .
Two special cases illustrate the above observations. If n ⪢ 1, then FST∗≈fS . If each subpopulation has l equally frequent alleles, then fS = 1/l, and hence FST∗ = (n − 1) / (nl − 1).
Example 3: Our third example is the island model (Moran 1959; Maruyama 1970; Maynard Smith 1970; Nagylaki 1983, 1986, and refs. therein). Generations are discrete and nonoverlapping. Each of n (≥2) panmictic (including selfing) subpopulations comprises N monoecious, diploid individuals. These colonies exchange gametes with no spatial effect on dispersion, i.e., if the migration rate is m (0 < m < 1), every colony receives a proportion m/(n − 1) of its gametes from each of the other colonies. Selection is absent, and every allele mutates to new alleles at the same rate u (0 ≤ u ≤ 1).
We posit that migration is weak and that mutation is weak relative to the stronger one of migration and random drift:
m≪1andu≪max(m,1∕N).
(63)
Then, at equilibrium,
ne≈n[m+u(4mNT+n−1)]nm+(n−1)u
(64)
(Nagylaki 1983), where NT = nN represents the total population number;
FST∗≈14Nmα+1,
(65)
where α = [n/(n − 1)]2 (Nei 1975, p. 123; Nagylaki 1983; Takahata 1983; Crow and Aoki 1984; Takahata and Nei 1984; Cockerham and Weir 1987); and differentiation is strong if and only if
4mN≪max(1,4NTu)
(66a)
and weak if and only if
4mN≫max(1,4NTu)
(66b)
(Nagylaki 1986). Using FST∗ to assess differentiation would replace (66a) and (66b) by 4mN ⪡ 1 and 4mN ⪢ 1, respectively, which is correct if and only if 4NTu ≤ 1. Thus, FST∗ provides the correct criterion for differentiation if and only if diversity is low (cf. Nagylaki 1983, 1986).
Example 4: Our last example is the unbounded, unidimensional stepping-stone model (Malécot 1949, 1950, 1951; Kimura 1953; Nagylaki 1989, and refs. therein). As in the island model, generations are discrete and nonoverlapping; selection is absent; and every allele mutates to new alleles at the same rate u (0 ≤ u ≤ 1). There are panmictic (including selfing) colonies of N monoecious, diploid individuals at all the integers. These demes exchange gametes at rates that depend on displacement, but not on initial and final positions separately, i.e., dispersion is homogeneous.
Let w denote the separation between the demes from which genes are sampled. We write the variance of the single-generation gametic displacement as ½σ2 and introduce the scaled, dimensionless separation
ξ=2uw∕σ.
(67)
For weak mutation (u ⪡ 1) and large neighborhood size (Nσ ⪢ 1), the probability at equilibrium that two distinct genes sampled from demes separated by a distance w (≥0) are the same allele is adequately approximated by (Nagylaki 1989, and refs. therein)
f(ξ)≈e−ξ1+β,
(68)
where β=4Nσu designates a dimensionless parameter. We set
h(ξ)=1−f(ξ).
(69)
The expected heterozygosity
h(0)≈β1+β.
(70)
is high if β ≳ 1 and low if β ⪡ 1.
Now consider two demes with scaled separation ξ. The effective number of alleles in these two demes is
ne=2f(0)+f(ξ)≈2(1+β)1+e−ξ,
(71)
so their diversity is high if β ⪢ 1 and low if β ≲ 1.
For high diversity, we use f(ξ)/f(0) as a simple index of differentiation between the two demes. Therefore, differentiation is strong if e−ξ ⪡ 1 and weak if e−ξ ≈ 1, independent of β. For low diversity, the measure h(0)/h(ξ) reveals that differentiation is strong if
β≪1−e−ξ
(72a)
and weak if
β≫1−e−ξ.
(72b)
From (61) we obtain
FST∗(ξ)=h(ξ)−h(0)h(ξ)+h(0)≈1+e−ξ1+2β−e−ξ.
(73)
Again, FST∗ yields the correct criterion for differentiation, if and only if diversity is low.
Acknowledgments
I thank Brian Charlesworth, James F. Crow, and Magnus Nordborg for useful comments on the manuscript. This work was supported by National Science Foundation grant DEB-9706912.
- Received April 30, 1997.
- Accepted October 3, 1997.
- Copyright © 1998 by the Genetics Society of America
