Bottlenecks, Genetic Polymorphism and Speciation
Masatoshi Nei

Anecdotal, Historical and Critical Commentaries on Genetics Edited by James F. Crow and William F. Dove

THIRTY years ago Takeo Maruyama, Ranajit Chakraborty, and I published an article about the effects of bottlenecks on genetic variation (Nei et al. 1975). This article was motivated by two controversies occurring in evolutionary genetics at that time. One was the selectionist-neutralist controversy concerning the maintenance of protein polymorphism, and the other was the controversy over Mayr's (1963) and Carson's (1971) idea that speciation is caused by the genetic revolution that occurs when population size is drastically reduced by bottleneck effects. I do not think that we resolved any of these controversies to everyone's satisfaction, but we clarified several aspects of the bottleneck effects in evolutionary genetics.

The first controversy was initiated when protein electrophoresis revealed a large amount of genetic variation in natural populations (Shaw 1965; Harris 1966; Lewontin and Hubby 1966). Historically, this controversy was a new version of the previous controversy concerning the maintenance of genetic variation. In the 1950s, population geneticists were divided into two camps, one camp supporting the “classical” theory and the other the “balance” theory (Dobzhansky 1955). The “classical” theory asserted that most genetic variation within species is maintained by mutation-selection balance, whereas the “balance” theory proposed that genetic variation is maintained primarily by overdominant selection or some other type of balancing selection. The major supporters of the “classical” theory were H. J. Muller, James Crow, and Motoo Kimura, and those supporting the “balance” theory were Theodosius Dobzhansky, Bruce Wallace, and E. B. Ford. During this controversy, it became clear that the amount of genetic variation maintained by overdominant selection can be much greater than that maintained by mutation-selection balance, but that overdominant selection incurs a large genetic load (genetic death or fertility excess required) that may not be bearable by mammalian organisms when the number of loci is large (Kimura and Crow 1964). For this reason, Lewontin and Hubby (1966) could not decide between the two hypotheses when they found electrophoretic variation. However, Sved et al. (1967), King (1967), and Milkman (1967) proposed that this genetic load can be reduced substantially if natural selection occurs by choosing only individuals in which the number of heterozygous loci is greater than a certain number (truncation selection). Soon after these articles were published, Crow (1970) and Nei (1971) argued that, unlike artificial selection, natural selection does not involve truncation selection. In the meantime, Robertson (1967), Crow (1968), and Kimura (1968) suggested that most protein polymorphisms are probably neutral and that the wild-type alleles in the “classical” hypothesis are actually composed of many iso-alleles or neutral alleles.

However, the “balance” camp did not accept this suggestion, because they believed that almost all genetic polymorphisms were maintained by balancing selection (Clarke 1971). At that time, surveying the average heterozygosity (H) for a number of species, Lewontin (1974)(p. 208) and Ohta (1974) noted that H was ∼6–18% for most species, irrespective of its population size. If the neutral theory were correct, H should increase with increasing population size (N). Therefore, Lewontin and Ohta took this observation as evidence against the neutral theory. Ayala (1972) estimated that the effective population size of Drosophila willistoni in South America is at least 109 so that the expected heterozygosity would be 0.976 if the mutation rate is 10−7/locus/year or 10−8/locus/generation. Under the neutral theory, the expected heterozygosity in an equibrium population is given by 4Nv/(4Nv + 1), where v is the mutation rate per generation. This expected heterozygosity was far greater than the observed value, i.e., 0.183. Because of this large discrepancy between the expected and the observed heterozygosities, Ayala concluded that the neutral theory must be wrong.

Ohta (1974) attempted to resolve this discrepancy by proposing that most mutations are slightly deleterious compared with the wild-type alleles. She argued that in small populations the slightly deleterious alleles would behave as though they are neutral and contribute substantially to the heterozygosity. In large populations, however, most slightly deleterious alleles will be selected against so that the average heterozygosity would not increase above that of small populations.

I was not convinced by this argument, because evolution cannot occur by the continuous accumulation of deleterious mutation except for some special cases such as the reduction of unused characters (Darwin 1859). If Ohta's hypothesis were correct, every gene would eventually deteriorate under the accumulation of deleterious mutations. In my view, a more plausible explanation was that natural populations occasionally go through bottlenecks and for this reason the long-term effective size is much smaller than the actual size. For example, the human population is currently >6 billion, but archeology and recent population surveys indicate that human populations were small until ∼10,000 years ago. Since heterozygosity increases only by new mutations, we would not expect a high level of heterozygosity for human populations even if the current population size is enormous.

The second controversy was concerned with speciation. Mayr (1963) had argued that most new species are formed when a population goes through a small bottleneck. He suggested that a kind of genetic revolution occurs when population size is reduced and this genetic revolution is the source of formation of new species (Provine 2004). Carson (1971) modified this hypothesis by proposing that the population bottleneck alone is not sufficient for the occurrence of a genetic revolution. It is necessary for the population size first to increase (population flush) before the bottleneck. This was because Carson assumed that a special combination of interacting (epistatic) genes is responsible for the formation of a new species and that the chance for creating a new combination of epistatic genes will be enhanced when population flush occurs before a bottleneck. The reason was that population flush would loosen the original combination of epistatic genes, producing a new combination before a bottleneck occurs. Both Mayr and Carson thought that a new species could be formed even from one fertilized female representing two parental individuals. Carson (1971) suggested that many species of Hawaiian Drosophila were formed in this way when a fertilized female was blown by wind from one island to another. Although geneticists were generally skeptical of this hypothesis, it was quite popular among evolutionists. I thought that it would be difficult to test this hypothesis without conducting experiments, but a theoretical study might help to understand some aspects of the hypothesis.

One day in 1974 I discussed these problems with Chakraborty, and we decided to study the bottleneck effect mathematically. We used the following recurrence formula for expected homozygosity (J), Math1where Jt is the homozygosity (not the inbreeding coefficient) in the tth generation and Nt−1 is the effective population size in the (t − 1)th generation. Later Maruyama, who was visiting my lab, joined us in this study. He invented a general formula for Jt, but we did not use it because the computation was more complicated than that of Equation 1. We assumed that the original population size was N0 and that population size (N) goes through a bottleneck (Nb). We also assumed that after the bottleneck N increases following a sigmoid curve to return to the original size, N0. Obviously, the expected heterozygosity (H) is given by H = 1 − J. We also studied the bottleneck effect on the expected number of alleles per locus.

The main conclusions obtained by this study were as follows:

  1. To have a drastic reduction of genetic variation or heterozygosity, the bottleneck size (Nb) must be very small relative to N0 and the rate of population growth (r) must be low. Otherwise, the reduction of heterozygosity is quite small. Even if Nb is 2, the reduction in H is <50% if N0 = 4 × 106, v = 10−8, and r is >0.5 (Figure 1).

    Figure 1.—

    Changes in average heterozygosity when population size goes through a bottleneck. The solid lines refer to the case where the bottleneck size (Nb) is 2, while the dashed lines refer to the case of Nb = 10. The population growth is assumed to be logistic, and r stands for the intrinsic rate of growth. The original and eventual heterozygosity is 0.138. Generations are given on a logarithmic scale, starting from the original population as generation 1. Reproduced from Nei et al. (1975).

  2. If Nb is >10, the amount of reduction in H is <50% unless r is <0.1.

  3. Once heterozygosity is reduced, it takes ∼1/v generations for H to recover its original level. This is true even if N increases rapidly to the original size (N0). This slow recovery of H occurs because the increase of heterozygosity is caused by new mutations and their slow increase in frequency by genetic drift. Therefore, it is possible that the heterozygosity is very low after the bottleneck even if the population size becomes as large as the original size for a large number of generations.

  4. When a population goes through a bottleneck, the reduction of the number of alleles per locus occurs faster than that of heterozygosity, but if the population size increases, the number of alleles increases more rapidly than heterozygosity. This happens because new mutations are counted as different alleles even if their frequencies are low.

The implications of these findings on genetic polymorphism were quite clear:

  1. To compute the expected heterozygosity, we must consider a long-term effective size representing the harmonic means of N for the last hundreds or thousands of generations. Since the population size of many species has expanded after the last glaciation (∼10,000 years ago), the expected heterozygosity computed by using the current size is likely to be higher than the observed heterozygosity. This is one explanation for the approximate constancy of heterozygosity observed by Lewontin (1974). Actually, later studies with a much larger number of species have shown that average heterozygosity varies considerably among species and tends to be higher in large populations than in small populations (Nei 1975; Nei and Graur 1984).

  2. The neutrality test based on the assumption of constant population size may give erroneous conclusions.

  3. Considering these factors and others, Nei (1987)(1988) concluded that the observed patterns of protein polymorphism do not deviate significantly from the neutral expectations. Of course, the selectionist-neutralist controversy is still continuing at finer levels of DNA sequences, but this is beyond the scope of this article.

The implications of our study on speciation through bottlenecks were less clear, because we did not consider epistatic genes. However, we concluded that a random loss of many different alleles through bottlenecks would make it difficult to have a new favorable epistatic combination. Later, Nei et al. (1983) developed a mathematical model of speciation considering developmental incompatibility genes and showed that speciation may occur more rapidly in small populations than in large ones. On the basis of this finding, they argued that the probability of fixation of different incompatibility alleles in different populations may be enhanced by bottlenecks. Yet, this was not support of Mayr and Carson's genetic revolution hypothesis, because the latter hypothesis was not based on any explicit genetic model. In contrast, Klein et al. (1990) showed that speciation can occur without bottlenecks. Their statistical analysis of genetic polymorphisms shared by humans and chimpanzees at the major histocompatibility complex loci suggested that modern humans probably have evolved without any appreciable bottleneck effects.

Our 1975 article was cited quite often because of the above controversies and became a citation classic in 1989. The Thompson ISI Web of Science indicates that it had been cited 1059 times by 2004 and that the number of citations per year has recently increased. It seems that more researchers are now interested in the problems that we studied 30 years ago.

Takeo Maruyama, who contributed significantly to the 1975 article as well as to our 1983 article, died unexpectedly in 1987 at the age of 51. He had mastered the stochastic theory of population genetics and could solve mathematical problems very quickly. Yet he was a humble man, deeply interested in biological issues. The idea of establishing the DNA Data Bank of Japan at the National Institute of Genetics in Mishima, Japan, was conceived by him. This bank and its associated research center, the Center for Information Biology, are now flourishing, and the latter has become a major research center of evolutionary genomics in Japan, if not in the world.


I thank Ranajit Chakraborty, Jan Klein, and Jongmin Nam for their comments on an earlier version of this article.


  • This article is dedicated to the memory of Takeo Maruyama.