Multilevel Selection 1: Quantitative Genetics of Inheritance and Response to Selection
Piter Bijma, William M. Muir, Johan A. M. Van Arendonk


Interaction among individuals is universal, both in animals and in plants, and substantially affects evolution of natural populations and responses to artificial selection in agriculture. Although quantitative genetics has successfully been applied to many traits, it does not provide a general theory accounting for interaction among individuals and selection acting on multiple levels. Consequently, current quantitative genetic theory fails to explain why some traits do not respond to selection among individuals, but respond greatly to selection among groups. Understanding the full impacts of heritable interactions on the outcomes of selection requires a quantitative genetic framework including all levels of selection and relatedness. Here we present such a framework and provide expressions for the response to selection. Results show that interaction among individuals may create substantial heritable variation, which is hidden to classical analyses. Selection acting on higher levels of organization captures this hidden variation and therefore always yields positive response, whereas individual selection may yield response in the opposite direction. Our work provides testable predictions of response to multilevel selection and reduces to classical theory in the absence of interaction. Statistical methodology provided elsewhere enables empirical application of our work to both natural and domestic populations.

IT is universally recognized that all plants and animals compete within or across species. These competitive interactions have important implications both for domestic breeding and for the outcome of evolutionary processes (Goodnight and Stevens 1997; Keller 1999; Clutton-Brock 2002). With respect to domestication and agriculture, reduction of competition and fighting and sharing of resources is critical for improving animal well-being and productivity in confined high-intensity rearing conditions (Muir 2005). Competition and fighting behavior is also a major limitation as to which species can be domesticated (Diamond 2002). Understanding how to reduce competitive interactions in artificial breeding programs could improve animal well-being of those species that are currently being used in animal agriculture, such as swine and poultry (Denison et al. 2003; Muir 2003), and also expand the range of species that can be domesticated, such as carnivorous and/or cannibalistic shell- and game fish. Even with domesticated species, classic quantitative genetic theory fails to explain why some traits, in particular those related to behavior, fail to respond to selection, even though there is heritable variation and a positive selection differential (e.g., Teichertcoddington and Smitherman 1988; Vangen 1993; Kruuk et al. 2001). Generalization of quantitative genetic theory is required to understand how interactions among individuals (Wolf et al. 1998) and selection acting on multiple levels of organization affect response to selection (Griffing 1967; Goodnight 2005).

With respect to evolutionary outcomes, there are several important issues that require a general concept of how competition influences adaptations within and between species. One of those issues is the evolution of altruism (Wilson 1985, 2005; Woodcock and Heath 2002; Fehr and Fischbacher 2003; Cooper and Wallace 2004; Hamilton and Taborsky 2005; Okasha 2005) and the importance of kin selection to this process (Hamilton 1964; Michod 1982; Day and Taylor 1997; Griffin and West 2002; Axelrod et al. 2004; Goodnight 2005; Wilson 2005). At the heart of the debate is how cooperation and altruism can persist in the face of cheating (Wade and Breden 1980; Hamilton and Taborsky 2005). Some have suggested that the solution to this problem is the level of selection (Slatkin and Wade 1978; Wade 1978; Wilson and Sober 1994; Keller 1999; Goodnight 2005; Wilson 2005). In both biology and the human sciences, social groups are sometimes treated as adaptive units, whose organization cannot be reduced to the individual level. In this view, group-level adaptations can evolve only by a process of natural selection acting at the group level (Wilson and Sober 1994). This group-level view is opposed by a more individualistic one that treats social organization as a by-product of self-interest, suggesting that altruism can evolve through individual selection depending on the degree of relatedness within a group (Hamilton 1964; Wade 1978b, 1980; Michod 1982). More recent approaches treat multilevel selection as a continuum, in which fitnesses of individuals depend on both individual and group properties, of which pure group selection and individual selection are limiting cases (Keller 1999).

The full impacts of heritable interactions on evolutionary outcomes are still not fully understood, partially due to the lack of a quantitative genetic framework including all levels of selection in groups with varying degrees of relatedness. In contrast, a special case of heritable interaction has a long and successful history in the field of animal breeding in the form of the maternal-effects model (Willham 1963; Cheverud 1984), in which phenotypic trait values of offspring are the sum of a direct effect of the offspring and a maternal effect of its dam. Animal breeders have developed general methods to estimate direct and maternal genetic effects and have successfully improved numerous domestic breeds using those methods (Willham 1963; Balcerzak et al. 1989). This suggests that, to a large degree, we can draw on experiences from quantitative genetics and artificial breeding programs to help understand how to estimate and interpret the effects of heritable interactions in natural populations and utilize them in artificial selection programs.

In all of these cases, there is a need for a general framework to provide testable quantitative predictions of the outcomes of selection, at all levels of organization and degrees of relatedness. Across a series of articles, Griffing (1967, 1976a,b, 1981a,b) attempted to provide such a framework. Unfortunately the work of Griffing has had limited impact for a variety of reasons. Primary among these were the inability to estimate parameters associated with the model and the difficulty of derivations that were limited to specific cases and could not be generalized to arbitrary group structures, degrees of relatedness, and levels of selection.

Here we provide the basics of a general quantitative genetic framework to predict response to multilevel selection for interaction among any number of individuals of any degree of relatedness. In the absence of interaction, that framework will reduce to classical quantitative genetic theory of response to selection. In this article, we present results for selection for a single trait in a panmictic population of a single species. The focus is on identifying the heritable components affecting the phenotype and on their response to multilevel selection. Results lay the foundation for extensions to numerous applications to agriculture and evolution, which are addressed in the discussion.


In this section we present the basic theory for the consequences of social interaction in populations undergoing multilevel selection, which is most clearly illustrated with a simple model. We, therefore, first consider a large noninbred panmictic population with discrete generations and random mating. At the end of this section we generalize our results to structured populations with inbreeding and nonrandom mating.

In the population, interaction is among individuals living in groups of n individuals each. A group defines merely the unit of organization upon which selection may operate and not necessarily physically separated units. Within that level of organization, individuals are associated; interaction among associated individuals may affect their phenotypes and subsequent selective success. These interactions can be social, as with higher organisms, or physical, due to competition for limited recourses, such as space, light, or nutrients, as with plants and lower organisms. Association among group members may be due to distance, e.g., for trees and other plants; immobile animals, such as corals; social structure, such as pack animals and social insects; or physical containment, e.g., for organisms on small islands, aquatic organisms in isolated pools or ponds, and domesticated animals reared in cages or pens. As such, interactions among associating individuals are referred to as associative effects (Griffing 1967). After selection, mating is at random at the population level, i.e., across groups.

To enhance interpretation, but without loss of generality, we assume constant group size and relatedness among group members. In the following we describe the trait model, the multilevel selection process, and response to selection, using the tools of quantitative genetics.

The quantitative genetic trait model:

Consider a single quantitative trait. The observed phenotypic value, Pi, of individual i is due to two unobserved phenotypic effects: a direct effect (Math) originating from the genes and the physical environment of individual i, and the sum of associative effects (Math) originating from of each of its n − 1 group members (Griffing 1967). The observed phenotypic value of each individual, therefore, is the sum of its own direct effect and the summed associative effects of its n − 1 group members,Math(1)(see Table 1 for notation). Equation 1 is general; it states merely that observed phenotypes are affected both by the individuals carrying the phenotypes and by the individuals they associate with. Each individual may affect both itself and its associates. Thus each individual has two unobserved effects: a direct effect expressed in its own phenotype and an associative effect expressed in the phenotypes of its associates. The associative effect may be interpreted as a heritable environmental effect provided by associates to the focal individual (Wolf et al. 1998). A maternal-effects model (Willham 1963), where association is between mother and offspring and the focal individual is the offspring, is a well-known special case of Equation 1.

View this table:

Notation key

The meaning of direct and associative effects may be clarified by comparing Equation 1 to the usual model for an altruistic behavior. With an altruistic behavior, the effects of interactions are usually defined in terms of fitness cost and benefit, whereas Equation 1 refers to effects on a trait value. The effect of the behavior on the phenotypic value of the focal individual (FI), i.e., the direct effect of the FI, is an analogy of cost, whereas the effect on the phenotypic value of an associate, i.e., the associative effect of the FI, is an analogy of benefit. In general, however, direct and associative effects are random quantitative variables that can take either negative or positive values, and values can differ among individuals. Moreover, the direct effect of an individual covers its full effect on its own phenotypic value; it is not restricted to the part related to interaction with other individuals.

Following quantitative genetic theory, phenotypic direct and associative effects in Equation 1 are both decomposed into a heritable component, referred to as the breeding value (A), and a nonheritable component, referred to as the residual or the environment (E),Math(2)in which Math is the direct breeding value (DBV) of individual i, and Math is the associative breeding value (SBV) of associate j. The DBV is equivalent to the classical breeding value (Lynch and Walsh 1998), whereas the SBV is a generalization of a breeding value for maternal effect (Willham 1963) to a breeding value for association between any type of individuals (Griffing 1967; Wolf et al. 1998). The DBV and the SBV represent the heritable components of the direct and associative effects, and both DBV and SBV may respond to selection.

Multilevel selection:

This section describes a within-population multilevel selection process, which may refer either to an artificially applied strategy in agriculture or the laboratory or to a natural population. We assume merely presence of multilevel selection; our interest is not in the origin of the selective forces. In this section we consider two levels, but later on we generalize the concept to any number of levels affecting the selective success of genes or breeding values. On the population level, selection among individuals may depend both on individual phenotypes and on phenotypes of group members. A general expression for the selection criterion, allowing differential emphasis on individual and group, isMath(3)in which Math is the sum of trait values of group members, and Pi and Pj are combinations of direct and associative effects as defined in Equation 1. Selection is for C on the population level. The factor g represents the degree to which selection operates on the group vs. the individual, i.e., the degree of multilevel selection. A g = 0 represents selection on individual phenotypes Pi, resulting in selection occurring among individuals without regard to their group. A g = 1 represents selection on the summed trait values of all group members including i, resulting in selection occurring among groups. A Math represents selection on the deviation of individual trait value from the mean trait value of the group members, Math, resulting in selection occurring on the relative performance of individuals within their group (i.e., soft selection). In general, values of g correspond to different combinations of between- and within-group selection, i.e., to different partitioning of individual fitness into within- and between-group components.

With artificial selection, Equation 3 may be interpreted as a selection index: the breeder chooses a value for g to purposely apply a certain selection strategy; e.g., g = 1 to apply selection between groups (Muir 1996). With natural selection, g will depend on the contributions of individual and group trait values to individual fitness, which may be estimated using contextual analyses (Goodnight et al. 1992; Stevens et al. 1995; Goodnight and Stevens 1997; Goodnight 2005). For example, groups with high trait values may acquire better habitats, increasing fitness of all members, so that g > 0.

Response to selection:

Both direct and associative effects may respond to selection. Their relative impact on response to selection follows from decomposition of the overall population mean into mean direct and associative effects, Math, where averages are taken over m individuals, m denoting population size. Total genetic response per generation, therefore, equals the increase of the DBV Math plus (n − 1) times the increase of the SBV Math per generation,Math(4)

In classical quantitative genetic theory, response to selection equals the change of the (direct) breeding value per generation (Lynch and Walsh 1998). With interaction among individuals, however, response contains a component due to SBV (Equation 4). We therefore generalize the definition of breeding value to incorporate interaction and define a total breeding value, TBVi = DBVi + (n − 1)SBVi. Analogous to classical theory, response equals the per generation change of the TBV, and the TBV replaces the usual breeding value. Note that the associative component of the TBV is expressed not in the focal individual or in its offspring, but in the phenotypes of their associates. In other words, the term Math in Equation 4 represents response to selection of the heritable social environment that individuals experience.

We predict response to selection as the regression coefficient of the TBV on the selection criterion multiplied by the selection differential. The selection differential equals the weighted mean C-value of the selected parents, expressed as a deviation from the population mean (Lynch and Walsh 1998). It can be expressed as Math, where Math denotes the standardized selection differential, usually referred to as “selection intensity,” and Math is the standard deviation of the selection criterion as defined in Equation 3. Response to selection, therefore, equals Math. It is convenient to rewrite Ci as the sum including the individual plus the remaining part of Ci, Math, where the summation now includes Pi. This gives Math. In the first term in brackets, the sum of phenotypes of all group members, Math, can be written as Math, where the summation groups terms that originate from the same individual, instead of those that are expressed in a single phenotype (as in Equation 1). Substitution gives Math, which can be split into a term due to i and a term due to its n − 1 associates, Math. Next, splitting P into A + E, with Cov(E, TBV) = 0, gives Math, in which r denotes the additive genetic relatedness among associates and Math the variance of TBVs. Collecting terms gives the general expression for response to selection, with any degree of multilevel selection, measured by g, and any degree of relatedness among associates, measured by r,Math(5)in which Math, which is the covariance between the phenotype of an individual and its TBV, and Math is the variance of TBVs in the population. Relatedness in Equation 5 refers to the usual population genetic definition of twice the coefficient of coancestry (Lynch and Walsh 1998). The term Math equals the selection gradient (Griffing 1962; de Jong and Bijma 2002), which is the common measure for the strength of selection in natural populations (Lande and Arnold 1983).

Equation 5 shows that for a given, n, g, and r, response to selection depends on the total heritable variation Math and on the covariance between phenotypic trait values and TBVs of individuals Math. Total heritable variation in turn depends on the (co)variances of DBV and SBV (Math, Math, Math) and on group size,Math(6)

The covariance between phenotypic trait values and TBVs of individuals Math represents the heritable relationship between the phenotype of an individual and its impact on the population mean trait value in the next generation and depends on the (co)variances of DBV and SBV, relatedness, and group size,Math(7)

Together, Equations 5–7 reveal the components determining response to multilevel selection with interaction among individuals, which are: (i) the magnitude of heritable direct and associative variances and covariances, (ii) the degree of relatedness, and (iii) the degree of multilevel selection.

Heritable variation:

Heritable interaction among individuals affects both components of response to selection: Math and Math. With respect to Math, the term Math in Equation 6 corresponds to the heritable variation in classical theory. The term Math shows that presence of heritable interaction can substantially increase the total heritable variation, which may explain the rapid responses observed with group selection (Muir 1996; Bijma et al. 2007, accompanying article, this issue). Essentially, the term Math represents the heritable variation present in the social environment. The term Math in Equation 6 shows that a negative genetic covariance between DBV and SBV (Math) reduces the total heritable variation. With negative Math, individuals with positive breeding values for their own phenotype (DBV) have on average negative effects on the phenotypes of their associates (SBV). Thus negative Math may be interpreted as “heritable competition.” (See also Griffing 1976a for a discussion on the interpretation of Math and its effect on response to selection.) Heritable competition, therefore, reduces the total heritable variation and thus the potential of the trait to respond to natural or artificial selection (Equations 5 and 6). In contrast, heritable cooperation (Math) increases the total heritable variation. With respect to Math, Equation 7 shows that competition (Math) may cause Math to be negative, so that response to individual selection (g = 0) will be in the direction opposite to selection, meaning that selection leads to maladaptation.


Relatedness among associates converts the covariance between individual phenotypes and TBVs to a variance (Equation 7). In the absence of relatedness, Equation 7 equals Math, which is a covariance that can take negative values with competition Math, resulting in response to individual selection (g = 0) to be negative (Equation 5). In contrast, with full relatedness, r = 1, Equation 7 is identical to Equation 6, Math, so that response to selection is proportional to the total heritable variation and is therefore always positive, irrespective of competition (Math) and the degree of multilevel selection (g). This result shows that relatedness can overcome the negative consequences of competition. Together, Equations 5, 6, and 7 quantify the contribution of kin selection, i.e., relatedness between interacting individuals, to the evolution of traits in terms of estimable components. [With kin selection we refer to the situation in which associated individuals are genetically related (r > 0). Our model (Equation 1) does not assume that interaction among kin differs from that among non-kin, although the model extends by analogy to incorporate that phenomenon; see the discussion.]

Multilevel selection:

In Equation 5, the first term represents response due to the selection pressure on group performance (g > 0), which is the product of selection pressure on the group, g, and the heritable variation among mean trait values of groups, Math. The second term represents response due to selection pressure on individual performance, which relies on the covariance between individual phenotypes and TBVs (Math). Thus selection at higher levels of organization (g > 0) makes response increasingly dependent on the total heritable variation, instead of on the covariance between phenotypes and TBVs. With competition (Math), Math can take negative values (see above), whereas Math is always positive. Selection among groups, therefore, always yields a positive response, irrespective of relatedness and competition (Math). This is because group selection captures the associative effects of individuals, which are not captured by individual selection in the absence of kin (Equation 7). Selection among groups (g = 1) satisfies Fisher's fundamental theorem of natural selection (Fisher 1958) applied to the group. The term Math is the heritable variation among group means, and Math is the selection gradient, translating truncation selection into fitness (Griffing 1962; de Jong and Bijma 2002). Multilevel selection and relatedness amplify each other: in the extreme case of group selection among clone groups (r = g = 1), response equals Math, which can be very large (Bijma et al. 2007).

In the absence of associative effects Math and multilevel selection, g = 0 so that Ci reduces to Pi and Equation 5 reduces to Math, which equals Math, where Math is heritability, and Math is the selection differential. This result is the well known “breeder's equation” (Lynch and Walsh 1998). Thus, as expected, the classical expression for response to selection is a limiting case of Equation 5.

Special cases:

Griffing (1967, 1976a,b) considered a number of special cases. Results for those cases follow directly from Equation 5 and are briefly discussed here. With individual selection among unrelated individuals, g = r = 0, Equation 5 reduces to Math. The first term within the brackets represents the direct response, which is positive always. The second term represents the correlated response in the associative effect, which is either positive or negative, depending on the sign of Math. With competition (Math), the negative correlated response in the associative effect may exceed the positive direct response, causing negative total response, which has been observed in real populations (Muir 1996, 2003, 2005; Muir and Craig 1998). Response to individual selection with groups consisting of clones, g = 0 and r = 1, equals Math, which is always positive. Response to selection among groups composed of unrelated individuals, g = 1 and r = 0, equals Math, which is always positive. Response to selection among groups composed of clones r = g = 1, equals Math, showing that relatedness has the potential to increase response to group selection by a factor of n, which is substantial.

Generalization to multiple levels:

The approach taken here extends by analogy to more than two levels of organization. (See also, e.g., Wade 1982 for a population genetic model of three levels of selection.) Consider, for example, the three levels of organization that would exist in a population consisting of k metagroups, each consisting of m groups, which in turn consist of n individuals each, so that population size equals kmn. With interaction at those three levels, phenotypic trait values may be described as the sum of three unobserved components, a direct effect due to the focal individual, Math, an associative effect due to its n − 1 group members, Math, and a second associative effect due to its (m − 1)n metagroup members, Math, Math. Hence, each individual would carry three unobserved effects, treated as potentially heritable traits: a direct effect affecting its own phenotype, an associative effect affecting its group members, and a second associative effect affecting its metagroup members. Selection of individuals, either artificially or naturally, may depend on all three levels, so that the selection criterion may be modeled as Math, where g′ represents the impact of the metagroup members on fitness of individual i. For example, g = g′ = 0 would represent selection on an individual trait value across the entire population irrespective of the group and metagroup; g = 1 and g′ = 0, selection between groups irrespective of the metagroup; and g = g′ = 1, selection between metagroups. The TBV, denoting the heritable impact of an individual on the population mean, becomes Math, and response to selection isMath(8)(see appendix for derivation), in which r′ is relatedness between an individual and its metagroup members. Equation 8 is a direct analogy of Equation 5. Selection at the highest level of association (g′) captures the total heritable variation, Math, whereas selection at lower levels acts on the covariances between TBVs and phenotypes at those levels: Math for the group (including i) and Math for the individual. The elements of the term for response of selection on the highest level, Math, are the sum of the relatedness coefficients of all mn individuals with the focal individual; the 1 is self-relatedness, the (n − 1)r is the summed relatedness with the (n − 1) group members, and the (m − 1)nr′ is the summed relatedness with the (m − 1)n metagroup members. Hence, when relatedness varies among individuals, this term may more conveniently be written as Math, in which ri,j is relatedness between individual i and associate j. The weights g′, (gg′), and (1 − g) in Equation 8 sum to one and can therefore be interpreted as a proportioning of the selection pressure. Equation 8 extends by analogy to any number of levels of selection, illustrating the generality of our approach.

Structured populations:

Here we extend the above results to structured populations with inbreeding and nonrandom mating. We consider a population structured into many demes, with limited migration of individuals among demes, random mating within demes, but potentially nonrandom association within demes (see also Agrawal et al. 2001). For example, a deme may be a larger subpopulation, within which association is among family members. Such a population has higher relatedness within demes than between demes, causing a redistribution of the additive genetic variance within and between demes, which follows from Wright's F-statistics (Wright 1951, 1965) and yields a total additive genetic variance ofMath(9)(see also Table 15.1 in Falconer and Mackay 1996), in which Math is the average inbreeding coefficient in the population expressed relatively to a hypothetical panmictic population with the same allele frequencies, and Math is the additive genetic variance within this hypothetical population. With random mating within deme, Math equals the average coefficient of kinship within deme, Math.

Total relatedness between associated individuals, which equals twice their coefficient of total kinship, r = 2f, depends on the degree of population subdivision among demes, measured by Math, and on the degree of nonrandom association within deme, measured by kinship between associates expressed relative to the deme, Math,Math(10)(Wright 1951, 1965). For example, when association within demes is among full sibs, so that Math, in a population in which Math, kinship among associates equals Math, so that total relatedness among associates, i.e., relatedness expressed relative to the hypothetical panmictic population, equals r = 2f = 1.1. The interpretation is that the covariance between breeding values of associates equals Math, i.e., 110% of the additive genetic variance with full panmixia.

The use of Equations 9 and 10 allows extension of our results (Equations 5–7) to structured populations. The result can be expressed either in terms of genetic parameters in the hypothetical panmictic population Math or in terms of the genetic parameters in the current population Math. The first parameterization follows from substituting Math, Math, and Math in Equations 5–7 by the corresponding versions of Equation 9, by substituting relatedness between associates by Math and by substituting self-relatedness by Math (Falconer and Mackay 1996), givingMath(11)in whichMath(12)

This parameterization is particularly useful when interest is in the impacts of population subdivision, because it directly relates the degree of subdivision (Math) to the response to selection. (Agrawal et al. 2001 assumed that the mating structure and the association structure coincided, meaning that association was assumed to be at random within deme, so that fw = 0 and Math.)

The second parameterization follows from substituting Math into Equations 11 and 12. The resulting expressions for response to selection are identical to Equations 5–7 on the condition that (i) Math,Math, and Math in Equations 5–7 refer to the current subdivided population, e.g., Math, etc., and (ii) relatedness is defined asMath(13)

This definition of relatedness equals both the regression and the correlation coefficient between breeding values of associates in the current population and is identical to Hamilton's regression definition of relatedness (Hamilton 1972). Therefore, depending on the objective of the research, one can either use a parameterization in terms of variances and covariances in a hypothetical panmictic population and account for population structure by defining relatedness relative to this hypothetical population, as in Equations 11 and 12 and Agrawal et al. (2001), or one can use genetic parameters of the current population, define relatedness as in Equation 13, and use Equations 5–7.

In conclusion, therefore, our basic results for response to selection (Equations 5–8) are valid for general population structures on the condition that (i) the genetic parameters refer to the current population and (ii) relatedness is defined as the correlation between breeding values of individuals in the current population (Equation 13). This result also implies that population subdivision does not fundamentally alter the outcome of multilevel selection. The effect of population subdivision on the outcome of multilevel selection arises entirely via the effect of population structure on heritable (co)variances of traits and on relatedness among individuals, an effect that is well described in classical theory (Wright 1951, 1965).


This work provides the basic quantitative genetic expressions for response to either natural or artificial multilevel selection for a single trait, with interactions among potentially related individuals. Equation 5 presents a generalized outcome from any degree of multilevel selection and shows how group, individual, and kin selection combine into response. Results provides whole-organism biologists with a mechanistic understanding of multilevel selection, expressed in accessible terms that can be estimated from real populations (Muir 2005; Bijma et al. 2007), such as means and variances of traits, and from which hypotheses can be tested.

Following quantitative genetic theory, we distinguish between (i) the heritable components, (ii) relatedness, and (iii) the selection process. Equation 2 identifies DBV and SBV as the heritable components affecting the observed trait, which combine into a total breeding value. Analogous to classical theory, genetic response to selection equals the per generation change of the total breeding value, which depends on both relatedness and the multilevel selection process. The multilevel selection process determines the selection pressures on direct and associative effects, whereas relatedness alters the genetic (co)variances translating the phenotypic selection differential into genetic response (Equations 6 and 7).

Results are expressed in common quantitative genetic parameters, such as variances and covariances of breeding values, and reduce to classical expressions for response to selection in the absence of associative effects and multilevel selection. This shows that our results are a generalization of the basic principles of quantitative genetics. In the accompanying article (Bijma et al. 2007, this issue), we provide the statistical methodology to estimate the genetic variances and covariances required to quantify the expressions for response to selection in general populations with any degree of relatedness between associates, such as natural populations. That article also provides an example of calculation and predictions of responses for a real population. Application of the theory presented here requires recording phenotypes for the trait of interest and identifying the individuals that are associated or measuring physical distance between individuals when their positions are fixed. Results will provide testable predictions of response to multilevel selection, which are urgently needed (West et al. 2001; Fehr and Fischbacher 2003). Detailed implications of this work for kin selection theory will be treated in a future article.

Theory presented here may clarify empirical results that are inexplicable using classical quantitative genetic theory. On the one hand, experimental studies applying group selection have commonly yielded responses substantially larger than predicted from classical theory, both in animals and in plants (Muir 1996; Goodnight and Stevens 1997; Griffin et al. 2004; Goodnight 2005). On the other hand, laboratory experiments and selection programs in animal and plant breeding applying individual selection have often failed to produce response, despite abundant heritable variation, and have occasionally yielded response in the opposite direction (Goodnight 1985; Teichertcoddington and Smitherman 1988; Vangen 1993; Kruuk et al. 2001; Muir 2005). This has occurred primarily in situations with strong behavioral interactions, such as cannibalism in Tribolium (Wade 1977) and poultry (Muir 1996), and in cases where associates compete for limited resources in confined rearing, such as with growth under restricted feeding in poultry, mice, fish, or pigs. These circumstances are expected to cause a negative covariance between direct and associative effects, explaining the lack or reversal of response (Equations 5–7). In contrast, for traits like milk yield in grazing animals, where social interactions are less intense, animals are not confined, or management practices are aimed to minimize competition for space or feed, individual selection has produced extraordinary responses (Boldman and Vanvleck 1984).

Our results show that interaction among individuals involves additional heritable components (SBV), which may increase the heritable variation and the potential of populations to respond to selection (Equations 5 and 6). This extra heritable variation does not surface in conventional data analyses (Bijma et al. 2007). In the accompanying article (Bijma et al. 2007, this issue) we show that the total heritable variation in survival days in a population of laying hens is threefold of the amount estimated using classical methods. In other words, two-thirds of the heritable variation in that population is hidden in classical analyses. In the absence of kin, group selection captures this extra variation, which may explain the large responses observed. Individual selection, in contrast, may yield negative correlated response in the associate effects, which explains lack of, or even negative, response (Equations 5 and 7).

Kin and group selection are sometimes interpreted as alternative, but equivalent, formulations of a single issue (Queller 1992, 2004), whereas our approach distinguishes between relatedness and selection. For example, Queller (2004) interpreted low (high) relatedness among group members as presence (absence) of within-group selection. Although that approach may be helpful in specific cases, we believe that distinguishing between relatedness and selection yields a more general and intuitively appealing approach. For example, when groups are either selected entirely or rejected entirely, it would be odd to interpret absence of relatedness among group members as within-group selection. Such a situation would more naturally be modeled as “selection between groups composed of unrelated individuals.” Our results show that response to selection is a function of both relatedness (r) and the strength of group selection (g). The dependency of response to selection on the product of r and g indicates that combined kin and group selection amplifies the consequences of interactions among individuals beyond their separate effects (Equation 5). Furthermore, distinguishing between relatedness and selection is consistent with quantitative genetic theory, allowing the application of powerful tools for data analyses, such as animal models (Kruuk 2004). Note that the distinction between relatedness and selection in our equations does not imply a fundamentally different kind of evolution between kin and group selection. The group affects merely fitness of the replicator, i.e., the gene or breeding value (Dugatkin and Reeve 1994; Keller 1999).

Relationship to previous work:

Goodnight (2005) provided a quantitative genetic formulation for response to multilevel selection. In that approach, individual phenotype and mean group phenotype were treated as two distinct heritable traits, even though one is simply the group mean of the other, which required the introduction of a parameter referred to as “group heritability.” However, as acknowledged by Goodnight, it is unclear how this parameter should be defined or estimated. In contrast, our approach separates phenotypic values into a direct component due to the individual carrying the phenotype and an associative component due to individuals it associates with, and both direct and associative effects are treated as heritable traits of the individuals. Goodnight (2005; Goodnight and Stevens 1997), furthermore, argued that interaction among individuals relies on nonadditive genetic variance. In quantitative genetic theory of complex traits, however, “additive” means merely “heritable,” i.e., passed to the offspring (Lynch and Walsh 1998). Our model (Equation 2), therefore, states merely that the effect of an individual on its associates is partly inherited by its offspring. This does not rely on nonadditive genetic variance, but is a straightforward generalization of the maternal-effects model that has a successful history in the field of livestock genetic improvement (Willham 1963).

Agrawal et al. (2001) presented a quantitative genetic model that is closely related to our Equation 1. They distinguished between an “effector trait” and an “interacting trait.” The effector trait is not affected by interaction, but it affects the interacting-trait values of all individuals in the same deme. Phenotypic values for the effector trait are given by Math, in which a1 denotes the breeding value, whereas phenotypic values for the interacting trait are given by Math + nonlinear terms, so that, after substitution of z1, z2 is given by Math + nonlinear terms (Agrawal et al. 2001). (We do not consider the nonlinear terms here.) For large group sizes, so that Math, Math corresponds to Math, and Math corresponds to Math. Fitness depended on both traits and was given by Math.

The principal difference between our model and that of Agrawal et al. (2001) is as follows. Their model assumes the existence of a single observable effector trait that is the sole cause of the social interactions. (The z1 needs to be observable to allow estimation of Math, Math, and Math.) In contrast, our model assumes that observed values for the interacting trait are the sum of two underlying effects, namely direct and associative effects, which may or may not be observable (Equation 1). In our model, the associative effect may be the net result of numerous other phenotypic traits, some of which may be observable and others may not. Similar to the well-known maternal-effects model (Willham 1963), the statistical analysis presented in Bijma et al. (2007) captures the total heritable variation due to interactions among individuals, irrespective of the number of causal components and whether or not those can be observed, and it provides estimated breeding values for the full associative effects of individuals. In the special case in which a single observable effector trait exists, the approach of Agrawal et al. (2001) provides more detailed information on the nature of the interaction, because their model is a more functional description of the interactions, whereas our model is a more statistical description. However, even in those cases, it remains unclear whether and how the required model components, such as Math and the genetic (co)variances of a1 and a2, can be estimated from experimental or field data.

Implications for agriculture:

Animal and plant breeders have some concern for interactions among individuals, but have lacked direction as to the best method to address these issues and are unsure as to the magnitude of the problem in their species, or even whether it exists at all. They are, however, well equipped with statistical tools and experimental populations. Theory and methodology provided here and elsewhere (Muir 2005; Bijma et al. 2007) will enable them to accurately estimate the genetic variance in SBV in large-scale experiments, which will provide insight into the genetic relevance of interaction among individuals for the first time. Knowledge of these parameters will allow formulation of optimal breeding programs to maximize total genetic improvement. This work may, therefore, contribute to sustained improvement of animal welfare (Muir 2003) and to global food security since it enables maximization of overall yield on the population level in animal, plant, and tree breeding, which in the past has been hindered by the inability to account for competitive effects among individuals in the selection decisions (Denison et al. 2003; Muir 2003).

Implications for evolutionary biology:

In the following, we discuss (i) the implications of this work for the impacts of group size on evolution and (ii) extensions to cases with asymmetric interaction.

Group augmentation:

In natural populations, group size often contributes to evolutionary success, a process referred to as group augmentation (Clutton-Brock 2002). Equation 6 shows the relationship between the total heritable variation and group size. At first glance, Equation 6 may suggest that increased group size increases the heritable variation. However, the relationship between the total heritable variation and group size is complicated by a potential relationship between the variance of SBV and group size. In other words, associative effects, which are expressed per individual receiving the effect, may become smaller in larger groups because they are distributed over a larger number of associates. The relationship between the variance of SBV and group size will depend critically on the nature of the trait. Consider, for example, behaviors of food sharing vs. alarm calling. When an individual distributes a fixed total amount of food over its n − 1 group members, average benefit for a single group member is proportional to 1/(n − 1). Consequently, variance of SBVs is proportional to 1/(n − 1)2 and Math is independent of group size (Equation 6). Thus, for food sharing, group augmentation may contribute little to the heritable variation. In contrast to food sharing, individual benefits of alarm calling will depend hardly on group size, because each group member receives the warning call, irrespective of group size. Consequently, variance of SBVs will be roughly independent of group size and, within certain limits, the total heritable variation may increase considerably with group size. Thus the relationship between group size and the variance of SBVs will be a key factor for the evolutionary impact of group augmentation. The statistical methodology presented elsewhere (Garant and Kruuk 2005; Bijma et al. 2007) in principle enables estimating that relationship, but collecting sufficient data from natural populations will be a considerable task. The relationship between the variance of associative effects and group size, and its dependency on the nature of the interaction, can be interpreted as a form of genotype-by-environment interaction. Such genotype-by-environment interaction may cause the outcome of multilevel selection to depend strongly on group size and the type of interaction.

Asymmetric interactions:

This section discusses extensions of the model to cases other than symmetric interaction, namely (i) interactions with kin vs. non-kin, (ii) interaction between sexes, and (iii) interaction between species.

It is observed frequently that interaction with kin differs from interaction with non-kin (West et al. 2002). In that case, group members of a focal individual may be categorized according to their relationship with the focal individual, so that Math, where ji, j = 1, l refers to kin and j = l + 1, n − 1 refers to non-kin. In this case there are three unobserved quantitative traits, the direct effect, the associative effect on kin, and the associative effect on non-kin. The genetic correlation between associative effects on kin and on non-kin measures the genetic dependency of behavior toward kin vs. behavior toward non-kin. For example, a negative genetic correlation indicates that individuals with higher associative effects on kin on average have lower associative effects on non-kin, so that there is a conflict between cooperation with kin vs. cooperation with non-kin. As for the above cases, response to selection follows from regression of breeding values on the selection criterion.

In many cases, interaction among individuals depends on their sex; male lions, for example, interact differently with other males than with females. In such cases, direct effects may be categorized according to the sex of the focal individual and associative effects according to both the sex of the focal individual and the sex of the recipient. For example, the observed phenotypic trait value for a male is given by Math, in which subscript m denotes males, f is females, fm is associative effects of females expressed in males, mm is associative effects of males expressed in males, j = 1, l refers to female associates, and j = l + 1, n − 1 refers to male associates. In this case there are six traits: for each sex there are the direct effect, the associative effect on the same sex, and the associative effect on the opposite sex.

Classical models of interaction between species (May 1972) predict that as the number of species in an ecosystem increases, the stability of the system decreases. This outcome is debated, but it is clear that competition between and within species is fundamental (McCann 2000; Ebenman and Jonsson 2005). What is missing from this debate is a critical genetic factor. May (1972) and later work (May 1973, 1975) assumed that species interactions are constant, when in fact they evolve, both in response to environmental challenges and in response to competition with individuals within and between species (Keller 1999). Thus the environments to which species adapt are constantly evolving, which can either increase or decrease stability of species interactions and ecosystems.

A true understanding of the implications of our model in the broader context of multispecies interactions may require a between-species formulation of the model. Nevertheless, the approach taken here may be extended to interaction between individuals of different species. In that case, individuals of the one species are affected by their own direct effect and by associative effects of individuals of the other species. Each species may possess direct and associative effects. For example, with interaction between two species, X and Y, trait values of individuals of species X may be given by Math and for species Y by Math, where j refers to individuals of species Y, l to individuals of species X, n is the number of Y individuals associating with an X individual, and m is the number of X individuals associating with a Y individual. In this case there are four genetically distinct traits, a direct and an associative effect for each species. Selection response observed in X will be the sum of selection response in direct effects of X plus (n − 1) times the response in associative effects of Y, and vice versa. Thus evolution of a species depends on response in its own direct effect and on response in the associative effect of the species it interacts with.

All models, including ours, are simplified representations of reality and do not capture all specifics of particular situations. For example, our model ignores nonadditive effects, such as dominance or epistasis, non-Mendelian types of inheritance, such as imprinting and meiotic drive, drift, selection for multiple traits, dynamic populations, and ecological factors. Lande and coworkers (Lande 1982; Lande and Arnold 1983) have extensively examined consequences of natural selection for correlated traits. Extension of our models to include multiple traits is straightforward and needed for a general understanding of impacts of multilevel selection on the multitrait phenotype. Goodnight and coworkers (Wade and Goodnight 1991; Goodnight 1995) have investigated the ability of multilevel selection together with random drift to capture nonadditive genetic variation, which is important but beyond the scope of this article.

In the short term, additive genetic (co)variances determine response to selection. We therefore feel our model will help clarify and quantify the processes of evolution where social interactions and multilevel selection exist. The strength of the model is that it provides a basis for testable predictions. Where those predictions fail, the model will need to be extended to add precision for specific cases. However, just as the breeder's equation has proven to be very predictive of responses to artificial selection and evolutionary processes in the short term, hopefully our model is equally robust and will result in a better understanding of the evolutionary process as well as in methods to greatly improve agricultural production to further secure the world's food supply.


With interaction at three levels of organization, phenotypic values of individuals are given by Math, TBVs are given by Math, and the selection criterion by Math, in which i denotes the focal individual, j one of its n − 1 group members, and k one of its (m − 1)n metagroup members (see main text). In the following, subscript j is used to refer to group members, either including or excluding i, and subscript k is used to refer to metagroup members, either including or excluding the group and i. The following derivation is an analogy of the derivation of Equation 5. Response to selection equals the regression coefficient of the TBV on the selection criterion multiplied by the selection differential, Math. It is convenient to rewrite Ci as a sum including all individuals at each level and subsequently adding the remaining terms of Ci, Math, where the first summation is over all mn individuals in a metagroup, the second summation is over all n individuals in a group, and the last term refers to the focal individual. This gives Math. The second and third terms of this expression correspond directly to the second and third terms of Equation 8. What remains is the first term. In the first term, the element Math represents the sum of all phenotypic values in the metagroup. This term can be rewritten by grouping direct, associative, and secondary associative effects according to the individual from which they originate (instead of grouping them according to the phenotype in which they are expressed). Each individual, k, expresses its direct effect once in its own phenotype, its associative effect (n − 1) times in its group members, and its secondary associative effect (m − 1)n times in its metagroup members. Therefore, Math, where the brackets group terms that originate from the same individual. Next, the sum of phenotypic values of all mn metagroup members, Math, can be split into (m − 1)n terms due to metagroup members of the focal individual, n − 1 terms due to group members of the focal individual, and a single term due to the focal individual itself. Substitution gives Math. The first term refers to the (m − 1)n metagroup members, denoted by k, the second term to the n group members, denoted by j, and the last term to the focal individual i. Next, splitting P into A + E, with Cov(E, TBV) = 0, gives Math, in which r′ denotes additive genetic relatedness between the focal individual and its metagroup members, k, and r additive genetic relatedness between the focal individual with its group members, j. Collecting terms gives Equation 8.


We thank Charles Goodnight for helpful comments.


  • 1 These authors contributed equally to this work.

  • Communicating editor: J. B. Walsh

  • Received June 27, 2006.
  • Accepted October 26, 2006.


View Abstract