## Abstract

Threshold models are useful for understanding the evolution of dimorphic traits with polygenic bases. Selection for threshold characters on individuals is expected to be frequency dependent because of the peculiar way that selection views underlying genetic and environmental factors. Selection among individuals is inefficient because individual phenotypes fall into only two discrete categories that map imperfectly to the underlying genes. Incidence, however, can be continuously distributed among groups, making among-group selection relatively more efficient. Differently put, the group-mean phenotype can be a better predictor of an individual's genotype than that individual's own phenotype. Because evolution in group-structured populations is governed by the balance of selection within and between groups, we can expect threshold traits to evolve in fundamentally different ways when group mean fitness is a function of morph frequency. We extend the theory of selection on threshold traits to include group selection using contextual analysis. For the simple case of linear group-fitness functions, we show that the group-level component of selection, like the individual-level component, is frequency dependent. However, the conditions that determine which component dominates when levels of selection are in conflict (as described by Hamilton's rule) are not frequency dependent. Thus, enhanced group selection is not an inherent property of threshold characters. Nevertheless, we show that predicting the effects of multiple levels of selection on dimorphic traits requires special considerations of the threshold model.

MANY interesting phenotypes have a polygenic basis but are expressed as discrete character states. Examples include wing dimorphism in insects, presence or absence of enlarged horns or other structures in male insects, male size and mating behavior dimorphism in various animal taxa, life-cycle dimorphism in salamanders, trophic dimorphism in salamanders and fish, and reproductive caste dimorphism in eusocial animals (reviewed by Roff 1996). Threshold models provide a quantitative genetic framework for studying the evolution of such dimorphic phenotypes. These models assume that polymorphisms are caused by variation in an unobservable but normally distributed phenotype termed “liability” (Wright 1934; Lush *et al*. 1948; Dempster and Lerner 1950; Falconer 1965a). Individuals with liability above a threshold value express one phenotypic character state (induced) and those with liability below the threshold express the alternate state (uninduced). Quantitative genetic parameters, such as narrow-sense heritability and genetic correlations, can be estimated on this liability scale by considering the joint patterns of induction among relatives (Mercer and Hill 1984; Sorensen *et al*. 1995).

Predictive evolutionary theory requires that we understand selection and inheritance (Fisher 1930; Robertson 1966; Price 1970). If we are interested in the evolution of a threshold trait, then selection must be considered on the liability scale. Classical theory predicts that mass selection on liability should be frequency dependent (Dempster and Lerner 1950; Crow and Kimura 1970; Falconer and Mackay 1996) because the threshold function shields part of the liability variation (the within-morph component) from the purifying effects of selection; the size of the cryptic fraction depends upon the proportion of the population that is induced (the population incidence). However, when populations are partitioned into groups (*e.g*., demes or family units), random genetic drift will lead to variations among groups in mean liability and, hence, mean incidence. Group-mean liability can vary continuously and the function that maps fitness to group-mean liability is free to take any shape. In this way, selection can discriminate better between groups than between individuals. The threshold function causes mass selection to be inefficient, leading some to recommend that breeders apply family-level selection to more effectively change the population incidence (Dempster and Lerner 1950; Crow and Kimura 1970; Mikami and Fredeen 1979; Falconer and Mackay 1996).

Current models of selection on threshold traits consider only the genes that map directly from individuals' genotypes to phenotypes (*i.e*., direct genetic effects). Consequently, “family-level selection” in the classical animal breeding literature implies artificial selection on the mean phenotype of family members that results only from direct effects. In reality, the genotypes of social partners (*e.g*., mothers, siblings, or coresidents) often affect the phenotypes of individuals through indirect genetic effects (Cheverud and Moore 1994; Moore *et al*. 1997; Wolf and Brodie 1998; Wolf *et al*. 1998; Agrawal *et al*. 2001), which are also known as associative effects (Griffing 1967, 1968, 1976, 1981; Muir 2005; Bijma *et al*. 2007a). Maternal effects are the best studied and perhaps most widespread type of indirect effect (Falconer 1965b; Willham 1972, 1980; Cheverud 1984) and have been shown to affect the expression of threshold traits in several species, including diapause in cricket eggs (Huestis and Marshall 2006), sex in reptiles with temperature-dependent sex determination (Freedberg and Wade 2001), the presence or absence of horns in male dung beetles (Moczek 1998; see Hunt and Simmons 2002), and reproductive caste in some species of ants (Linksvayer 2006; Schwander *et al*. 2008). Indirect effects arising from both genetic and environmental effects can cause differences in group-mean fitness that lead to group-level selection that may work in concert or in conflict with individual-level selection (Moore *et al*. 1997; Wade 1998). Family-level selection in the social evolution literature implies selection among families that have been kept intact so that social interactions contribute to the phenotypes expressed by the social members.

The evolution of traits that are affected by direct and indirect social factors can be understood more fully by partitioning selection into individual and group-level components, using the regression-based method of contextual analysis (Goodnight *et al*. 1992; Heisler and Damuth 1987; Okasha 2004; Goodnight 2005). This approach has been used to generalize Hamilton's rule (Hamilton 1964), a definition of the conditions necessary for the spread of an “altruistic” trait that has opposing group and individual effects (Hamilton 1970; Wade 1980). Despite the special significance given to dichotomous traits in discussion of the evolution of altruism, such as the evolution of discrete queen and worker reproductive castes in eusocial animals (Linksvayer and Wade 2005), selection on threshold traits has not yet been studied using contextual analysis. Here we use this approach to explore how multilevel selection operates on a threshold trait with an underlying genetic model that includes indirect genetic effects.

## THE LIABILITY MODEL

Liability is a phenotype composed of contributions from genetic and environmental effects. Both of these can be further decomposed into direct or indirect effects. Direct effects are genetic (*A*_{D}) or environmental effects (*E*_{D}) that are intrinsic to individuals and affect their phenotype, regardless of the individuals' social interactions. Indirect effects act upon focal individuals, but are generated by the accumulation of genetic (*A*_{S}) and environmental effects (*E*_{S}) experienced by social partners and transferred to the focal individual (Moore *et al*. 1997; Wolf *et al*. 1998). Here we apply the phenotypic model of direct and indirect effects described by Bijma *et al*. (2007a) to a liability phenotype. The liability *z* of a focal individual *i* that is affected by interactions with a group of *n* − 1 social partners, each with an indirect effect on individual *i*, is(1)where μ is the population mean liability and *j* indicates a social partner that interacts with focal individual *i*. This model is general and applies to populations composed of groups of socially interacting individuals, *e.g*., demes of interacting individuals or families with maternal effects.

The maternal-effect case is special because all individuals within a family experience the same indirect (maternal) effect; *i.e*., the summation terms in Equation 1 are simply the maternal genetic and environmental effect, which can be written as *A*_{m} *+ E*_{m} (Willham 1972; Cheverud 1984). We assume that the distributions of each of the effects in Equation 1 are Gaussian and i.i.d. across individuals and groups. Direct and indirect genetic effects can covary in our model and, in fact, evidence of direct–indirect genetic correlations is frequently observed (Cheverud 1984; Cheverud and Moore 1994; Linksvayer 2006). The distribution of liability is standardized such that σ_{z} = 1. The variance among groups is and the within-group variance is . The liability phenotype of any individual *i* translates into an incidence phenotype by a simple step function,An individual is said to be induced if and only if its liability phenotype exceeds zero. Group and population mean liability can be inferred from the inverse cumulative normal distribution and the appropriate incidence and liability variation for each,(2a)(2b)where is the frequency of induction in the population (the population incidence) and is the incidence of a group with mean liability . There is a one-to-one map of to .

Selection may act upon the phenotypes of both individuals and groups. Selection components are found using the covariances between relative fitness and the phenotype at the level of the individual and the group: and . Because these covariances are not independent of one another (Heisler and Damuth 1987; Frank 1997; Okasha 2004), contextual analysis is used to fully disentangle the levels of selection, thereby decomposing total selection into components of individual and group-level selection. We apply this approach to explore multilevel liability selection in the next section.

## RESULTS

Here we explore the ramifications of the threshold model when (1) individual phenotypes determine fitness, (2) group phenotypes determine fitness, and (3) individuals and group phenotypes determine fitness. When fitness depends upon both individual and group-level phenotypes, total selection can be partitioned into individual and group-level components. Because we find that individual-level and group-level selection are both frequency dependent, we investigate if the *relative* efficiency of these levels of selection may change with the mean incidence.

#### Fitness and individual phenotypes:

When fitness depends only upon the phenotype of the individual, selection is simply the covariance between relative fitness and individual liability. This is the “hard-selection” model (Goodnight *et al*. 1992). The observable phenotype of the uninduced state is 0, which occurs in the population with frequency , and the induced state is 1, which occurs with frequency . From the perspective of selection, individuals of the same state have the same liability phenotype, which is the expectation of the appropriately truncated normal distribution (Falconer and Mackay 1996). We define selection by first expanding the covariance between relative fitness and individual liability,(3)The expectations follow from Barr and Sherrill (1999),(4a)(4b)We substitute Equations 4a and 4b into Equation 3 and simplify to find selection in the absence of group selection,(5)where , the height of the unit normal curve at the population's liability mean. Individual liability selection in the absence of group selection is frequency dependent, approaching a maximum magnitude of ∼0.8 as and a minimum of zero as . We note that because there are only two possible relative fitness values, the slope of the regression of relative fitness on individual incidence is always linear and proportional to the difference in relative fitness between morphs. Equation 5 shows that evolution, measured as change in mean liability, stops when the fitnesses of the two morphs are equal, making .

#### Fitness and group phenotypes:

Here we consider the case when fitness depends only upon the mean phenotype of the group; selection is simply the covariance between relative fitness and group-mean liability. This is the “group-selection” model (Goodnight *et al*. 1992). Because group means are continuously distributed (unlike the dichotomous liability values of individuals), we find the covariance by integrating over all groups. We infer group liability means using Equation 2b because liability cannot be observed directly. Given some function that relates relative fitness to the observed group incidence , the covariance between relative fitness and group liability follows from the definition of a covariance and , the probability density function (pdf) of group incidence,(6)

The probit function that transforms incidence to mean liability is nonlinear. We expect that this will cause the covariance in Equation 6 to depend upon the population incidence. We investigate how the threshold model causes frequency-dependent selection by assuming a linear relationship between group fitness and group incidence. By the chain rule, the instantaneous slope of relative fitness on group-mean liability is(7)We find by finding the mean slope taken over all values of ,(8)where . Because we have assumed that relative group fitness changes linearly with incidence,(9)This ensures that for all group incidences and(10)Group incidence follows from the normal cumulative distribution function (cdf) with variance . Because the slope of the cdf of a distribution is its pdf, the slope of group incidence on group-mean liability is the ordinate of the normal curve at with parameters μ = 0 and ,(11)Substituting Equation 11 and the definition of given by Equation 8 into Equation 10 and rearranging, we find(12)where . The antiderivative of is(13)where . We solve for by evaluating the integral . Noting that as and as and the remainder of is insensitive to changes in , we find that(14)This covariance depends upon the population incidence by virtue of its proportional relationship to , just as the covariance between relative fitness and individual liability (Equation 5). We can now examine the joint effects of individual and group-level liability selection now that we have expressions that describe the covariance between relative fitness and individual liability (Equation 5) and group liability (Equation 14).

#### Partitioning components of liability selection:

When fitness depends upon the phenotype of the individual and the group, we can define individual- and group-level liability selection using partial covariances. In the general case (*i.e*., for both continuous and threshold traits), selection on the individual-level phenotype is the partial covariance between relative fitness and individual phenotype holding the group phenotype constant, . Selection on the group-level phenotype is the partial covariance between relative fitness and group phenotype holding the individual phenotype constant, . We can restate these components of selection in terms of variances and covariances:(15)(16)(Li 1975; Goodnight *et al*. 1992). For liability selection, Equation 16 simplifies slightly because . These equations help to clarify the contributions of direct and indirect effects to group-level selection. Both direct and indirect effects can contribute toward the variation among groups. If the groups are made up of related individuals, there are by definition genetic differences between groups that can be direct and indirect in nature. We can see how this variation increases the strength of group selection by restating the right-hand side of Equation 16 in terms of regression coefficients: . Indirect effects contribute mainly toward the among-group component of liability variance because members of the social group experience similar indirect effects [see Equation 1; note that when all group members experience the *same* indirect effect, as with maternal effects, indirect effects contribute *only* to the among-group components (Wade 1998)]. In contrast, direct effects generally contribute to variation within groups (*e.g*., direct genetic effects will be variable within groups unless the individuals are clones). This limits the relative amount of among-group phenotypic variance generated by direct effects. As a result, indirect effects contribute more to group selection than direct effects.

#### Conflict between levels of liability selection:

Models of multilevel selection are frequently applied to situations in which there is conflict between levels of selection; the evolution of altruism (Hamilton 1964, 1970) is a notable example. Hamilton's rule defines the conditions necessary for selection to favor the spread of an altruistic phenotype as , where *c* is the cost to the individual if it expresses the altruistic character, *b* is the benefit to the group conveyed by the altruistic individual, and *r* is the “relatedness” between individuals. Whereas relatedness was originally intended as a measure of genetic kinship, it is now more broadly recognized to be the fraction of the total phenotypic variance that is sequestered between groups (Frank 1997; Goodnight 2005). The absolute value of Equation 15 is the cost term *c* and the absolute value of Equation 16 is *br* in the parlance of contextual analysis. We apply the more general form of Hamilton's rule to the two components of selection given in Equations 15 and 16 (Goodnight *et al*. 1992; Goodnight 2005):(17)That is, selection will favor the spread of an altruistic phenotype when group selection is stronger than individual selection. We find the ratio of slopes of relative fitness on group to individual phenotype by substituting Equations 15 and 16 into Equation 17 and rearranging,(18)The threshold function generates frequency-dependent liability selection on individuals in the absence of group selection (Equation 5) and frequency-dependent selection on groups in the absence of individual-level selection (Equation 14). We want to know if the threshold function causes Hamilton's rule to become frequency dependent, too. We apply this general rule to the case of a threshold trait where the relationship between group fitness and group incidence is linear. We substitute Equations 5 and 14 into Equation 18 and rearrange and find the rule in terms of the ratio of slopes on the incidence scale,(19)which is the same as for conventional continuous traits (compare to Equation 18) because . Thus, the conditions of Hamilton's rule for threshold traits do not depend upon population incidence. Furthermore, the rule can be evaluated using the fitness regression slopes on either the incidence scale (Equation 19) or the liability scale (Equation 18). This result requires the assumption that group-mean fitness is a linear function of group incidence. We note that Hamilton's model and most formal expressions for it (*e.g*., Wade 1980; Gardner *et al*. 2007) make exactly the same assumption of a linear group fitness function—for example, substituting two nonaltruists with altruists incrementally raises group mean fitness by twice the effect of a single substitution.

## DISCUSSION

Classical quantitative genetic theory predicts that selection for threshold traits has two interesting properties. First, mass selection on threshold characters should be frequency dependent. Second, family-level selection can be more effective than individual-level selection when the less-fit morph is rare. If these predictions hold when selection acts at both levels simultaneously, then they suggest profound implications for the evolution of threshold traits in structured populations. For example, the threshold model could cause the conflict between levels of selection to have a frequency-dependent outcome. Consider the evolution of reproductive altruism in eusocial animals where selection for fertility at the level of the individual acts in opposition to selection at the level of the group. When the frequency of altruism is very low, individual-level selection against sterility might be intense and total selection may favor a decrease in the likelihood of individual sterility. As fertility increases, however, individual-level selection would relax, thereby intensifying the relative importance of the family-level selection for increased sterility. Stable equilibria would evolve to an intermediate population incidence.

We explored this possibility by developing a formal multilevel selection model for threshold traits. Using contextual analysis, we showed how threshold selection can be decomposed into individual and group-level components of liability selection. We confirmed that both components of selection are frequency dependent when considered independent of one another (see Equations 5 and 14). However, they are both proportionally frequency dependent to the same degree. As a result, the conditions that determine which of the conflicting levels of selection dominates total liability selection (*i.e*., Hamilton's rule) can be independent of the population incidence. Frequency dependence of Hamilton's rule may exist if the relationship between group incidence and fitness is nonlinear but our study demonstrates that this is not an emergent property of the threshold model. We infer that Hamilton's rule applies to threshold traits just as it does for continuous traits (Gardner *et al*. 2007).

The threshold model was developed largely to allow a breeder to predict a response to mass selection on incidence (*e.g*., Lush *et al*. 1948; Robertson and Lerner 1949; Dempster and Lerner 1950). These early quantitative genetic studies carefully defined the strength of selection on liability and described the relationship between narrow-sense heritability on the scales of liability and incidence. They appreciated that family-level selection (in the classical sense that excludes social effects) could increase the response to selection on incidence above what could be expected from mass selection. However, these early models do not consider indirect effects or selection at two levels simultaneously. Recently, Bijma *et al*. (2007a) showed that increased responses to artificial selection can be made by constructing a multilevel selection index and simultaneously performing selection at two levels. Our results indicate that given the constraint that group-level fitness is a linear function of group incidence, this optimal selection index does not change with incidence. In other words, the combination of selection gradients that works best at one population-level incidence (the optimal index) will work best at all incidences. In the general case (*e.g*., with nonlinear group-fitness functions), this optimal index may change with incidence, but the degree to which this happens is governed by the specific group-fitness function.

The relationship between group incidence and fitness can be nonlinear with natural selection (Foster 2004) or artificial selection (Mikami and Fredeen 1979). This is not an impediment to the contextual analysis framework used here because the definition of the covariance between group incidence and relative fitness (Equation 6) is general to all group fitness functions. Indeed, Hamilton's rule for threshold traits may become frequency dependent with a nonlinear group-fitness function, just as it would be if the rule were applied to outwardly continuous traits with a nonlinear group fitness function.

There are two compelling reasons to evaluate Hamilton's rule on the scale of liability when dealing with polygenic dimorphisms. In fact, these reasons apply equally to any evolutionary study of threshold traits that involves multilevel selection such as hard *vs*. soft selection, maternal effects, or kin selection (Wade 1985, 1998; Goodnight *et al*. 1992; Goodnight 2005). The first reason involves the partitioning of variance into within- and among-group components. This is important because the among-group component figures prominently into how we partition levels of liability selection (Equation 16). In conventional quantitative traits, the among-group phenotype variance is readily observable. However, the among-group component of liability variance must be inferred. Fortunately, several methods have been devised to infer narrow-sense heritability on the liability scale using the joint incidence patterns of relatives (reviewed in Lynch and Walsh 1998). Whereas these methods are designed to find the slope of the parent–offspring liability regression, they apply equally to the regression of individual on group liability. The square of this slope is the among-group component of liability variance because the total liability variance is defined as one.

The second reason for considering the liability scale when we study the evolution of polygenic dimorphisms is that liability is the metric with which we must model inheritance. We cannot predict the response to selection on threshold traits without estimates of heritability and genetic correlations. Fortunately, there is a rich literature dedicated to extracting these liability parameters at the level of the individual (Mercer and Hill 1984; Sorensen *et al*. 1995; Lynch and Walsh 1998) that makes predictions of evolutionary trajectories possible when selection acts at the level of the individual. Analogous group-level parameters are needed to accurately predict a response to multilevel selection (Goodnight 2005). New approaches for estimating these parameters for outwardly continuous traits have been proposed (Bijma *et al*. 2007b). Available methods for estimating relevant group-level liability parameters are still underdeveloped, however (but see Linksvayer 2006). More general approaches for estimating these liability parameters will need to be developed if we are to better understand the evolution of threshold characters.

Predicting the evolution of threshold characters requires an understanding of selection and inheritance on the latent scale of liability. Current models of liability selection and inheritance are adequate if an individual's phenotype is determined solely by direct effects. In the presence of indirect effects, however, a multilevel selection approach coupled with genetic models incorporating indirect genetic effects is required to predict evolutionary trajectories (Bijma *et al*. 2007a). We extend the contextual analysis methodology to multilevel liability selection. We show how the arbitrary relationships between fitness and group incidence, fitness and individual induction, and group structure can be transformed onto a scale that is useful for making evolutionary inferences. Although we show that the threshold transformation generates frequency-dependent selection at both levels, we find that the transformation does not distort Hamilton's rule.

## Acknowledgments

We thank David Hall, Daniel Promislow, Kenneth Ross, Michael Wade, Bruce Walsh, and two anonymous reviewers for helpful discussion and comments on this manuscript. This work was funded by a senior scholar award to D. Promislow from the Ellison Medical Foundation, a National Science Foundation (NSF) grant (0717234) to J.M., and a NSF postdoctoral fellowship to T.L.

## Footnotes

Communicating editor: J. B. Walsh

- Received January 11, 2008.
- Accepted March 14, 2008.

- Copyright © 2008 by the Genetics Society of America