## Abstract

The change in the frequency of a rare mutant allele under constant sex-differentiated viability selection in an infinite, partial full-sib mating population is studied. The diplo-diploid and haplo-diploid polygynous models are considered with a Poisson distribution for the number of offspring produced by every mated female. Reproduction is followed by weak selection among the offspring and then mating to form the next generation. It is shown that the rate of change with respect to the frequency of the mutant allele and the intensity of selection can be expressed in terms of costs or benefits of substituting the mutant type for the wild type, which correspond to average excesses in viability in females and males, multiplied by coefficients of relatedness to the individuals affected by such a substitution and reproductive values associated to the sexes of these individuals. This reveals hidden interactions between mated individuals and between males for mating, the former having positive effects on the reproductive success of related individuals and the latter having negative effects. Such interactions are the result of reproductive constraints when a fixed proportion of females must mate with a male sib and all females are fertilized as long as one mate is available. However, they affect the change in allele frequency because there is inbreeding or relatedness between mates and more generally relatedness between interacting individuals. Surprisingly, the effects of these interactions cancel out in a diploid population when the number of offspring is large enough so that the possibility for a female to have no male sib to mate with can be neglected and the viability differences are the same in both sexes.

STUDYING the change in the frequency of an allele at one locus in a partial sib-mating population under weak viability selection, Pollak (1995) found a discrepancy with the formula that is entailed by Wright's (1942) adaptive topography in the case of a partially inbred population that is close to equilibrium: the change obtained was actually given by Wright's formula times one plus the correlation coefficient between the frequencies of the allele in mated individuals. Caballero (1996) pointed out that Wright's formula may not apply to Pollak's model for the reason that it is a kind of fertility model with selection occurring after mating and showed that the formula should be fully valid if viability selection takes place before mating. Actually, with selection after mating, there is a kin selection effect (Hamilton 1964) due to the fact that mated individuals are related to one another and their reproductive success depends on the survival of both. This is the case even without interactions between kin affecting viability. The effects of such interactions before or after mating with small differences among phenotypes or fitnesses were studied still recently (Lessard and Rocheleau 2003, 2004) and led to the introduction of generalized coefficients of relatedness conditional on the inbreeding state of the interacting individuals (extending coefficients proposed in Lessard 1992) to describe the underlying factors responsible for the change in allele frequency. But the results obtained did not actually challenge the validity of Wright's formula when viability selection precedes mating, any kin selection effect seeming to disappear in the absence of interactions between related individuals affecting viability. In this model, however, differences in viability were assumed to be the same in both sexes. According to Uyenoyama (1984), sex-differentiated viabilities induce multiplicative models even in the presence of interactions whose effects combine additively. The analysis of such models should require more intricate, and therefore less tractable, measures of relatedness (see, *e.g.*, Uyenoyama and Feldman 1982).

A sex-differentiated viability model without interactions between individuals affecting viability is considered in this note. We also introduce stochastic variations on the number of offspring. The first objective is to confirm that kin selection is at work when there is some level of inbreeding in the population even when selection precedes mating. The second objective is to understand how coefficients of relatedness come into play in such models and what exact factors can explain their presence. The partial full-sib mating model with single insemination of females and a number of offspring following a Poisson distribution for diplo-diploid as well as haplo-diploid polygynous populations is used as an illustration but the conclusions should be extendable to many other models.

## MODEL AND RESULTS

#### Diplo-diploid population:

We consider an infinite, diploid population undergoing discrete, nonoverlapping generations and we assume viability selection before mating determined by two alleles at a single locus, *A*_{1} and *A*_{2}, such that the probability of surviving from conception to maturity associated to the *A _{i}A_{j}* genotype is

*f*=

_{ij}*f*(1 +

*u*) in females and

_{ij}s*m*=

_{ij}*m*(1 +

*v*) in males, independently for all individuals. The parameter

_{ij}s*s*measures the intensity of selection. It is assumed throughout to be positive and small, which models weak selection.

Following selection, every female mates with, and is inseminated once by, either a male chosen at random among its full-sibs with probability β or a male chosen at random among all males in the population at large with probability 1 − β. In the former case the female is said to be sib mated and in the latter randomly mated. Note that a female that has no male sib because none has been produced or has survived as a result of stochastic effects is assumed not to mate with probability β. To begin the next generation, every mated female produces a random number of offspring that follows a Poisson distribution of parameter 2λ, the offspring being male or female with probability independently of one another such that the numbers of male and female offspring are independent Poisson variables of mean λ.

Denoting by *x*_{1}, *x*_{2}, *x*_{3}, *x*_{4}, *x*_{5}, and *x*_{6} the frequencies of the mating types *A*_{1}*A*_{1} × *A*_{1}*A*_{1}, *A*_{1}*A*_{1} × *A*_{1}*A*_{2}, *A*_{1}*A*_{1} × *A*_{2}*A*_{2}, *A*_{1}*A*_{2} × *A*_{1}*A*_{2}, *A*_{1}*A*_{2} × *A*_{2}*A*_{2}, and *A*_{2}*A*_{2} × *A*_{2}*A*_{2}, respectively, in the current generation and assuming Mendelian segregation, the frequencies of the mating types in the next generation are given by the recurrence equations in Table 1.

Note that the expected contribution of a mated female in number of randomly mated female offspring is proportional to the mean viability of its female offspring, while this contribution in number of sib-mated female offspring is proportional to the mean viability of its female offspring times the probability that at least one male sib is produced and survives.

If the mated female is of type *A*_{1}*A*_{2} × *A*_{1}*A*_{2}, for instance, the numbers of its female and male offspring, respectively, of genotypes *A*_{1}*A*_{1}, *A*_{1}*A*_{2}, *A*_{2}*A*_{2}, respectively, that survive to maturity are independent Poisson variables of means and , respectively. Therefore, the number of its female offspring that mate with a random male in the population is a Poisson variable of mean , where . On the other hand, the number of its female offspring that mate with a random sib is a Poisson variable of mean with probability and 0 with probability , where . Moreover, given at least one mature male sib, such a sib chosen at random is of one of the genotypes *A*_{1}*A*_{1}, *A*_{1}*A*_{2}, *A*_{2}*A*_{2} with the probabilities , respectively. (This is a property of independent Poisson variables; see, *e.g.*, Lemire and Lessard 1997 for a proof and another application.) As a consequence, the mean number of *A*_{1}*A*_{1} female offspring that mate with an *A*_{1}*A*_{2} male sib, for instance, is times . The expected contributions of all other mating types in numbers of mated female offspring can be obtained in a similar way.

Observe that the proportion of sib-mated females in the next generation is(1)where(2)and(3)with and being the mean viabilities of male and female offspring, respectively, produced by mated females of types labeled from *i* = 1 to *i* = 6, respectively, as listed above. The proportion β_{s} is less than β since a fraction of the female offspring do not have male sibs to mate with and it is frequency dependent unless there is no viability difference in males. In such a case, this proportion reduces to(4)As λ grows to infinity, both β_{s} and β_{0} tend to β. The quantity *b* = *r*β_{0}, where(5)also plays a role in the analysis. Actually, we have(6)and this can be interpreted as the rate of increase in the proportion of sib-mated female offspring produced by a mated female with respect to an increase in the viability of its male offspring.

Assuming that allele *A*_{1} is a rare mutant, the frequencies *x*_{1}, *x*_{2}, *x*_{3}, *x*_{4}, *x*_{5} are all close to 0 and their transformation from one generation to the next has a matrix of linear approximation whose value and derivative with respect to *s* evaluated at *s* = 0, denoted by **M** and , respectively, are given in Table 2. Observe that the matrix **M** is nonnegative and admits an eigenvalue 1 with associated left and right positive eigenvectors **y** and **z**, respectively, also given in Table 2. The entries of **y** are proportional to the frequencies of *A*_{1} in the mating types and the entries of **z** are proportional to the frequencies of the mating types in the long run near the fixation state of *A*_{2}.

The frequency of allele *A*_{1}, given by , is represented by *p*. Its change from one generation to the next, after enough generations have passed and as long as *A*_{1} remains rare and selection is weak, is approximated by the formula(7)where(8)with T denoting matrix transposition (see, *e.g.*, Taylor 1985, 1989 for similar statements and Lessard and Rocheleau 2003 for a formal proof). In this approximation, terms of order *ps*^{2} or *p*^{2}*s* as well as all smaller terms when *p* and *s* are small are ignored. The quantity *L* can be seen as a rate of change with respect to the frequency of *A*_{1} and the intensity of selection.

Using Table 2 and following tedious algebraic manipulations, the following expression,(9)can be found, where(10)and(11)with(12)being the inbreeding coefficient under neutrality, that is, the probability that the two genes of an individual chosen at random in the population at equilibrium in the absence of selection (*s* = 0) are identical by descent (IBD). Moreover, we have(13)and(14)Here, *I* stands for a female, *J* for a male, *I*_{M} and *J*_{M} for the mates of *I* and *J*, respectively, and *S*_{M} for the mate of a sister. Moreover, represents the coefficient of relatedness of an individual *X* to an individual *Y* under neutrality (Michod and Hamilton 1980), which corresponds in the case of the neutral partial full-sib mating model at hand to the expected fraction of genes in *Y* IBD to one or more genes in *X* independently of the event that the genes in *X* are IBD or not (Lessard 1992). On the other hand, the fraction in front of these coefficients in Equation 9 corresponds to the reproductive value of the sex of *Y* (female if *I* or *J*_{M}, male if *J*, *I*_{M}, or *S*_{M}) in a neutral diplo-diploid population.

The quantities *a*(*I*) and *a*(*J*) multiplied by the intensity of selection give the average excesses in viability of *A*_{1} over *A*_{2} in females and males, respectively, which are inbred with probability *F* and outbred with probability 1 − *F*, when allele *A*_{1} is rare in the population, in agreement with Fisher's (1941) definition. They can be interpreted as costs (if negative) or benefits (if positive) of substituting *A*_{1} for *A*_{2} in a female and a male, respectively, in a population fixed for allele *A*_{2}. An extra cost or benefit is given by *ba*(*J*) when there is some positive probability that no male sib is produced and survives, as a result of stochastic effects, to fertilize the females that must sib mate.

#### Haplo-diploid population:

Assuming diploid females and haploid males with viabilities *f _{ij}* =

*f*(1 +

*u*) and

_{ij}s*m*=

_{i}*m*(1 +

*v*) for the genotypes

_{i}s*A*and

_{i}A_{j}*A*, respectively, the recurrence equations for the frequencies of the mating types

_{i}*A*

_{1}

*A*

_{1}×

*A*

_{1},

*A*

_{1}

*A*

_{2}×

*A*

_{1},

*A*

_{2}

*A*

_{2}×

*A*

_{1},

*A*

_{1}

*A*

_{1}×

*A*

_{2},

*A*

_{1}

*A*

_{2}×

*A*

_{2}, and

*A*

_{2}

*A*

_{2}×

*A*

_{2}represented by

*x*

_{1},

*x*

_{2},

*x*

_{3},

*x*

_{4},

*x*

_{5}, and

*x*

_{6}, respectively, are given in Table 3. This time, we find(15)(see Table 4), with the same definitions and expressions as previously for all coefficients but(16)and(17)With and being the reproductive values of females (the sex of

*I*and

*J*

_{M}) and males (the sex of

*J*,

*I*

_{M}, and

*S*

_{M}), respectively, in a neutral haplo-diploid population, the rate

*L*for a haplo-diploid population given in Equation 15 has the same structure as the rate given in Equation 9 for a diplo-diploid population.

## INTERPRETATION AND DISCUSSION

The rate of change in the frequency of a rare mutant allele in a partial full-sib mating population under weak sex-differentiated viability selection taking place before mating reveals interactions between individuals affecting reproductive success even in the absence of interactions affecting viability. Actually, an increase in the viability of males or females increases the reproductive fitness of their mates, but such an increase in males also decreases the reproductive fitness of other males. This explains the signs, positive or negative, of the different factors in Equation 9 and Equation 15. The first effect is due to the assumption that all females that survive are fertilized as long as one mate is available, and the second is due to the ancillary fact that the males are in competition to fertilize the females. With partial full-sib mating, a fixed proportion of females are constrained to sib mate, which creates mate competition between male sibs if some are produced and survive. Moreover, the females are related to their mates, which is responsible for inbreeding, and the males are related to competitors for mating. As a result, any excess in viability in females or males carrying the rare mutant allele, which corresponds to a benefit if positive or a cost if negative, is increased or decreased by an additive factor given by some coefficient of relatedness to the individual affected by such an excess times the total reproductive value associated to the sex of that individual. Then, we are in a framework of kin selection (Hamilton 1964; see also Hamilton 1972 for haplo-diploid populations) and it is revealed by sex differences in viability both in diplo-diploid and in haplo-diploid models.

When the number of offspring produced per mated female is small, the actual proportion of female offspring that sib mate is determined not only by the probability of having to mate with a male sib but also by the probability of having at least one male sib to mate with, and this uncertainty has an effect on the reproductive success of males and also indirectly on the reproductive success of their mates. When the number of offspring becomes large, this stochastic effect disappears but not the kin selection effects caused by inbreeding and local mate competition unless the population is diploid and the viability differences are the same in both sexes. It might be for the same reason that kin selection effects are not apparent in partial selfing models (Lessard and Rocheleau 2003).

The interactions between mates have an effect on the allelic frequencies because there is inbreeding, while related males interact because there is local mate competition. Each one of these factors was put forward to explain biased sex ratios in partial sib-mating populations and more generally in structured populations (Hamilton 1967; Maynard Smith and Stenseth 1978; see, *e.g.*, Karlin and Lessard 1986 and references therein). Our results enlighten the exact role of these factors in the change of allele frequencies at least in partial full-sib mating populations. Moreover, we have made some specific assumptions such as polygyny and single insemination. However, it is expected that the conclusions can be adapted to other models with different mating assumptions and constraints. Allowing multiple insemination, for instance, might remove competition between males, while allowing a female to mate with a male in the population at large when no male sib is available might remove stochastic effects entailed by a small number of offspring. Nevertheless, kin selection should be at work in a very wide range of models but be apparent only when there is some asymmetry between the sexes.

The coefficient of relatedness between the interacting individuals that comes into play in our analysis is the expected fraction of genes in the affected individual that are IBD to one or more genes in the affecting individual as originally proposed by Hamilton (1964) (see, *e.g.*, Michod and Hamilton 1980 for more general definitions) even if the effects of the interactions on reproductive success are not *a priori* additive and there is inbreeding in the population. But we have to remind ourselves that selection is assumed to be weak and that the linear approximation used under this assumption is tantamount to an additive model. Moreover, in partial full-sib mating populations, the coefficient of relatedness of an individual to another turns out not to depend on whether the individual is inbred or outbred (Lessard 1992). This is not generally the case in structured populations (Rocheleau 2003), for which we may have to resort to generalized coefficients of relatedness.

Finally, we have focused on the change in the frequency of a rare mutant allele in an infinite population for which approximations have been previously established (Lessard and Rocheleau 2003). It might be interesting to extend the analysis to the case of a mutant allele that has reached any frequency and even to the case of a finite population using related approximations as in Caballero and Hill (1992) and Roze and Rousset (2004).

## Acknowledgments

I thank two anonymous referees for their comments on a previous version of this manuscript and one of them in particular for suggesting to look at the effect of a finite number of offspring. This research was supported in part by the National Sciences and Engineering Research Council of Canada.

## Footnotes

Communicating editor: M. Uyenoyama

- Received November 29, 2004.
- Accepted May 26, 2005.

- Copyright © 2005 by the Genetics Society of America