A maximum likelihood method for estimation of pairwise relatedness in structured populations
Amy D Anderson 1* and Bruce S Weir 1
1 University of Washington
* To whom correspondence should be addressed. E-mail: ada891{at}u.washington.edu.
Submitted on July 10, 2006
Revised on November 29, 2006
Accepted on 16 February 2007
 |
Abstract |
|---|
A maximum likelihood estimator for pairwise relatedness is presented for the situation in which the individuals under consideration come from a large outbred subpopulation of the population for which allele frequencies are known. We demonstrate via simulations that a variety of commonly-used estimators that do not take this kind of misspecification of allele frequencies into account will systematically overestimate the degree of relatedness between two individuals from a subpopulation. A maximum-likelihood estimator that includes Fst as a parameter is introduced with the goal of producing the relatedness estimates that would have been obtained if the subpopulation allele frequencies had been known. This estimator is shown to work quite well, even when the value of Fst is misspecified. Bootstrap confidence intervals are also examined and shown to exhibit close to nominal coverage when Fst is correctly specified.
Key Words:
Fst, coancestry coefficient, genetic markers, natural populations, pairwise relatedness
