Empirical Bayes Inference of pairwise FST and its distribution in the genome

Populations often have very complex hierarchical structure. Therefore, it is crucial in genetic monitoring and conservation biology to have a reliable estimate of the pattern of population subdivision. FSTs for pairs of sampled localities or subpopulations are crucial statistics for the exploratory analysis of population structures, such as cluster analysis and multidimensional scaling. However, the estimation of FST is not precise enough to reliably estimate the population structure and the extent of heterogeneity. This paper proposes an empirical Bayes procedure to estimate locus-specific pairwise FSTs. The posterior mean of the pairwise FST can be interpreted as a shrinkage estimator, which reduces the variance of conventional estimators largely at the expense of a small bias. The global FST of a population generally varies among loci in the genome. Our maximum likelihood estimates of global FSTs can be used as sufficient statistics to estimate the distribution of FST in the genome. We demonstrate the efficacy and robustness of our model by simulation and by an analysis of the microsatellite allele frequencies of the Pacific herring. The heterogeneity of the global FST in the genome is discussed based on the estimated distribution of the global FST for the herring and examples of human SNPs.

Key Words: distribution of FST, empirical Bayes, genome-wide estimation, pairwise FST, shrinkage estimator

Online ISSN: 1943-2361  Print ISSN: 0016-6731
Copyright 2009 by the Genetics Society of America
phone: 412-268-1812   fax: 412-268-1813   email: genetics-gsa{at}andrew.cmu.edu