PT - JOURNAL ARTICLE
AU - Rosen, Zvi
AU - Bhaskar, Anand
AU - Roch, Sebastien
AU - Song, Yun S.
TI - Geometry of the Sample Frequency Spectrum and the Perils of Demographic Inference
AID - 10.1534/genetics.118.300733
DP - 2018 Oct 01
TA - Genetics
PG - 665--682
VI - 210
IP - 2
4099 - http://www.genetics.org/content/210/2/665.short
4100 - http://www.genetics.org/content/210/2/665.full
SO - Genetics2018 Oct 01; 210
AB - The sample frequency spectrum (SFS), which describes the distribution of mutant alleles in a sample of DNA sequences, is a widely used summary statistic in population genetics. The expected SFS has a strong dependence on the historical population demography and this property is exploited by popular statistical methods to infer complex demographic histories from DNA sequence data. Most, if not all, of these inference methods exhibit pathological behavior, however. Specifically, they often display runaway behavior in optimization, where the inferred population sizes and epoch durations can degenerate to zero or diverge to infinity, and show undesirable sensitivity to perturbations in the data. The goal of this article is to provide theoretical insights into why such problems arise. To this end, we characterize the geometry of the expected SFS for piecewise-constant demographies and use our results to show that the aforementioned pathological behavior of popular inference methods is intrinsic to the geometry of the expected SFS. We provide explicit descriptions and visualizations for a toy model, and generalize our intuition to arbitrary sample sizes using tools from convex and algebraic geometry. We also develop a universal characterization result which shows that the expected SFS of a sample of size n under an arbitrary population history can be recapitulated by a piecewise-constant demography with only epochs, where is between and The set of expected SFS for piecewise-constant demographies with fewer than epochs is open and nonconvex, which causes the above phenomena for inference from data.