Originally published as Genetics Published Articles Ahead of Print on December 30, 2005.

Genetics, Vol. 172, 1809-1820, March 2006, Copyright © 2006
doi:10.1534/genetics.105.044099

Conditional Coalescent Trees With Two Mutation Rates and Their Application to Genomic Instability

Mathieu Emily and Olivier François1

TIMC–TIMB Department, Faculty of Medicine, Institut de l'Ingénierie de l'Information de Santé, 38706 La Tronche, France

1 Corresponding author: TIMC–TIMB Department, Faculty of Medicine, Institut de l'Ingénierie de l'Information de Santé, 38706 La Tronche, France.
E-mail: olivier.francois{at}imag.fr

Humans have invested several genes in DNA repair and fidelity replication. To account for the disparity between the rarity of mutations in normal cells and the large number of mutations present in cancer, an hypothesis is that cancer cells must exhibit a mutator phenotype (genomic instability) during tumor progression, with the initiation of abnormal mutation rates caused by the loss of mismatch repair. In this study we introduce a stochastic model of mutation in tumor cells with the aim of estimating the amount of genomic instability due to the alteration of DNA repair genes. Our approach took into account the difficulties generated by sampling within tumoral clones and the fact that these clones must be difficult to isolate. We provide corrections to two classical statistics to obtain unbiased estimators of the raised mutation rate, and we show that large statistical errors may be associated with such estimators. The power of these new statistics to reject genomic instability is assessed and proved to increase with the intensity of mutation rates. In addition, we show that genomic instability cannot be detected unless the raised mutation rates exceed the normal rates by a factor of at least 1000.