Skip to main content
  • Facebook
  • Twitter
  • YouTube
  • LinkedIn
  • Google Plus
  • Other GSA Resources
    • Genetics Society of America
    • G3: Genes | Genomes | Genetics
    • Genes to Genomes: The GSA Blog
    • GSA Conferences
    • GeneticsCareers.org
  • Log in
Genetics

Main menu

  • HOME
  • ISSUES
    • Current Issue
    • Early Online
    • Archive
  • ABOUT
    • About the journal
    • Why publish with us?
    • Editorial board
    • Early Career Reviewers
    • Contact us
  • SERIES
    • Centennial
    • Genetics of Immunity
    • Genetics of Sex
    • Genomic Prediction
    • Multiparental Populations
    • FlyBook
    • WormBook
    • YeastBook
  • ARTICLE TYPES
    • About Article Types
    • Commentaries
    • Editorials
    • GSA Honors and Awards
    • Methods, Technology & Resources
    • Perspectives
    • Primers
    • Reviews
    • Toolbox Reviews
  • PUBLISH & REVIEW
    • Scope & publication policies
    • Submission & review process
    • Article types
    • Prepare your manuscript
    • Submit your manuscript
    • After acceptance
    • Guidelines for reviewers
  • SUBSCRIBE
    • Why subscribe?
    • For institutions
    • For individuals
    • Email alerts
    • RSS feeds
  • Other GSA Resources
    • Genetics Society of America
    • G3: Genes | Genomes | Genetics
    • Genes to Genomes: The GSA Blog
    • GSA Conferences
    • GeneticsCareers.org

User menu

  • Log out

Search

  • Advanced search
Genetics

Advanced Search

  • HOME
  • ISSUES
    • Current Issue
    • Early Online
    • Archive
  • ABOUT
    • About the journal
    • Why publish with us?
    • Editorial board
    • Early Career Reviewers
    • Contact us
  • SERIES
    • Centennial
    • Genetics of Immunity
    • Genetics of Sex
    • Genomic Prediction
    • Multiparental Populations
    • FlyBook
    • WormBook
    • YeastBook
  • ARTICLE TYPES
    • About Article Types
    • Commentaries
    • Editorials
    • GSA Honors and Awards
    • Methods, Technology & Resources
    • Perspectives
    • Primers
    • Reviews
    • Toolbox Reviews
  • PUBLISH & REVIEW
    • Scope & publication policies
    • Submission & review process
    • Article types
    • Prepare your manuscript
    • Submit your manuscript
    • After acceptance
    • Guidelines for reviewers
  • SUBSCRIBE
    • Why subscribe?
    • For institutions
    • For individuals
    • Email alerts
    • RSS feeds
Previous ArticleNext Article

Naive Application of Permutation Testing Leads to Inflated Type I Error Rates

G. A. Churchill and R. W. Doerge
Genetics January 1, 2008 vol. 178 no. 1 609-610; https://doi.org/10.1534/genetics.107.074609
G. A. Churchill
The Jackson Laboratory, Bar Harbor, Maine 04609 and Department of Statistics, Purdue University, West Lafayette, Indiana 47907
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
R. W. Doerge
The Jackson Laboratory, Bar Harbor, Maine 04609 and Department of Statistics, Purdue University, West Lafayette, Indiana 47907
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: doerge@purdue.edu
  • Article
  • Info & Metrics
Loading

Abstract

Failure to account for family structure within populations or in complex mating designs via uninformed applications of permutation testing will lead to inflated type I error rates. Careful consideration of the design factors is essential since some situations allow several valid permutation strategies, and the choice that maximizes statistical power will not always be intuitive.

WHEN any statistical method is applied incorrectly, misleading conclusions are to be expected. With this in mind, we are motivated by both the application of a simple permutation test and the message conveyed by Zouet al. (2006) in which the performance of a permutation test is assessed. Zou et al. address a problem in QTL mapping analysis of recombinant inbred lines (RILs), which has been previously addressed by Belknap (1998), concerning the use of strain means vs. individual level data models. Although the essentially equivalent performance of these two models is well known anecdotally, Zou et al. focus on the performance of a simple permutation application which unfortunately is applied incorrectly in the context of the full data model.

The concept of permutation, as proposed by Fisher (1935) and as applied to QTL mapping by Churchill and Doerge (1994), relies on exchangeability. In simple experimental designs, such as an intercross or a backcross mapping population, the individual units can safely be assumed to be exchangeable. In more complex designs it is often, but not always, possible to identify exchangeable units and thus to construct a permutation test. It is important to realize that several other designs exist in which a simple permutation test has been or could readily be misapplied, for example, the advanced intercross (AI) and heterogeneous stock (HS) breeding designs. In both the AI and HS designs, the animals that are assayed may number in the hundreds or even thousands, but these animals are often the progeny of a penultimate generation consisting of a much smaller number of lineages. Failure to account for the family structure in these populations by naive application of permutation testing will lead to inflated type I error rates.

Permutation tests require relatively few assumptions and can be applied in a wide variety of settings. However, they are not without potential pitfalls. Correlation structure, whether it is known from the design or hidden due to unaccounted factors, can produce misleading results. The optimal choice of permutation strategy requires careful consideration of the experimental design. Design factors may be fixed or random, nested or crossed, and it is these features that determine which strategy should be used. To construct a permutation test, one must decide which units are to be permuted, whether the permutations should be restricted, and whether it is best to permute raw data or residuals. The implications of these choices have been exhaustively characterized in all combinations by Anderson and Ter Braak (2003). Their results sometimes run counter to our intuitions. In general they find that permutation of residuals under a reduced model has the best power while still controlling type I errors. However, this does not apply as a universal recommendation.

Using the notation of Zouet al. (2006) with the explicit addition of the random line effect, Bi, the ANOVA model for the RI line experiment considered by Zou et al. can be written asEmbedded Imagewhere i = 1, … , L and j = 1, … , ni, such that there are a total of L lines used for QTL mapping, and each line has ni individuals measured for the quantitative trait yij. The additive effect, assuming a single-marker analysis, of the putative QTL is denoted as ak. Each individual within a given line i will have the same genotype xik, and thus the random effect Bi is nested within the fixed effect. Typically, one would want to test the higher-order, fixed effect (i.e., the additive effect). For this case, Anderson and Ter Braak (2003) recommend the exact test in which data are permuted as units within levels of the random factor. This is the permutation test recommended by Zouet al. (2006). All other permutation strategies, including restricted permutation of residuals under the reduced model ak = 0 and unrestricted permutation of the data, can produce inflated type I error rates when errors are not normal. Mostly important to this discussion is the fact that the effect is most pronounced when the variance of the nested factor (the polygenic background variance) is large. Furthermore, to maximize the power on a per-measurement basis, investigators are well advised to increase the number of lines and to use minimal within-line replication (also see Belknap 1998).

In summary, there are many ways to construct a permutation test. Careful consideration of the nature of the design factors is essential to making the correct choice. In some cases there are several valid permutation strategies and the choice that maximizes power will not always be intuitively obvious.

Footnotes

  • Communicating editor: L. McIntyre

  • Received April 15, 2007.
  • Accepted October 18, 2007.
  • Copyright © 2008 by the Genetics Society of America

References

  1. ↵
    Anderson, M. J., and C. J. F. Ter Braak, 2003 Permutation tests for multi-factorial analysis of variance. J. Statist. Comput. Simul. 73(2): 85–113.
    OpenUrl
  2. ↵
    Belknap, J. K., 1998 Effect of within-strain sample size on QTL detection and mapping using recombinant inbred mouse strains. Behav. Genet. 28(1): 29–38.
    OpenUrlCrossRefPubMedWeb of Science
  3. ↵
    Churchill, G. A., and R. W. Doerge, 1994 Empirical threshold values for quantitative trait mapping. Genetics 138: 963–971.
    OpenUrlAbstract/FREE Full Text
  4. ↵
    Fisher, R. A., 1935 The Design of Experiments, Ed. 3. Oliver & Boyd, London.
  5. ↵
    Zou, F., Z. Xu and T. Vision, 2006 Assessing the significance of quantitative trait loci in replicable mapping populations. Genetics 174: 1063–1068.
    OpenUrlAbstract/FREE Full Text
View Abstract
Previous ArticleNext Article
Back to top

PUBLICATION INFORMATION

Volume 178 Issue 1, January 2008

Genetics: 178 (1)

ARTICLE CLASSIFICATION

Notes
Genetics of complex traits
Multiparental Populations
View this article with LENS
Email

Thank you for sharing this Genetics article.

NOTE: We request your email address only to inform the recipient that it was you who recommended this article, and that it is not junk mail. We do not retain these email addresses.

Enter multiple addresses on separate lines or separate them with commas.
Naive Application of Permutation Testing Leads to Inflated Type I Error Rates
(Your Name) has forwarded a page to you from Genetics
(Your Name) thought you would be interested in this article in Genetics.
Print
Alerts
Enter your email below to set up alert notifications for new article, or to manage your existing alerts.
SIGN UP OR SIGN IN WITH YOUR EMAIL
View PDF
Share

Naive Application of Permutation Testing Leads to Inflated Type I Error Rates

G. A. Churchill and R. W. Doerge
Genetics January 1, 2008 vol. 178 no. 1 609-610; https://doi.org/10.1534/genetics.107.074609
G. A. Churchill
The Jackson Laboratory, Bar Harbor, Maine 04609 and Department of Statistics, Purdue University, West Lafayette, Indiana 47907
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
R. W. Doerge
The Jackson Laboratory, Bar Harbor, Maine 04609 and Department of Statistics, Purdue University, West Lafayette, Indiana 47907
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: doerge@purdue.edu
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
Citation

Naive Application of Permutation Testing Leads to Inflated Type I Error Rates

G. A. Churchill and R. W. Doerge
Genetics January 1, 2008 vol. 178 no. 1 609-610; https://doi.org/10.1534/genetics.107.074609
G. A. Churchill
The Jackson Laboratory, Bar Harbor, Maine 04609 and Department of Statistics, Purdue University, West Lafayette, Indiana 47907
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
R. W. Doerge
The Jackson Laboratory, Bar Harbor, Maine 04609 and Department of Statistics, Purdue University, West Lafayette, Indiana 47907
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: doerge@purdue.edu

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero

Related Articles

Cited By

More in this TOC Section

Notes

  • Transgene-Free Genome Editing in Caenorhabditis elegans Using CRISPR-Cas
  • Targeted Heritable Mutation and Gene Conversion by Cas9-CRISPR in Caenorhabditis elegans
  • Heritable Gene Knockout in Caenorhabditis elegans by Direct Injection of Cas9–sgRNA Ribonucleoproteins
Show more Notes

Genetics of complex traits

  • Evidence for Weak Selective Constraint on Human Gene Expression
  • The Genetic Basis of Mutation Rate Variation in Yeast
  • Decoupling the Variances of Heterosis and Inbreeding Effects Is Evidenced in Yeast’s Life-History and Proteomic Traits
Show more Genetics of complex traits

Multiparental Populations

  • Evidence for Weak Selective Constraint on Human Gene Expression
  • The Genetic Basis of Mutation Rate Variation in Yeast
  • Decoupling the Variances of Heterosis and Inbreeding Effects Is Evidenced in Yeast’s Life-History and Proteomic Traits
Show more Multiparental Populations
  • Top
  • Article
    • Abstract
    • Footnotes
    • References
  • Info & Metrics

GSA

The Genetics Society of America (GSA), founded in 1931, is the professional membership organization for scientific researchers and educators in the field of genetics. Our members work to advance knowledge in the basic mechanisms of inheritance, from the molecular to the population level.

Online ISSN: 1943-2631

  • For Authors
  • For Reviewers
  • For Subscribers
  • Submit a Manuscript
  • Editorial Board
  • Press Releases

SPPA Logo

GET CONNECTED

RSS  Subscribe with RSS.

email  Subscribe via email. Sign up to receive alert notifications of new articles.

  • Facebook
  • Twitter
  • YouTube
  • LinkedIn
  • Google Plus

Copyright © 2019 by the Genetics Society of America

  • About GENETICS
  • Terms of use
  • Advertising
  • Permissions
  • Contact us
  • International access