## Abstract

With respect to the multiple-tests problem, recently an increasing amount of attention has been paid to control the false discovery rate (FDR), the positive false discovery rate (pFDR), and the proportion of false positives (PFP). The new approaches are generally believed to be more powerful than the classical Bonferroni one. This article focuses on the PFP approach. It demonstrates via examples in genetic association studies that the Bonferroni procedure can be more powerful than the PFP-control one and also shows the intrinsic connection between controlling the PFP and controlling the overall type I error rate. Since controlling the PFP does not necessarily lead to a desired power level, this article addresses the design issue and recommends the sample sizes that can attain the desired power levels when the PFP is controlled. The results in this article also provide rough guidance for the sample sizes to achieve the desired power levels when the FDR and especially the pFDR are controlled.

FOR multiple tests, the classical approach is to control the overall type I error rate [*i.e.*, the familywise error rate (FWER)]. Bonferroni correction is often used to this end. The method, however, often leads to a very stringent significance level for each test. As a remedial measure, the false discovery rate (FDR) was thus introduced recently and controlled in many investigations. Benjamini and Hochberg (1995) define the (unconditional) FDR aswhere *R* is the number of positives, and *V* is the number of false positives. In practice, FDR will be controlled by α if the hypotheses corresponding to the *k* smallest *P*-values are rejected, where *k* is the largest *j* such that the *j*th smallest *P*-value , and *m* is the number of the hypotheses tested. Some numerical results show that controlling FDR can lead to higher powers (see, for example, Benjamini and Hochberg 1995; Sabatti *et al.* 2003). However, it should be pointed out that these power comparisons are not based on the same FWER level and hence do not necessarily imply that using FDR is more powerful than using Bonferroni correction at the same overall type I error level, because controlling the former has a different meaning from controlling the latter. Recently, the positive (or conditional) false discovery rate (pFDR) was discussed by Storey (2002) withStorey (2002) also argues that it is more suitable than FDR in practice. Another related concept, the proportion of false positives (PFP), is suggested by Southey and Fernando (1998) and Fernando *et al.* (2004). They define the PFP asThese authors show through simulation studies that PFP is often close to FDR and pFDR. In general, PFP is closer to pFDR than to FDR. In fact, Storey (2003) shows that pFDR = PFP when the tests are independent and follow a mixture distribution. Clearly, when most null hypotheses are true, the discrepancy between FDR and either pFDR or PFP will increase. The connections and differences of these various measures have been discussed in Zaykin *et al.* (2000), Storey (2003), and Fernando *et al.* (2004). In practice, Benjamini and Hochberg's FDR is the easiest to estimate whereas pFDR is most difficult to control (although very important). Noting that PFP is closer to pFDR, we focus on PFP below.

In genetic association studies, it is usually very likely that a marker tested is not associated with the disease of interest; that is, most null hypotheses are true. This means that a very stringent significance level may be sometimes appropriate in these settings. Otherwise, the false positive rate will be very high, a situation the applied researchers often want to avoid. On the basis of this observation, we present some examples that compare the power of the classical FWER controlling procedure with the PFP controlling one when the levels of FWER and PFP are set to be the same. It turns out that the Bonferroni approach can outperform the PFP approach in the aforementioned settings. Furthermore, we demonstrate that for a specified problem, controlling PFP is in fact approximately equivalent to Bonferroni correction by setting corresponding (different) levels of PFP and FWER. In this regard, Westfall *et al.* (1997) consider the conditions under which the Bonferroni correction behaves as the posterior probability (*i.e.*, essentially pFDR or PFP).

On the other hand, if we control only the PFP level, then a low power may result; that is, controlling only PFP is not sufficient to achieve a desired power level in multiple tests. An intuitive explanation is that PFP considers only the tests that are rejected. When the power is low, most (often even all) true alternatives will not be rejected. Therefore, simultaneous consideration of both PFP and power is very relevant and important. This article considers mainly the design issue for multiple case-control association tests when PFP is controlled. The sample sizes that can lead to the desired PFP and power levels are recommended.

## METHODS

#### Genetic model and power calculation:

We first give the formula for calculating the power under various genetic models (*cf.*, Zou and Zhao 2004). We consider two alleles, *A* and *a*, at a candidate marker, whose frequencies are *p* and in the population, respectively. For simplicity, we consider a case-control study with *n* cases and *n* controls. Let *X*_{i} denote the number of alleles *A* carried by the *i*th individual in the case group, and *Y*_{i} is defined similarly for the *i*th individual in the control group. Assuming Hardy-Weinberg equilibrium, each *X*_{i} and *Y*_{i} has a value of 2, 1, 0 with respective probabilities *p*^{2}, *2pq*, and *q*^{2} under the null hypothesis of no association between the candidate marker and disease. When the candidate marker is associated with disease, we assume that the penetrances are *f*_{2} for genotype *AA*, *f*_{1} for genotype *Aa*, and *f*_{0} for genotype *aa*. Without loss of generality, we let . Note that the two alleles may be true functional alleles or may be in linkage disequilibrium with true functional alleles. Under this genetic model, the probabilities of having *k* copies of *A* among the cases, , and among the controls, , areLet *n*_{A} and *n*_{U} denote the observed numbers of allele *A* in the case group and the control group and *p*_{A} and *p*_{U} denote the population frequencies of allele *A* in these two groups, respectively. Then the statistic to test the association between the candidate marker and disease iswhere .

Consider a one-sided test and use a significance level of α. The power of the test statistic *t* is(1)where is the cumulative standard normal distribution function, *z*_{α} is the upper 100αth percentile of the standard normal distribution, is the expected frequency of allele *A* under the genetic model, μ is the expected difference of estimated allele frequencies between cases and controls, which is given byand σ^{2}/*n* is the corresponding variance with σ^{2} being given byThus, assuming *M* markers are tested, when Bonferroni correction is used, the power that a disease-associated marker is detected is given by

#### Approximate equivalence of controlling FWER and PFP:

In the genetic association test between gene and disease, for the case of testing *M* markers, Southey and Fernando (1998) define the PFP as(2)where α_{i} is the significance level, 1 − β_{i} is the power at the *i*th marker whose calculation formula for different genetic models is provided in Equation 1, and is the prior probability of the null (alternative) hypothesis being true for the *i*th test. Fernando *et al.* (2004) show that if for each test the PFP is controlled by γ, then for all *M* tests it is controlled still by γ.

We now show that for a specified scenario, *i.e.*, for a given number *K* of truly disease-associated markers, genetic models, and population allele frequencies, controlling FWER is substantially equivalent to controlling PFP. In fact, for any given PFP level of γ, we can find an overall significance level of α such that this PFP level can be obtained: First, we choose the type I error rate α_{i} for the *i*th test, which satisfies(3)(or Equation 2). Then we take . Clearly, such a choice of the overall significance level (correspondingly, the significance level for *i*th test is taken as α_{i}, which can be regarded as a generalized Bonferroni correction) will lead to the PFP level of γ. On the other hand, for any given overall significance level of α, we can find a PFP level of γ such that this overall significance level is attained: First, we calculatewith . Then we take [here we assume that the prior probability . Obviously, . Such a choice of the PFP level (correspondingly, the PFP level for *i*th test is taken as γ_{i}) can lead to the overall type I error rate of α.

Note that in practical genetic studies, the true number of disease-associated markers and the genetic models are unknown to us; the equivalence should be approximate by using their estimates.

## RESULTS

We have seen that controlling FWER is in fact equivalent to controlling PFP by setting their different levels. Here we provide examples that show that controlling FWER can lead to higher powers than controlling PFP, even though their levels are set to be the same.

For comparing the powers based on Bonferroni correction and based on controlling PFP, we assume that the overall significance level α for *M* tests is the same as the PFP level γ (=α) for these tests and consider *M* = 1000. Then for each test, the significance level α_{i} = α/*M* if we use Bonferroni correction, and PFP if we control PFP. The power results are summarized in Table 1, where the true number of disease-associated markers *K* is set to be 1, 2, and 5, and the prior probability is assumed to be .

From Table 1, we observe that for a small allele frequency difference (5%) between the case and control groups, using Bonferroni correction for multiple tests often leads to larger power, especially for the case of small sample size (say, 200) and small number of disease-associated markers. This is true even for larger allele frequency difference when the sample size is smaller. On the other hand, controlling PFP can lead to higher power for large allele frequency difference (10%) between the cases and controls, especially for large sample sizes (1000) and a large number of disease-associated markers. This is true for smaller allele frequency difference when the sample size is larger. Note that the number of disease-associated markers is unknown in practical genetic association studies. Thus, overall, to detect disease markers with small allele frequency difference (such as 5%), using Bonferroni correction for sample sizes that are not large (such as 500 and 200) can give a higher level of power; to detect disease markers with large allele frequency difference (such as 10%), using PFP for sample sizes that are not small (such as 1000 and 500) will lead to a better result in power.

To calculate the sample size for attaining a desired PFP level γ and a desired power level that a disease-associated marker is detected, we assume that the prior probability of the marker tested being truly disease associated is 0.0001 and use Equation 3. We find that the sample size required depends on the genetic model and population allele frequency substantially through the allele frequency difference between the cases and controls and population allele frequency *p* (the results for the PFP level of 0.05 under dominant, recessive, multiplicative, and additive models with are presented in Table 2; a similar conclusion for the power of the two-stage design can be found in Y. Zuo, G. Zou and H. Zhao, unpublished results). On the basis of this, we consider a recessive genetic model and let in our calculation for simplicity when the PFP levels of 0.20 and 0.50 are used. Tables 2–4⇓⇓ present the sample sizes to attain the PFP levels , 0.20, and 0.50 and the power level for various allele frequency differences between the cases and controls and population allele frequencies, respectively. Interestingly, it can be seen from Tables 2–4⇓⇓ that reducing the level of PFP will not necessarily lead to a great increase in sample size required. Therefore, we can use the sample sizes derived by setting PFP at a small level. This will not significantly increase the cost of the experiment.

## DISCUSSION

In multiple tests, there is an increasing trend to use FDR, pFDR, and PFP as measures of global error instead of using overall type I error rate. This article gives the examples on the power comparison between controlling FWER and PFP when their levels are set to be the same (as is usually done in the literature), which show that using Bonferroni correction does not necessarily lead to a lower power. This article also shows that controlling FWER and controlling PFP, seemingly two different approaches, based on prior and posterior probabilities, respectively, are actually intrinsically equivalent.

Note that controlling only PFP does not necessarily lead to a desired level of power. We work out the sample size to attain the desired power that a disease-associated marker is detected under various population allele frequencies and various allele frequency differences between the cases and controls when PFP is controlled. Our results reveal that lowering the PFP level alone will not give rise to much increase in sample size required to attain a desired power level. Therefore, taking a small PFP level may be appropriate in general in multiple case-control association tests. Further, as we have seen, FDR and especially pFDR are often close to PFP. Combining this and the above fact that the effect of PFP level on the sample sizes required is not large, we see that the sample sizes we obtained should be useful when FDR and especially pFDR are controlled.

Finally, we remark that the sample size calculation in this article is done for unrelated individual data. We note that family-based data are also often used in genetic epidemiological studies. The design issue for such data, which is not pursued here, is no doubt an interesting topic when PFP (or FDR or pFDR) is controlled.

## Acknowledgments

The authors are grateful to the associate editor Rebecca Doerge and to the two reviewers for their constructive comments and suggestions that led to substantial improvements of the original manuscript. This work is supported in part by grants DMS0234078 from the National Science Foundation (to Y. Zuo) and 70221001 and 10471043 from National Natural Science Foundation of China (to G. Zou).

## Footnotes

Communicating editor: R. W. Doerge

- Received August 14, 2005.
- Accepted September 21, 2005.

- Copyright © 2006 by the Genetics Society of America