Power Analysis for Testing the Difference between Indep= endent Means Using G*Power and SPSS
Larry A. Pa= ce, Ph.D.
Anderson Un= iversity
and
TwoPaces, L= LC
Posted July= 23, 2008
Please ackn= owledge the author if you use this tutorial
Address all correspondence to larry@twopaces.com<= /a> or lpace@andersonuniversity.ed= u
This tutorial reviews the concept of statistical power= and illustrates the computations required for a power analysis comparing the means of two independent groups. This is the most commonly conducted power analysis in t= he behavioral sciences. The tutorial makes reference to a freely-available pow= er analysis tool called G*Power and to SPSS.
We can define statistical power as the likelihood of f= inding a significant result (rejecting the null hypothesis) when the null hypothes= is is false in the population (Welkowitz, Cohen, & Ewen, 2006). Another wa= y to state this is that power is the probability of finding an effect or differe= nce in a sample when the effect or difference is real in the population.
Before we discuss power any further, let us review some basic terms. In an experiment comparing the means of independent groups, we have a null hypothesis and an alternative or research hypothesis. The null hypothesis states that in the population the means of the two independent groups are equal, or equivalently that the difference between the two means= is zero. This is a statement of no difference. The alternative hypothesis stat= es that in the population the means of the two groups are not equal, or equivalently that the difference between the two means is not zero.
When performing a statistical test such as a t test, z test, or ANOVA to compare the means of the two independent groups, we determine the probability of observing a difference as large as = or larger than the difference in our sample results when the null hypothesis is true. In order to determine how big= a difference must be in order to qualify as “significant,” we establish an al= pha level in advance of the hypothesis test. This alpha level is the probabilit= y of rejecting the null hypothesis when it is true. If the null hypothesis is tr= ue in the population and we reject it on the basis of our sample results, we h= ave committed a Type I error. Alpha (α) is simply another name for the probability of Type I error. Because probabilities must add up to 1, 1 – &#= 945; would be the probability of retaining the null hypothesis when it is true in the population, which of course would be a correct decision.
It is also possible to make a different kind of error = with hypothesis testing. Assume that the null hypothesis is actually false in the population, that is, there really is a difference or effect. We could decid= e on the basis of our sample results to reject the null hypothesis, and that wou= ld be the correct decision. But we could also decide on the basis of our sample results to retain the null hypothesis even though it is false in the population. That is a Type II error, retaining a false null hypothesis. We = call Type II error beta (β). Therefore 1 – β is the probability of rejecting the null hypothesis when it is false in the population. This of course is our definition of power. The following table may be helpful in visualizing the relationship between Type I and Type II errors.
Decision = Made (Based on= Sample Results) |
True State of Affairs (What is True in the Population about the Null Hypothe= sis) |
|
|
H0 is True |
H0 is False |
|
|
Retain th= e Null Hypothesis |
Correct Decision 1 – α |
Incorrect Decision Type II Error β |
|
Reject th= e Null Hypothesis |
Incorrect Decision Type I Error α |
Correct Decision Power 1 – β |
Three major factors are involved in understanding and analyzing statistical power (Welkowitz, Cohen, & Ewen, 2006). These inc= lude the alpha level, the sample size, and the population effect size. These fac= tors are related to each other in such a way that any one of them can be determi= ned mathematically by a combination of the others. I will discuss the influence= of each factor on power separately before describing their relationship.
It should be obvious that the alpha level, our chosen significance criterion (usually .05), controls the probability of rejecting= the null hypothesis when it is true. But making the alpha level smaller in orde= r to decrease Type I error makes it more difficult to reject the null hypothesis both when it is true and when it is false. Thus, reducing the alpha level m= akes a test less powerful, and increasing the alpha level makes a test more powerful, though it does so at the expense of increased Type I error.
Another influence on statistical power related to ^= 5; is the use of a one-tailed or a two tailed hypothesis test. Remember that a two-tailed test for the difference between means places α/2 in each ta= il of the test distribution. This effectively increases the critical value and reduces the power of the test. When the researcher performs a one-tailed te= st, α is in one tail of the test distribution, effectively lowering the critical value and making the test more powerful (as long as the results are consistent with the alternative hypothesis).
Like the standard error terms for other parametric tes= ts, the standard error terms for comparing two independent sample means contain= a term representing the sample size. As a rule larger samples produce smaller standard errors. It follows directly that, other things being equal, error decreases and power increases when = N increases.
When the null hypothesis is false, it is not simply fa= lse: it is false to some degree. It might be only slightly wrong, somewhat wrong= , or very wrong. The population effect s= ize is an indication of how wrong t= he null hypothesis is in the population (Howell, 2008). The population effect = size index is a standardized measure of the degree to which the null hypothesis = is false, or how large or small the “effect” is in the population. For the purposes of this tutorial we will label the population effect size d. Other writers often use the te= rms g or gamma (γ) to refer to the population effect size.
With other things equal, the larger the effect in the population, the more likely the effect is to be detected, and the smaller t= he effect, the less likely it is to be detected. It is also true that very lar= ge effects can be detected with very small samples, while very small effects m= ay require very large samples to be detected. Thus the size of the effect in t= he population is directly related to the power of the statistical test.
Another index will be useful in discussing statistical= power for a two-sample test. This index combines effect size and sample size, and= is commonly called the noncentrality parameter. This combination is the quanti= ty δ (lowercase delta), which we can define as:
where n is=
the
size of each of the two samples, and thus 2n
cases are used in all. In cases where the two sample sizes are unequal, it =
is
customary in power analysis to use the harmonic
mean of the sample sizes and substitute that value =
We now have sufficient background to discuss post hoc = power determination and the a priori determination of required sample size for the comparison of two means.
For the comparison of two population means, we will as= sume that the standard deviation of the population is σ, and that for the t= wo populations, σ1 =3D σ2 =3D σ. The popu= lation effect size, d, then, can be calculated as
Note the similarity of the above equation to the defin= ition of a z score, and see that the division by the standard deviation creates a standardized effect size index that can be interpreted in standard deviation units. Also note that the ord= er of the subtraction of the means is immaterial in a two-sample test when the test is two-tailed, and therefore that an effect of -2.24 is equivalent to = an effect of 2.24.
Let us consider an example. Suppose that we are intere=
sted
in determining the effect size for the difference in two means where µ=
1
=3D 35 and µ2 =3D 27. Assume further that the sample sizes are 1=
8 each
for a total of 36 observations. Finally, assume that σ1 =3D
σ2 =3D σ =3D 16. The effect size, then, would simply b=
e
From the statistical literature, we know that we can interpret this value as a “medium” effect size. We can now calculate δ= :
Locating δ =3D 1.5 in a standard power table in a statistics text allows us to determine the statistical power of the hypothe= sis test comparing these two means. For our tutorial, we will assume an alpha l= evel of .05 and a two-tailed test. Both these choices are arbitrary but common. = We find that for our current case the power is .32. The interpretation is straightforward. We have a probability of .32 of correctly rejecting a false null hypothesis. Of course we also have a .68 probability of failing to rej= ect a false null hypothesis, which, as we know, is a Type II error.
The calculations described above can be reversed to determine the required sample size for a given level of power. For example, assume that we would be satisfied with a power of .80 (a commonly chosen fi= gure in power analysis). It should be obvious from the above discussion that a larger sample size must be used to achieve this level of power. By a little algebraic manipulation of the formula for δ we can arrive at the follo= wing equation:
Looking at the statistical table, we can determine tha= t for a power of .80 and a two-tailed test at alpha =3D .05, the value of δ = is 2.8. Substituting the required values, we determine that
So we will need approximately 2n =3D 126 total observations to achieve a power of .80 with the= given effect size and a .05 alpha level for a two-tailed significance test.
When conducting a post hoc power analysis for a two-sa= mple hypothesis test, one may conveniently use the sample means as estimators of= the population means, and the pooled standard deviation as an estimator of `= 3;. As mentioned above, for n one s= hould use the harmonic mean of the two sample sizes if they are unequal.
Although the calculations discussed above are not very difficult, they are simplified even further by the use of a free program ca= lled G*Power (Erdfelder, Faul, & Buchner, n.d.).
The computations of the above problem are now shown in G*Power 3.0.8. We enter the values for the number of tails, the alpha level, and the calculated effect size in the dialog, making sure we have selected a two-tailed test at alpha =3D .05 for comparing the means of independent sam= ples (see below). After entering the appropriate values click on Calculate.
Note that the G*Power program’s calculations are sligh= tly different from our hand calculations above. This because our calculations w= ere based on the standard normal (z) distribution, while the G*Power program makes use of the noncentral t distribution. The noncentral t distribution is so-named because unlike the standard t distribut= ion, which is centered around a mean of zero, the noncentral distribution is centered around some value other than zero. G*Power calls the value of ^= 8; we calculated above the noncentrality parameter. The output from the G*Power program is below:
We can also use G*Power to determine the required samp= le size to achieve a given level of power. Note that this is called an a priori analysis. The completed d= ialog is below:
When we click the Calculate button, the results are as follows:
Note once again the similarity of the output of the G*= Power program to the result of our hand calculations above. The estimated sample sizes are 64 each (we calculated 63), and the total sample size needed is 1= 28. Clicking on the tab labeled “Protocol of power analysis” allows one to save= and print the results of the analysis.
Although the determination of power and required sample sizes involves fairly simple calculations, the G*Power program is nonethele= ss quite instructive and useful because of its effective use of graphics and its intuitive interface. It is also possible to use SPSS to find observed power= in a two-sample mean comparison if one knows which menus to access. The calcul= ated power in SPSS in my personal experience agrees with the calculations of G*Power.
The standard approach to performing the comparison of = the means from two independent samples in SPSS is the independent-samples t test. This procedure is accessed= from the Analyze menu. Select Analyze, Compare Means, Independent-Samples T Test, as shown below:
However, this procedure does not calculate observed po= wer or effect size. One may use the General Linear Model submenu and perform an equivalent analysis of variance (the square root of the F ratio will be the same as the value of t). Furthermore, the General Linear Model approach allows the estimation of effect size and observed power. To access these features, cho= ose Analyze, General Linear Model, = Univariate as shown below:
In the result= ing dialog, notice the Options but= ton. Click on that to reveal the following:
Note that one may choose estimates of effect size and observed power. The effect-size estimates in SPSS will be partial eta-squar= es (proportions of variance) rather than d as discussed in this tutorial, and the noncentrality parameter is from the noncentral F distribution, rath= er than t. However, as mentioned earlier, the square root of the F ratio produces the value of t calculated in the independent-samples t test, and the power calculation agrees with that of G*Power.
Additionally, when one chooses the descriptive statist= ics option in this SPSS menu, the means and standard deviations for both groups= are computed as well as the overall mean and the pooled standard deviation, mak= ing that calculation unnecessary. This feature is not found in the menus for AN= OVA and t tests.
The following SPSS output is from the General Linear M= odel analysis of the comparison of the two means for the Appendix D data file fr= om Howell (2008), comparing males and females on the variable addsc. You may retrieve a= copy of the data file. To perform the ANOVA, select Analyze, General Linear Model, Univariate as discu= ssed above. In the resulting dialog, enter addsc as the Dependent variable and gende= r as the Fixed Factor (see below). Click on Options and select Descriptive statistics, Estimates of effect size, and Observed p= ower as discussed above.
The SPSS Viewer output is shown below. Again, note tha= t the pooled (Total) standard deviation is displayed, eliminating the need to calculate it:
The comparison of the means is performed by ANOVA, and= both effect size and observed (post hoc) power are calculated and displayed:
Calculating the power by hand substituting sample valu= es for the population ones, and using the pooled standard deviation of 12.478 and = the formulas discussed in this tutorial, we obtain:
and
In this case the noncentrality parameter is equal to t= he value of the t statistic we calculate! Note equivalently that the value above when squared equals the v= alue of F reported above both as the significance test value and as the noncentrality parameter by SPSS. Finding δ =3D 1.006 in the power table with a .05 alpha level and a two-tailed= test yields an observed power of approximately .17, similar to the calculated .1= 68 above.
When we use the G*Power program to do the analysis, our results are as follows. After selecting the independent group comparison and post hoc analysis, one may click on Determine and enter the sample values when the sample standard deviations or the samp= le sizes are unequal. I entered these by copying them directly from the SPSS Viewer window:
Click on Calc= ulate and transfer to main window. Next enter the appropriate number of tails= for the test, the sample sizes, and the alpha level:
Click on Calc= ulate for the results (see below). Note that when the actual sample standard devi= ations are used, SPSS and G*Power produce equivalent observed power estimates.
SPSS has no facility of which I am aware to conduct the analysis of required sample sizes, so the reader may find it convenient to = use G*Power or a spreadsheet program like Microsoft Excel for this purpose.
Erdfelder, E., Faul, F., & Buchner, A. (n.d.) A general power analysis program, fr= ee for Mac and PC. Retrieved January 19, 2008 from http://= www.psycho.uni-duesseldorf.de/aap/projects/gpower/
Howell, D. (2008). Fundamental Statistics f= or the Behavioral Sciences, 6th ed.). Belmont, CA: Thomson Wadswort= h.
Welkowitz, J., Cohen, B., & Ewen, R. (2006). Introductory Statistics for the Behavioral Sciences, 6th ed. New York: Wi= ley.
= = Power Analysis 15