Power and Sample Size for Two-sample Tests | Statistical Consulting Group

An often used method in applied statistics is determining the sample size necessary to view statistically significant results. Given the intended power, we can calculate the required sample size. Given the intended sample size, we can calculate the resulting power. Before we go in to how this works, we need to define a few things.
Error Types

		Truth
		H₀	H₁
Test	Negative Don’t Reject	True Negative	False Negative β
Test	Positive Reject	False Positive α	True Positive Power = 1 – β

In looking at a two-sample test, what we’re actually focusing on is the difference of the sample means. Let’s say we have 2 samples.

$\begin{equation*} \ux = \{x_1,\ldots,x_{n_x}\}\hspace{2cm}\uy = \{y_1,\ldots,y_{n_y}\} \end{equation*}$

For purposes of this demonstration, we will assume that the data in these two samples follow a normal distribution. Therefore, their sample mean follows a normal distribution. We also assume that $x$ and $y$ have the same variance. And we also assume that $x$ is independent of $y$ . Therefore,

$\begin{align*} x_1,\ldots,x_n \sim N(\mu_x,\sigma_x^2)\hspace{.5cm}&\hspace{.5cm}y_1,\ldots,y_n\sim N(\mu_y,\sigma_y^2)\\\iff\hspace{.5cm}\xbar \sim N(\mu_x,\frac{\sigma_x^2}{n_x})\hspace{.5cm}&\hspace{.5cm}\ybar \sim N(\mu_y,\frac{\sigma_y^2}{n}) \end{align*}$

Furthermore, with the equivariance assumption, we are assuming that $\sigma_x^2 = \sigma_y^2 = \sigma^2$ .

We know the difference of population means to be $\mu_y - \mu_x$ . As we don’t know either population mean, we approximate that difference with sample means. The variance of the difference of sample means is simply the sum of variance of individual sample means.

$\begin{equation*} \ybar - \xbar \sim N(\mu_y - \mu_x,\frac{\sigma^2}{n_x} + \frac{\sigma^2}{n_y}) \end{equation*}$

Since we see the variance of the difference, we can calculate the standard error very simply.

$\begin{equation*} Var(\ybar - \xbar) = \frac{\sigma^2}{n_x} + \frac{\sigma^2}{n_y} \iff SE(\ybar - \xbar) = \sqrt{\frac{\sigma^2}{n_x} + \frac{\sigma^2}{n_y}} \end{equation*}$

Furthermore, we can declare $n_y$ to be a function of $n_x$ . We might choose to sample more from one group over the other because the cost of sampling from that group is cheaper. In order to get the greatest value for the money, we’re better off sampling more from the cheaper group. We set $r$ to be the ratio of $\frac{n_y}{n_x}$ , such that $n_x = rn_y$ . Therefore, the standard error is calculated as:

$\begin{equation*} \sqrt{\frac{\sigma^2}{n_x} + \frac{\sigma^2}{rn_x}} = \frac{(r+1)\sigma^2}{rn_x} \end{equation*}$

Now that we have our standard error for the difference, we can proceed with calculating the sample size.

One Sided Test
In the one sided test, we are establishing whether or not a particular population’s mean is greater than the other.

$\begin{align*} H_0\hspace{1cm}\mu_y - \mu_x \leq 0\\ H_1\hspace{1cm}\mu_y - \mu_x > 0\\ \end{align*}$

Alternatively, if we were trying to establish the converse,

$\begin{align*} H_0\hspace{1cm}\mu_y - \mu_x \geq 0\\ H_1\hspace{1cm}\mu_y - \mu_x < 0\\ \end{align*}$

$\begin{align*} Z_{\beta} &= \frac{\mu_y - \mu_x}{\sqrt{\frac{(r+1)\sigma^2}{rn_x}}} - Z_{\alpha}\\ Z_{\beta} + Z_{\alpha} &= \frac{\mu_y - \mu_x}{\sqrt{\frac{(r+1)\sigma^2}{rn_x}}}\\ (Z_{\beta} + Z_{\alpha})^2 &= (\frac{\mu_y - \mu_x}{\sqrt{\frac{(r+1)\sigma^2}{rn_x}}})^2\\ (r+1)\sigma^2(Z_{\beta}+Z_{\alpha})^2 &= rn_x(\mu_y - \mu_x)^2\\ n_x &= \frac{(r+1)\sigma^2(Z_{\beta} + Z_{\alpha})^2}{r(\mu_y - \mu_x)^2}\\ n_x &= \frac{(r+1)}{r}\frac{\sigma^2(Z_{\beta} + Z_{\alpha})^2}{(\mu_y - \mu_x)^2} \end{align*}$

Two Sided Test

$\begin{align*} H_0:\hspace{1cm}\mu_y - \mu_x = 0\\ H_1:\hspace{1cm}\mu_y - \mu_x \neq 0\\ \end{align*}$

The only thing that changes for the two sided test is we use $Z_{\frac{\alpha}{2}}$ instead of $Z_{\alpha}$ . Therefore,

$\begin{equation*} n_x = \frac{(r+1)}{r}\frac{\sigma^2(Z_{\beta} + Z_{\frac{\alpha}{2}})^2}{(\mu_y - \mu_x)^2} \end{equation*}$

Additional Links