One of the more challenging things about learning math in general (and statistics in particular) is how the formulas, often with Greek symbols, translate to things we can see and experience.
The abstractness of these formulas often means we just have to take them at face value, believing that someone smarter than us made sure they work!
After teaching statistics for over two decades, we understand how challenging it can be to grasp the formulas for practical statistics such as confidence intervals and tests of significance. We’ve found it’s even harder to work backward from those formulas to compute sample sizes. It’s no wonder that people think it’s a sort of dark art and tend to rely on crude rules of thumb or overconfident online opinions.
Most people understand that small sample sizes limit accuracy of measurement and larger sample sizes increase accuracy. But because it’s not a linear relationship, it can be hard to understand how increasing your sample size helps to improve accuracy.
Interestingly, the relationship between sample sizes and precision follows a pattern we can observe in the physical world.
MeasuringU is based in Colorado, so many of us like spending time outdoors in the mountains camping under the stars (Figure 1). You may feel like you’ve escaped math during your time in nature, but instead, you’re surrounded by physical forces that have a close mathematical relationship to sample size effects—the inverse square law.
Figure 1: Even when camping you’re surrounded by examples of the inverse square law.
When camping and sitting next to a campfire you stay warm. As you sit farther away, you feel less heat. Did you know that sitting twice as far from a bonfire means you receive only one-fourth the heat intensity?
As you walk away from the campfire into the darkness, the light emitted from your flashlight gets dimmer to your friends.
Moving from 10 feet to 20 feet from a flashlight cuts the brightness (more generally, intensity) by one-fourth because the surface area the light covers quadruples (Figure 2).
Figure 2: Brightness (intensity, I) as a function of the distance (r) from an energy source with a set amount of power (P). Because r2 is in the denominator, any increase in r reduces I by the square of the increase.
Walking away from the gathering, the strumming sound of the guitar gets quieter. When the distance doubles, the intensity of the sound from the guitar drops to one-fourth.
Away from the city lights, you’re really able to see the stars and the brightness of the moon, which looks like it’s just hanging in the sky.
The moon is almost 250,000 miles away but orbits Earth because of its gravity. As objects get farther from Earth, the gravity decreases. In fact, we see the inverse square relationship again.
As distance (r) doubles, the gravitational force (F) decreases to one-fourth of its original strength (Figure 3).
Figure 3: Gravitational force (F) as a function of the distance (r) between two masses. Because r2 is in the denominator, any increase in r reduces F by the square of the increase.
The camping scene illustrates the inverse square law in multiple ways. This same law applies to determining sample size requirements.
To be twice as precise in your estimates, you need to roughly quadruple your sample size. So, what do we mean by precision?
When computing the sample sizes required to estimate values to a specified precision, precision refers to the width of the margin of error for a confidence interval (d). Precision can also be used to refer to the size of the smallest important difference (often referred to as the target effect size or critical difference) when making comparisons.
Figure 4 and Table 1 show the relationship between sample size and precision for estimation (width of the confidence interval around the estimated value) and comparison (estimate of the confidence interval around the difference between two values), with d analogous to r and sample size analogous to the dispersion of intensity in Figure 2 (the larger the sample size, the closer the estimates are to the true population represented by the flashlight).
Figure 4: Quadrupling the sample size cuts measurement error in half, in other words, doubling the precision.
| Sample Size Each Group | Estimation (Margin of Error) | Comparison (Critical Difference) |
|---|---|---|
| 25 | 20% | 40% |
| 100 | 10% | 20% |
| 400 | 5% | 10% |
| 1600 | 2.5% | 5% |
Table 1: Relationship among sample sizes, margins of error, and critical differences.
Sample Size for Precision
Some sample size formulas are very complex, but others are fairly simple. One of the simplest is the formula for estimating a mean when the sample size is fairly large (n > 30), shown in Figure 4 (z and s are effectively constants in this process in which z is the z-score for the desired confidence level and s is the estimated standard deviation of the mean). Using the estimation formula in Figure 4, the steps to demonstrate the inverse squared relationship between n and d are:
n = z2s2/d2
Multiply d by ½.
z2s2/(½d)2
Square the ½.
z2s2/¼d2
Multiply the numerator and denominator by 4.
4z2s2/¼(4)d2
This gives you:
4z2s2/d2
Because n = z2s2/d2, this is equal to 4n.
To double the estimation precision (i.e., cut the margin of error in half), you need to quadruple the sample size (n).
Sample Size for Comparing
Because the formulas for estimation and comparison in Figure 4 are the same except for the extra 2 in the numerator for comparison (related to the computations involving two independent sets of data rather than just one), the steps that demonstrate why cutting the critical difference in half requires doubling the sample size are the same (just replace z2 with 2z2). Note that this estimates the sample size for one group, but if you’re making a comparison, there are at least two groups, so the total sample size requirement is 2n when there are two groups, 3n when there are three groups, and so on.
When you need to make comparisons (such as which product has a higher SUS score, are the completion rates different), the precision estimate is the minimum difference you want to detect (i.e., will be statistically significant, the critical difference). The difference is usually expressed as an unstandardized effect size. For example, a 20% difference in completion rates is an unstandardized effect size of 20%.
To double the comparison precision (i.e., reduce the critical difference by half), you need to quadruple the sample size.
Summary
Although the mathematics behind sample size calculations can look complicated (and they sometimes are), the concept is something you can see and experience. Like many natural phenomena, sample size and precision have an inverse square relationship. For estimation or comparison:
To double the precision, you need to quadruple the sample size.