But knowing how many participants you should use is not a simple question.
What’s particularly difficult about learning how to compute the right sample size for a study is that books and articles can get overly technical; it’s hard to know whether the advice is relevant to an applied research setting. In general, it can be hard to know where to start.
Here are five steps to help you start computing the right sample size for your study.
1. The Sample Size Matters
The sample size you use for your research does matter. You may hear people dismiss the whole question of what sample size you need as irrelevant. It is very relevant. This is an unfortunate consequence where I believe people get confusing and conflicting advice.
When you can’t measure all your users, you have to deal with the very real consequences of sampling error. The number of participants you test has an impact on how generalizable your findings are. This applies to both qualitative and quantitative research as well as generative and evaluative research.
2. One Size Does NOT Fit All
There’s not a one-size-fits-all answer. The sample size you need is not always 5, 30, 100, or 1,000.
Instead, the sample size you need depends on the goals and type of study. In general, most UX research falls into one of three study types, each with a different sample size computation.
- Discovery: Finding problems, insights, behaviors, challenges, and incidents from observation. The sample size you need is a function of how common a problem or behavior is. This is where the famous five users recommendation comes from. Know when five is and isn’t enough in discovery studies.
- Parameter Estimation: When you estimate the completion rate, likelihood to recommend, SUS score, or a sentiment of ALL your customers from a sample, it’s called parameter estimation. This is often what’s done in a survey or stand-alone usability benchmark. The sample size is a function of how narrow you want your margin of error to be.
- Comparison: Making comparisons, such as between designs, customers types, or competitors, requires you to account for two samples instead of one as well as the hypothesized difference between the samples. The size of the difference between samples is usually the biggest factor that determines sample sizes. Comparative studies often require the largest sample sizes to overcome the effects of sampling error.
3. Representativeness Is Different than Sample Size
The representativeness of a sample of participants is different than the size of the sample. Having responses from a lot of the wrong people doesn’t make a lot of sense, so you want it to represent the population from which you are making inferences. This is representativeness.
This concept often gets confused with the sample size. People will often ask whether a sample size of 20, 30, or some amount is representative. Knowing the size alone is not enough to answer that question. For a generally homogenous group of customers, a sample size of 2 can be representative, as can a sample size of 200. The difference is in the level of precision. Higher sample sizes afford you with more precision about the information you obtain.
4. The Magnitude of the Effect Often Matters Most
When making comparisons, the size of the difference matters a lot. Smaller differences require larger sample sizes. Large differences between whatever you are comparing means a smaller sample size will suffice. When you want to detect differences, such as completion rates, two groups in a survey, or times to register on an old versus new app, you first need to think if such a difference does exist, whether it’s likely to be small or large.
This concept also applies to problem discovery studies. When discovering problems, the more users a problem impacts, the smaller the sample size needed to uncover these problems. Conversely, problems that affect a small number of users are very hard to detect in small sample studies and require larger sample sizes. The same applies to observing behaviors in a contextual inquiry.
5. Practice Finding the Sample Size for Common UX Research Studies (And the Theory Too)
Understanding a concept and putting it to use helps make it understandable and actionable.
To give you some practice, here are eight examples for finding the right sample size in a formative usability test, contextual inquiry, surveys, benchmarks, and comparative studies.
And to better understand the theory, Jim and I have formulas and sources for how we come up with the methods for finding the right sample size for each study in the 2nd Edition of Quantifying the User Experience, Chapters 6, 7, and 10.
The math behind sample size calculations is only half the equation. The other half is the very real constraints of time and money and the ability to obtain enough qualified participants—a topic for another article.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Fall 2020: Delivered Online|