Unless you plan to survey every member of your target population, you’ll need to work with a sample.

But even in cases where you can survey everyone, you might not want to because of survey fatigue and costs. What’s more, most populations are fluid. There may be only a hundred users of a financial product this month, but what about new users who join next month or next year?

Sampling a portion of your population is by far the most common approach when conducting surveys—especially because, unlike the U.S. census, answering a survey is not required by law.

When sampling, it’s good to have some idea about the different types of strategies covered in this article and in Surveying the User Experience.

How to Think about Sampling without the Math

We provide extensive sample size tables for surveys on our website and in Chapter 17 of Surveying the User Experience. It can get a bit daunting to know which tables to use, but sample size estimation is only one step in the sample planning process, and it’s not necessarily the first one.

It’s also important to understand how sampling can provide a good picture of a population even if you can’t measure all or even most of its members. And that means we need to think about soup—and who doesn’t like a good soup?

Think of survey sampling like tasting a sample from a pot of soup. You want to be sure the soup tastes right and doesn’t need more salt, or maybe it needs to cook longer. But you’re not going to drink down the whole pot. Instead, you’ll sample a small amount, usually with a spoon (Figure 1). And because you’re taking only a small amount, there will always be some uncertainty about the taste of the parts you aren’t sampling.

Figure 1: Regardless of the size of the pot, after thorough stirring, you can sample the soup with one tablespoon.

Assuming the soup is reasonably stirred, the size of the pot doesn’t matter. A sample, even a small one, is a surprisingly reliable way of assessing the entire pot of soup (or population). The same principle applies to survey sampling, which brings us to some common questions.

Answers to Key Questions about Survey Sampling

What’s the first step in deciding on a sampling strategy?

There are different approaches to sampling (not the soup-making kind) and different ways to categorize them. At a high level, there are two helpful distinctions: probability sampling and nonprobability sampling, with some varieties of each.

If you want all members of a population to have a chance of participating in the survey, you’ll need a probability sample.

What are the different ways to collect a probability sample?

While we’ve encountered several ways to delineate probability samples, we use four categories: simple, stratified, proportional, and census-matching.

• In simple random sampling, each person in the population has been identified for random selection.
• Stratified random sampling is similar, but the population is subdivided into groups (e.g., new versus existing users of a product) to ensure adequate coverage of each group in the sample.
• Proportional sampling is similar to stratified sampling with subdivisions of the population defined by external benchmarks (e.g., what an enterprise knows about the percentages of customers in the US and the EU).
• Census-matching sampling is a special type of proportional sampling where the external benchmarks are based on a government census (usually age and gender).

Oh, and you might have heard of the term cluster sampling—not the sort of cluster when things go really wrong, but one that’s confused with stratified sampling. Cluster sampling involves sampling all people of a certain group (e.g., all patients from a sample of urban hospitals compared with all patients from a sample of rural hospitals), but it’s less common than stratified sampling in applied survey research.

So why not just use a simple random sample or stratified sample every time? Well, it turns out it can be quite difficult to ensure that every person in your population has a chance, much less a random one, to participate in your survey. The first logistical issue is determining whether you even have the ability to contact the potential participants. If you have a database of customer emails, are they all working? Do people really check them? These very real logistical hurdles mean you’ll often need to consider a nonprobability sample.

What are the different types of nonprobability samples?

When you can’t use a probability sample, you’ll use a nonprobability sample. Two of the most common nonprobability samples are convenience and snowball.

• In a convenience sample, you ask anyone you can get to take the survey. You can send emails to friends and family, post on social media, or use online panels.
• In snowball sampling, you start with a group of known participants who fit your profile, then encourage them to not only take the survey but also invite others to take the survey. This strategy is useful for special profiles that are difficult to recruit (such as TikTok accounts with 100,000+ followers).

We recommend snowball sampling as a method of last resort. The reason is there’s a clear dependence between respondents—one person knows the person they invite (and may share their sentiments). While this helps increase your sample size, it may provide a misleading picture of your population. This was the technique used in the famous Kinsey Reports on human sexuality, which were criticized for their generalizability problems because of the dependence between respondents.

Is it OK to use a nonprobability sample?

If you don’t care about budget or time, and your population is relatively large and everyone has a chance of participating, then going with a probability sample makes sense.

In applied research, however, we often start with lofty aspirations about the type of sampling strategy, but the reality of budget, time, and participant constraints means probability samples are not viable options (Figure 2).

Figure 2: Stages of sampling decisions in the real world (simple or stratified random sampling, then proportional sampling, then convenience sampling).

Does that mean nonprobability samples are OK? The short answer is “probably” (sorry, couldn’t help that one). Most UX and CX decisions need good-enough input, not perfect input. In practice, some unrepresentativeness is common, and disproportionate samples happen. When you have these concerns, however, be careful about overgeneralizing your findings. When the stakes are high and you need precise generalization, investing in a probability sample is best if you can, and when you can’t, be aware of potential unrepresentativeness.

Summary

Table 1 summarizes the pros and cons of the sampling methods covered in this article.

Looking to Learn More?

We’ve written more on sampling strategies and the entire survey creation process, from writing questions and selecting response options to collecting and analyzing data. You can find plenty more in our book, Surveying the User Experience. We also have a companion course that follows the book on MeasuringUniversity.com.