Sample size estimation is a critical step in research planning, including when you’re trying to detect differences in measures like Net Promoter Scores. Too small of a sample and you risk not being able to differentiate real differences from sampling error. Too large of a sample and you risk wasting resources—researchers’ and respondents’ time and,
Making decisions with data inevitably means working with statistics and one of its most common frameworks: Null Hypothesis Significance Testing (NHST). Hypothesis testing can be confusing (and controversial), so in an earlier article we introduced the core framework of statistical hypothesis testing in four steps: Define the null hypothesis (H0). This is the hypothesis that
In architecture, form follows function. In survey design, question format follows content. Earlier we described four classes of survey questions. These four classes are about the form, or format, of the question (e.g., open- vs. closed-ended). But before you can decide effectively on the format, you need to choose the content of the question and
Advice on rating scale construction is ubiquitous on the internet and in the halls of organizations worldwide. The problem is that much of the advice is based not on solid data but rather on conventional wisdom and what’s merely thought to work. Even published papers and books on survey design can present a perspective that
When we conduct a survey, we want the truth, even if we can’t handle it. But standing in the way of our dreams of efficiently collected data revealing the unvarnished truth about customers, prospects, and users are the four horsemen of survey errors. Even a well-thought-out survey will have to deal with the inevitable challenge
Sample size estimation is a critical step in research planning. It can also seem like a mysterious and at times controversial process. But sample size estimation, when done correctly, is based mostly on math, not magic. The challenge is that the math can get complex, so it becomes easier to defer to simple rules or
The Net Promoter Score (NPS) is a popular business metric used to track customer loyalty. It uses a single likelihood-to-recommend (LTR) question (“How likely is it that you will recommend our company to a friend or colleague?”) with 11 scale steps from 0 (Not at all likely) to 10 (Extremely likely). In NPS terminology, respondents
The Net Promoter Score (NPS) is a widely used metric, but it can be tricky to work with statistically. One of the first statistical steps we recommend that researchers take is to add confidence intervals around their metrics. Confidence intervals provide a good visualization of how precise estimates from samples are. They are particularly helpful
During the fall in the northern hemisphere, leaves change colors, birds fly south, and the temperature gets colder. Do the birds change the color of the leaves, and does their departure make the temperature colder? What if you gave participants two versions of a rating scale, with the first having responses ordered from strongly disagree
We typically recommend small sample sizes (5–10) for conducting iterative usability testing meant to find and fix problems (formative evaluations). For benchmark or comparative studies, where the focus is on detecting differences or estimating population parameters (summative evaluations), we recommend using larger sample sizes (20–100+). Usability testing can be used to uncover problems and assess the
