Statistical Hypothesis Testing: What Can Go Wrong?

Making decisions with data inevitably means working with statistics and one of its most common frameworks: Null Hypothesis Significance Testing (NHST). Hypothesis testing can be confusing (and controversial), so in an earlier article we introduced the core framework of statistical hypothesis testing in four steps: Define the null hypothesis (H0). This is the hypothesis that

Read More »

Classifying Survey Questions into Four Content Types

In architecture, form follows function. In survey design, question format follows content. Earlier we described four classes of survey questions. These four classes are about the form, or format, of the question (e.g., open- vs. closed-ended). But before you can decide effectively on the format, you need to choose the content of the question and

Read More »

Do Too Many Response Options Confuse People?

Advice on rating scale construction is ubiquitous on the internet and in the halls of organizations worldwide. The problem is that much of the advice is based not on solid data but rather on conventional wisdom and what’s merely thought to work. Even published papers and books on survey design can present a perspective that

Read More »

Four Types of Potential Survey Errors

When we conduct a survey, we want the truth, even if we can’t handle it. But standing in the way of our dreams of efficiently collected data revealing the unvarnished truth about customers, prospects, and users are the four horsemen of survey errors. Even a well-thought-out survey will have to deal with the inevitable challenge

Read More »

Sample Size Estimation for NPS Confidence Intervals

Sample size estimation is a critical step in research planning. It can also seem like a mysterious and at times controversial process. But sample size estimation, when done correctly, is based mostly on math, not magic. The challenge is that the math can get complex, so it becomes easier to defer to simple rules or

Read More »

Evaluating NPS Confidence Intervals with Real-World Data

The Net Promoter Score (NPS) is a popular business metric used to track customer loyalty. It uses a single likelihood-to-recommend (LTR) question (“How likely is it that you will recommend our company to a friend or colleague?”) with 11 scale steps from 0 (Not at all likely) to 10 (Extremely likely). In NPS terminology, respondents

Read More »

Confidence Intervals for Net Promoter Scores

The Net Promoter Score (NPS) is a widely used metric, but it can be tricky to work with statistically. One of the first statistical steps we recommend that researchers take is to add confidence intervals around their metrics. Confidence intervals provide a good visualization of how precise estimates from samples are. They are particularly helpful

Read More »

Latin and Greco-Latin Experimental Designs for UX Research

During the fall in the northern hemisphere, leaves change colors, birds fly south, and the temperature gets colder. Do the birds change the color of the leaves, and does their departure make the temperature colder? What if you gave participants two versions of a rating scale, with the first having responses ordered from strongly disagree

Read More »

What Do You Gain from Larger-Sample Usability Tests?

We typically recommend small sample sizes (5–10) for conducting iterative usability testing meant to find and fix problems (formative evaluations). For benchmark or comparative studies, where the focus is on detecting differences or estimating population parameters (summative evaluations), we recommend using larger sample sizes (20–100+). Usability testing can be used to uncover problems and assess the

Read More »

Sample Size in Usability Studies: How Well Does the Math Match Reality?

We’ve written extensively about how to determine the right sample size for UX studies. There isn’t one sample size that will work for all studies. The optimal sample size is based on the type of study, which can be classified into three groups: Comparison studies: Comparing metrics for statistical differences Standalone studies: Estimates a population

Read More »
0
    0
    Your Cart
    Your cart is emptyReturn to Shop
    Scroll to Top