One box, two box, red box, blue box … Box scoring isn’t just something they do in baseball. Response options for rating scale data are often referred to as boxes because, historically, paper-administered surveys displayed rating scales as a series of boxes to check, like the one in Figure 1. Figure 1: Illustration of “boxes”
Few things seem to elicit more opinions, exaggerations, and accusations than rating scale response options. From the “right” number of points, the use (or not) of labels, and the presentation order (to name a few), it seems we all have thoughts on what to do (or not to do) regarding conventions and “rules” when selecting
The standard deviation is the most common way of measuring variability or “dispersion” in data. The more the data is dispersed, the more measures such as the mean will fluctuate from sample to sample. That means higher variability (higher standard deviations) requires larger sample sizes. But exactly how much do standard deviations—whether large or small—impact
Are customers more satisfied this quarter than last quarter? Do users trust the brand less this year than last year? Did the product changes result in more customers renewing their subscriptions? When UX researchers want to measure attitudes and intentions, they often ask respondents to complete multipoint rating scale items, which are then compared with
“Don’t include numbers when reporting the results of small-sample research studies!” “If you must, definitely don’t use percentages!” “And of course, don’t even think about using statistics!” We regularly hear variations of this advice from well-intentioned researchers, often senior ones. In 2005, we encountered this debate among UX professionals when we participated in a workshop
Sample size estimation is an important part of study planning. If the sample size is too small, the study will be underpowered, meaning it will be incapable of detecting sufficiently small differences as statistically significant. If the sample size is too large, the study will be inefficient and cost more than necessary. A critical component
Sample size estimation is a critical step in research planning, including when you’re trying to detect differences in measures like Net Promoter Scores. Too small of a sample and you risk not being able to differentiate real differences from sampling error. Too large of a sample and you risk wasting resources—researchers’ and respondents’ time and,
Making decisions with data inevitably means working with statistics and one of its most common frameworks: Null Hypothesis Significance Testing (NHST). Hypothesis testing can be confusing (and controversial), so in an earlier article we introduced the core framework of statistical hypothesis testing in four steps: Define the null hypothesis (H0). This is the hypothesis that
In architecture, form follows function. In survey design, question format follows content. Earlier we described four classes of survey questions. These four classes are about the form, or format, of the question (e.g., open- vs. closed-ended). But before you can decide effectively on the format, you need to choose the content of the question and
Advice on rating scale construction is ubiquitous on the internet and in the halls of organizations worldwide. The problem is that much of the advice is based not on solid data but rather on conventional wisdom and what’s merely thought to work. Even published papers and books on survey design can present a perspective that
