“That confirms what I expected.” The same data, two different conclusions. A 90% completion rate from 20 participants on a usability test of a checkout flow. Is that completion rate better than the historical average of 78%? One researcher says yes, definitely. Another says no, it’s in line with the historical average. Both are using
“Follow the data. Update your beliefs.” We like the idea of applying iterative Bayesian thinking to how we test hypotheses and conduct UX research. The idea is simple, but modern Bayesian math can be opaque and hard to understand. We have questions about how well Bayesian analysis works relative to frequentist analysis. We are also
Since its development in the 1980s, System Usability Scale (SUS) score analysis has focused on estimating and comparing means. The mean is one type of average. Would it be more appropriate to estimate and compare SUS medians instead of means? To investigate this question, we analyzed and compared SUS means and medians collected from over
In an earlier article, we demonstrated how it’s completely permissible from a statistical perspective to report numbers when studies have very small sample sizes (fewer than ten people). When you use numbers, you can present them as raw numbers, fractions, or percentages. You can present them in a report, on a train, or on a
“Don’t include numbers when reporting the results of small-sample research studies!” “If you must, definitely don’t use percentages!” “And of course, don’t even think about using statistics!” We regularly hear variations of this advice from well-intentioned researchers, often senior ones. In 2005, we encountered this debate among UX professionals when we participated in a workshop
Proportional data is common in both UX research and the larger scientific literature. You can use proportions to help make data-driven decisions just about anywhere: Which design converts more? Which product is preferred? Does the new interface have a higher completion rate? What proportion of users had a problem with registering? Consequently, you’ll likely want to
The two-sample t-test is one of the most widely used statistical tests, assessing whether mean differences between two samples are statistically significant. It can be used to compare two samples of many UX metrics, such as SUS scores, SEQ scores, and task times. The t-test, like most statistical tests, has certain requirements (assumptions) for its
Understanding who your users are and what they think about an experience is an essential step for measuring and improving the user experience. Part of understanding your users is understanding how they are similar and different with respect to demographics, psychographics, and behaviors. These groupings are often called clusters or segments to refer to the
Researchers rely heavily on sampling. It’s rarely possible, or even makes sense, to measure every single person of a population (all customers, all prospects, all homeowners, etc.). But when you use a sample, the average value you observe (e.g. a completion rate, average satisfaction) differs from the actual population average. Consequently, the differences between designs
You can’t see customer satisfaction. You can’t see usability. There isn’t a thermometer that directly measures someone’s intelligence. While we can talk about satisfied customers, usable products, or smart people, there isn’t a direct way to measure these abstract concepts. And clearly these concepts vary. We’ve all had experiences that left us feeling unsatisfied or
