MeasuringU: All Articles

Seven Reasons Survey Questions Are Answered Incorrectly

By Jeff Sauro, PhD and Jim Lewis, PhD . April 6, 2021 . Tags: Questionnaires, Rating Scales, Rating Scale, Sample Size

Surveys are an essential method for collecting data. But like all research methods, surveys have their limitations. Unless the survey is administered by a facilitator, a respondent has only the survey's instructions, questions, and response options for guidance. Earlier we wrote about ways to improve the clarity of your questions and certain words to watch for when writing questions. But, despite your best efforts, inevitably

UX and Net Promoter Benchmarks of Food Delivery Services

By Jim Lewis, PhD and Jeff Sauro, PhD . March 30, 2021 . Tags: MUIQ, SUPR-Q, Net Promoter Score, Key Driver

The food delivery market has been growing significantly over the past five years. That growth exploded in the United States from $17 billion in 2018 to $26 billion in 2020 (partly due to COVID-19). This market is highly competitive and has ultra-thin profit margins. Technologies such as route optimization enable faster and cheaper delivery, but dissatisfied customers can easily switch providers. Service providers that don’t

Seven Reasons People Misinterpret Survey Questions

By Jeff Sauro, PhD and Jim Lewis, PhD . March 23, 2021 . Tags: Rating Scales, Satisfaction, Rating Scale

Like in all research methods, many things can go wrong in surveys, from problems with sampling to mistakes in analysis. To draw valid conclusions from your survey, you need accurate responses. But participants may provide inaccurate information. They could forget the answers to questions or just answer questions incorrectly. One common reason respondents answer survey questions incorrectly is that they misinterpret the question. Understanding the

Exploring Another Alternate Form for the UMUX-Lite Usefulness Item

By Jim Lewis, PhD and Jeff Sauro, PhD . March 16, 2021 . Tags: UMUX-lite, Rating Scale, Sample Size

When thinking about user experiences with websites or software, what is the difference between capabilities and functions? Is there any difference at all? In software engineering, a function is code that takes inputs, processes them, and produces outputs (such as a math function). The word capability doesn’t have a formal definition, but it most often appears in the phrase Capability Maturity Model, a formal model

Four Reasons Why Research Participants Forget

By Jeff Sauro, PhD and Jim Lewis, PhD . March 9, 2021 . Tags: Questionnaires, Surveys

Post-mortems and retrospective accounts are valuable for understanding what went well and what went poorly. This applies not only to investigations of critical events, such as crimes and plane accidents, but also to experiences with products and services. But the usefulness of people's recollections of events and experiences rests on the accuracy of their memories. (Measuring the perception of experiences might be just as valuable

Sample Size Estimation for NPS Confidence Intervals

By Jim Lewis, PhD and Jeff Sauro, PhD . March 2, 2021 . Tags: Net Promoter Score, NPS, Sample Size

Sample size estimation is a critical step in research planning. It can also seem like a mysterious and at times controversial process. But sample size estimation, when done correctly, is based mostly on math, not magic. The challenge is that the math can get complex, so it becomes easier to defer to simple rules or a one-size-fits-all number. Before 2016, there was no well-defined method

Nine Words to Watch for When Writing Survey Questions

By Jeff Sauro, PhD and Jim Lewis, PhD . February 23, 2021 . Tags: Rating Scales, Rating Scale, Surveys, Survey

In UX research, both studies and surveys contain a lot of questions. Getting those questions right can go a long way in improving the clarity and quality of the findings. For example, we’ve recently written about how to make survey questions clearer. And while there are many stories of how the change of a single word in a survey question can lead to different results

Evaluating NPS Significance Tests with Real-World Data

By Jim Lewis, PhD and Jeff Sauro, PhD . February 16, 2021 . Tags: Net Promoter Score, Rating Scale, NPS, Sample Size

The Net Promoter Score (NPS) is widely used by organizations. It’s often used to make high-stakes decisions on whether a brand, product, or service has improved or declined. Net Promoter Scores are often tracked on dashboards, and any changes (for better or worse) can have significant consequences: adding or removing features, redirecting budgets, even impacting employee bonuses. Random sampling error, however, can often explain many

Seven Ways to Make Survey Questions Clearer

By Jeff Sauro, PhD and Jim Lewis, PhD . February 9, 2021 . Tags: Questionnaires, Surveys, Survey

The first questionnaires appeared in the mid–18th century (e.g., the “Milles” questionnaire). Scientific surveys have been around for almost a hundred years. Consequently, there are many sources of advice on how to make surveys better. The heart of each survey is the questions asked of respondents. Writing good survey questions involves many of the principles of good writing, in addition to survey-specific guidelines and advice.

How to Statistically Compare Two Net Promoter Scores

By Jim Lewis, PhD and Jeff Sauro, PhD . February 2, 2021 . Tags: Statistical Significance, Net Promoter Score, Confidence, Confidence Intervals, NPS

When we wrote Quantifying the User Experience, we put confidence intervals before tests of statistical significance. We generally find fluency with confidence intervals to be easier to achieve and of more value than with formal hypothesis testing. We also teach confidence intervals in our workshops on statistical methods. Most people, even non-researchers, have been exposed to the concept of margins of error—political polls include them.