MeasuringU: All Articles

Rating Scale Best Practices: 8 Topics Examined

By Jeff Sauro, PhD and Jim Lewis, PhD . November 17, 2020 . Tags: Sensitivity, Emoji scale, Star Scale, sliders, LTR, UMUX-lite, Sample Size

Rating scales have been around for close to a century. It’s no wonder there are many questions about best practices and pitfalls to avoid. And like any topic that’s been around for that long, there are urban legends, partial truths, context-dependent findings, and just plain misconceptions about the “right” and “wrong” way to use and interpret rating scales. We’ve been researching and conducting our own

Are Sliders More Sensitive than Numeric Rating Scales?

By Jeff Sauro, PhD and Jim Lewis, PhD . November 10, 2020 . Tags: Sensitivity, slider, sliders, Reliability, Sample Size

Sliders are a type of visual analog scale that can be used with many online survey tools such as our MUIQ platform. The literature on their overall effectiveness is mixed (Roster et al., 2015). On the positive side, evidence indicates that sliders might be more engaging to respondents. On the negative side, evidence also indicates that sliders can be more cognitively and physically challenging than

Latin and Greco-Latin Experimental Designs for UX Research

By Jim Lewis, PhD and Jeff Sauro, PhD . November 3, 2020 . Tags: Latin Squares, Greco-Latin Squares, Research design, Questionnaires, Survey

During the fall in the northern hemisphere, leaves change colors, birds fly south, and the temperature gets colder. Do the birds change the color of the leaves, and does their departure make the temperature colder? What if you gave participants two versions of a rating scale, with the first having responses ordered from strongly disagree to strongly agree and the second reversing the order, and

Improving the Prediction of the Number of Usability Problems

By Jeff Sauro, PhD and Jim Lewis, PhD . October 27, 2020 . Tags: Usability Problems, Sample Size, Usability Testing

Paraphrasing the statistician George Box, all models are wrong, some are useful, and some can be improved. In a recent article, we reviewed the most common way of modeling problem discovery, which is based on a straightforward application of the cumulative binomial probability formula: P(x≥1) = 1 - (1-p)n. Well, it’s straightforward if you like playing around with these sorts of formulas like Jim and

Revisiting the Evidence for the Left-Side Bias in Rating Scales

By Jim Lewis, PhD and Jeff Sauro, PhD . October 20, 2020 . Tags: Bias, TAM, Questionnaires, Rating Scales

Are people more likely to select response options that are on the left side of a rating scale? About ten years ago, we provided a brief literature review of the published evidence, which suggested that this so-called left-side bias not only existed but also was detected almost 100 years ago in some of the earliest rating scales. Across the publications we reviewed, the effect size

Predicting the Number of Usability Problems Using Average Problem Occurrence

By Jeff Sauro, PhD and Jim Lewis, PhD . October 13, 2020 . Tags: Probability, Sample Size, Usability Testing

Finding and fixing problems in an interface is one of the fundamental priorities of a formative usability test. But how many users should you test with? And how many usability problems are there to be uncovered? These questions have been discussed and debated for decades. Early work on problem discovery suggested that the first few users will uncover most of the common problems. This isn’t—or

The UX of Vacation Rental Websites

By Jeff Sauro, PhD and Jim Lewis, PhD . October 6, 2020 . Tags: SUPR-Q, Net Promoter Score, Rating Scales, Sample Size

The COVID-19 pandemic has led to significant changes in how people have vacationed in 2020. To get away from it all without spending time in crowded places, vacationers have turned to vacation rental websites and have planned longer stays. For example, Airbnb recently reported a year-to-year doubling of long-term (>28 days) rentals and a shift from urban to rural stays. These changes make sense given

Simplifying the UMUX-Lite

By Jim Lewis, PhD and Jeff Sauro, PhD . September 29, 2020 . Tags: Measure, UMUX-lite, Measurement

It seems like every few years a new standardized UX measure comes along. Standardization of UX measurement is a good thing for researchers and practitioners. Having common methods and definitions helps with objectivity, generalization, economy, and professional communication. At MeasuringU, we pay a lot of attention to the continuing evolution of standardized UX measurement. The UMUX-Lite is a relatively recent measure that has been attracting

How to Code Errors in Unmoderated Studies

By Jeff Sauro, PhD and Jim Lewis, PhD . September 22, 2020 . Tags: coding, Errors, Reliability

Errors can provide a lot of diagnostic information about the root causes of UI problems and the impact such problems have on the user experience. The frequency of errors—even trivial ones—also provides a quantitative description of the performance of a task. The process of observing and coding errors is more time-consuming and dependent on researcher judgment than recording task completions or task times. Consequently, errors

Should You Use Negative Numbers in Rating Scales?

By Jim Lewis, PhD and Jeff Sauro, PhD . September 15, 2020 . Tags: negative scale, UMUX-lite

There are a lot of opinions about the best formats for agreement scales. Sometimes those opinions are strongly held and can lead to lengthy, heated discussions within research teams. When format differences affect measurement properties, those discussions may be time well spent, but when the formats don’t matter (or matter very little), the time is wasted. That’s why we have an ongoing research goal to