MeasuringU

Revisiting the Evidence for the Left-Side Bias in Rating Scales

By Jim Lewis, PhD and Jeff Sauro, PhD . October 20, 2020 . Tags: Bias, TAM, Questionnaires, Rating Scales

Are people more likely to select response options that are on the left side of a rating scale? About ten years ago, we provided a brief literature review of the published evidence, which suggested that this so-called left-side bias not only existed but also was detected almost 100 years ago in some of the earliest rating scales. Across the publications we reviewed, the effect size

The UX of Vacation Rental Websites

By Jeff Sauro, PhD and Jim Lewis, PhD . October 6, 2020 . Tags: SUPR-Q, Net Promoter Score, Rating Scales, Sample Size

The COVID-19 pandemic has led to significant changes in how people have vacationed in 2020. To get away from it all without spending time in crowded places, vacationers have turned to vacation rental websites and have planned longer stays. For example, Airbnb recently reported a year-to-year doubling of long-term (>28 days) rentals and a shift from urban to rural stays. These changes make sense given

How to Convert Between Five- and Seven-Point Scales

By Jim Lewis, PhD and Jeff Sauro, PhD . June 16, 2020 . Tags: Rating Scales, Rating Scale, Surveys, Survey

There is plenty of debate about the best way to quantify attitudes and experiences with rating scales. And among those debates, perhaps the most popular question is the “right” number of response options to use for rating scales. For example, is an eleven-point scale too difficult for people to understand? Is a three-point scale insufficient for capturing extreme attitudes? Most research on this topic shows

Confounded Experimental Designs, Part 1: Incomplete Factorial Designs

By Jim Lewis, PhD and Jeff Sauro, PhD . June 2, 2020 . Tags: Variables, Research, Rating Scales, Surveys

UX research and UX measurement can be seen as an extension of experimental design. At the heart of experimental design lie variables. Earlier we wrote about different kinds of variables. In short, dependent variables are what you get (outcomes), independent variables are what you set, and extraneous variables are what you can’t forget (to account for). When you measure a user experience using metrics—for example,

Three Questionnaires for Measuring Voice Interaction Experiences

By Jim Lewis, PhD and Jeff Sauro, PhD . March 3, 2020 . Tags: Voice Interaction, Rating Scales, Customer Experience, Questionnaire

Human: Computer, can you recognize speech? Computer: I think you said, can you wreck a nice beach? Both the quality of synthesized speech and the capability of communicating with a computer using your voice have come a long way since the debut of this technology in the 1970s. One of the most famous synthetic voices was the one used by Stephen Hawking. Some of its

Comparing Fully vs. Partially Labeled Five- and Seven-Point Scales

By Jeff Sauro, PhD and Jim Lewis, PhD . January 21, 2020 . Tags: Rating Scales, Satisfaction, UX

In our earlier article, Jim Lewis and I reviewed the published literature on labeling scales. Despite some recommendations and “best practice” wisdom, we didn’t find that fully labeled scales were measurably superior to partially labeled scales across the 17 published studies that we read. In reviewing the studies in more detail, we found many had confounding effects when comparing between full labeling and partial labeling—meaning

Should All Scale Points Be Labeled?

By Jeff Sauro, PhD and Jim Lewis, PhD . January 7, 2020 . Tags: Rating Scales, Rating Scale, Surveys

To have a reliable and valid response scale, people need to understand what they’re responding to. A poorly worded question, an unclear response item, or an inadequate response scale can create additional error in measurement. Worse yet, the error may be systematic rather than random, so it would result in unmodeled bias rather than just increased measurement variability. Rating scales have many forms, with variations

Five Scales to Measure Customer Satisfaction

By Jeff Sauro, PhD . October 1, 2019 . Tags: Rating Scales, Satisfaction

While customer satisfaction may be thought of as one concept, there’s isn’t a single “official” way to measure it. By one estimate there are more than 40 instances of different customer satisfaction scales described in the published literature. That, in part, is a consequence of how common satisfaction is as a measure. Satisfaction is measured on more than just brands, products, and features. It’s used to

Effects of Labeling the Neutral Response in the NPS

By Jeff Sauro, PhD . March 5, 2019 . Tags: Net Promoter Score, Rating Scales, Survey

Should you label all points on a scale? Should you include a neutral point? What about labeling neutral points? How does that affect how people respond? These are common questions when using rating scales and they’ve also been asked about the Net Promoter Score: What are the effects of having a neutral label on the 11-point Likelihood to Recommend (LTR) item used to compute the

Practical Tips for Running a PURE Evaluation

By Jeff Sauro, PhD . August 21, 2018 . Tags: PURE, Rating Scales, Usability

In an earlier article, I described the PURE methodology. PURE stands for Practical Usability Rating by Experts. Evaluators familiar with UX principles and heuristics decompose tasks into small steps and rate each step based on a pre-defined scoring system (called a rubric), as shown in Figure 1. [table id=30 /] Figure 1: Scoring rubric for PURE. The PURE method is analytic. It’s not based on

Rating Scales

by Jim Lewis, PhD and Jeff Sauro, PhD | October 20, 2020

Browse Content by Topic

Revisiting the Evidence for the Left-Side Bias in Rating Scales

By Jim Lewis, PhD and Jeff Sauro, PhD . October 20, 2020 . Tags: Bias, TAM, Questionnaires, Rating Scales

The UX of Vacation Rental Websites

By Jeff Sauro, PhD and Jim Lewis, PhD . October 6, 2020 . Tags: SUPR-Q, Net Promoter Score, Rating Scales, Sample Size

How to Convert Between Five- and Seven-Point Scales

By Jim Lewis, PhD and Jeff Sauro, PhD . June 16, 2020 . Tags: Rating Scales, Rating Scale, Surveys, Survey

Confounded Experimental Designs, Part 1: Incomplete Factorial Designs

By Jim Lewis, PhD and Jeff Sauro, PhD . June 2, 2020 . Tags: Variables, Research, Rating Scales, Surveys

Three Questionnaires for Measuring Voice Interaction Experiences

By Jim Lewis, PhD and Jeff Sauro, PhD . March 3, 2020 . Tags: Voice Interaction, Rating Scales, Customer Experience, Questionnaire

Comparing Fully vs. Partially Labeled Five- and Seven-Point Scales

By Jeff Sauro, PhD and Jim Lewis, PhD . January 21, 2020 . Tags: Rating Scales, Satisfaction, UX

Should All Scale Points Be Labeled?

By Jeff Sauro, PhD and Jim Lewis, PhD . January 7, 2020 . Tags: Rating Scales, Rating Scale, Surveys

Five Scales to Measure Customer Satisfaction

By Jeff Sauro, PhD . October 1, 2019 . Tags: Rating Scales, Satisfaction

Effects of Labeling the Neutral Response in the NPS

By Jeff Sauro, PhD . March 5, 2019 . Tags: Net Promoter Score, Rating Scales, Survey

Practical Tips for Running a PURE Evaluation

By Jeff Sauro, PhD . August 21, 2018 . Tags: PURE, Rating Scales, Usability