MeasuringU: All Articles

Should You Use Negative Numbers in Rating Scales?

By Jim Lewis, PhD and Jeff Sauro, PhD . September 15, 2020 . Tags: negative scale, UMUX-lite

There are a lot of opinions about the best formats for agreement scales. Sometimes those opinions are strongly held and can lead to lengthy, heated discussions within research teams. When format differences affect measurement properties, those discussions may be time well spent, but when the formats don’t matter (or matter very little), the time is wasted. That’s why we have an ongoing research goal to

Are Face Emoji Ratings Better than Numbered Scales?

By Jim Lewis, PhD and Jeff Sauro, PhD . September 8, 2020 . Tags: graphic scale, Emoji scale, UMUX-lite

Somewhat agree, very satisfied, extremely likely. The labels used on the points of rating scales can affect responses in often unpredictable ways. What’s more, certain terms can get lost in translation when writing surveys for international usage. Some terms may have subtly different meanings, possibly making cross-cultural comparisons problematic. While numbers are universally understood and don’t need translation, does "5" on a seven-point scale properly

What Do You Gain from Larger-Sample Usability Tests?

By Jeff Sauro, PhD and Jim Lewis, PhD . September 1, 2020 . Tags: Sample Sizes, Formative, Summative, SEQ, Rating Scale, Sample Size

We typically recommend small sample sizes (5–10) for conducting iterative usability testing meant to find and fix problems (formative evaluations). For benchmark or comparative studies, where the focus is on detecting differences or estimating population parameters (summative evaluations), we recommend using larger sample sizes (20–100+). Usability testing can be used to uncover problems and assess the experience. Many usability tests will play both roles simultaneously, formative

How to Handle Bad Data

By Jeff Sauro, PhD and Jim Lewis, PhD . August 25, 2020 . Tags: Questionnaires, Rating Scale, Market Research, Data

Decisions should be driven (or at least informed) by data. Raw data is turned into information by ensuring that it is accurate and has been put into a context that promotes good decision-making. The pandemic has brought a plethora of COVID-related data dashboards, which are meant to provide information that helps the public and public officials make better decisions. With the pressure to report data

The Evolution of the Mean Opinion Scale: From MOS-R to MOS-X2

By Jim Lewis, PhD and Jeff Sauro, PhD . August 18, 2020 . Tags: Mean Opinion Scale, Voice Interaction, Market Research, Customer Experience

The Mean Opinion Scale (MOS) is a standardized questionnaire used to assess synthetic speech. The quality of synthetic speech strongly affects the user experience of working with conversational systems, with listeners making rapid and unconscious judgments of the speaker’s personality, so it’s important to have standardized methods for its assessment. In an earlier article, we described how the MOS was originally developed to assess speech

The UX of Video Streaming Entertainment Websites & Apps

By Jeff Sauro, PhD and Jim Lewis, PhD . August 11, 2020 . Tags: SUPR-Q, Key Driver, NPS, Market Research, User Research

We’ve all spent a lot of time at home this year. The pandemic has made already-popular video streaming services seem essential. The popularity makes sense given the relatively inexpensive subscription fees, the lack of long-term contracts, and the many channels of access (through websites, mobile apps, smart TVs), and there is a LOT of content (albeit distributed across different services). But no matter how good

What Is the Mean Opinion Scale (MOS)?

By Jim Lewis, PhD and Jeff Sauro, PhD . August 4, 2020 . Tags: MOS, MOS-R, Voice Interaction

The quality of the electronic transmission of the human voice has come a long way since Bell summoned Watson. But even with all the advancement in technology, “Can you hear me now?” is still part of our modern lexicon. Voice—both human and digital—plays an increasingly important role in interactions with our devices. But before you can interpret what is being said you need to be

Are Within-Subjects Designs Invalid?

By Jim Lewis, PhD and Jeff Sauro, PhD . July 28, 2020 . Tags: Anchoring, Polarization, Carryover, Within-subjects

One of the best ways to make UX metrics more meaningful is to have a comparison. For example, when conducting a UX benchmark study we often recommend adding at least one competing product (especially if it’s the first benchmark). Comparable interfaces help stakeholders easily interpret context-sensitive task metrics, such as completion rates and task time. While attitudinal measures such as the SEQ and SUS have

Are Star Ratings Better Than Numbered Scales?

By Jim Lewis, PhD and Jeff Sauro, PhD . July 21, 2020 . Tags: Linear Numeric Scale, Star Scale, UMUX-lite, Satisfaction

Five-star reviews. Whether you're rating a product on Amazon, a dining experience on Yelp, or a mobile app in the App or Play Store, you can see that the five-star rating system is quite ubiquitous. Does the familiarity of stars offer a better rating system than traditional numbered scales? We recently reported a comparison between standard five-point linear numeric scales with 0–100-point slider scales made with UMUX-Lite

Are Sliders Better Than Numbered Scales?

By Jim Lewis, PhD and Jeff Sauro, PhD . July 14, 2020 . Tags: slider, Visual Analog Scale, sliders, UMUX-lite

There are many ways to format rating scales. Recently we have explored Labeling neutral points Labeling all or some response options Altering the number of response options Comparing agreement vs. item-specific endpoint labels Each of these formatting decisions has a variety of opinions and research, both pro and con, in the scientific literature at large. Our controlled studies on these topics in the context of