It was another busy year at MeasuringU.

We posted 50 new articles, added new features to MUIQ—our UX testing platform, hosted our 6th UX Bootcamp, and released the book, Benchmarking the User Experience.

We also moved into a bigger new space in Denver’s Cherry Creek neighborhood. It’s three times the size of our old space with state-of-the-art labs and we hosted UX Book Club this year.

Over 1.2 million people read our articles and reports or used our calculators in 2018; thank you. A lot of work goes into writing these as we include original research (often taking months to complete).

The topics we covered include: usability testing, UX metrics (SUS and NPS), methods, and UX industry reports. Here’s a summary of the articles I wrote in 2018 with takeaways from each.

Usability Testing

Usability testing remains a popular evaluation method. Here’s what we covered:

  • How much does a usability test cost? 10K to 100K+ based on licensing technology platform, participant recruitment/honorariums, and professional services costs such as facilitation and analysis time.
  • What you need to conduct a moderated usability test. A quiet space and some software at a minimum. It can get complicated fast but efforts should be focused on creating better task scenarios and finding the right participants.
  • How to build a dedicated usability lab. Our tips on everything from one-way mirrors, audio, cameras, and software based on our experience building labs this year.
  • 8 ways to minimize no shows in UX research. We often have dozens of participants in our labs each week at MeasuringU. Here’s what we do to make sure participants show up.
  • How large is the evaluator effect in usability testing? With evaluators watching the same participants in the same study, the average any-2 agreement is 59% (ranging from 28% to 75%).
  • Do novices or experts uncover more usability issues? Novices likely will uncover more issues than experts in a usability test, but not always (and it’s not always clear what makes a user a novice or an expert).

System Usability Scale (SUS)

The SUS remains a widely used questionnaire to measure perceived ease of use. We added to the large body of research, including:

  • Can you use a single item to predict SUS scores? Surprisingly we found that one item—The system is easy to use—predicts a SUS score with ~90% accuracy (when aggregated at the product level).
  • 5 ways to interpret a SUS score. Grades, percentiles, acceptability, adjectives, and NPS designations all help add more meaning to raw SUS scores.
  • Interpreting single items from the SUS. We provided benchmarks for the ten individual SUS items.

Net Promoter Score

The Net Promoter Score may be overhyped but it’s wrong to dismiss or ignore this metric. We dug deep into the NPS claims and discussed some of our findings at BayCHI.

  • Do detractors really say bad things about a company? Yes! We found strong evidence to support the detractor designation. In our data, we found 90% of negative comments from 500 consumers were associated with scores from 0 to 6 on the 11-point LTR item.
  • Measuring the reliability of the Net Promoter Score. We found the NPS to be as reliable or more reliable than other popular metrics. Reliability is necessary, but not sufficient to have validity.
  • Is the Net Promoter Score a better measure than satisfaction? Not really. Published studies (and our own work) generally show the NPS is the same or worse than multi-item measures of satisfaction when predicting growth or other outcomes. However, it’s easily understood and short, meaning it may be “good enough” in many situations.
  • Does the Net Promoter Score predict company growth? A review of the published literature was disappointing with methodological flaws and biases for or against authors. We found the NPS predicted growth in the U.S. airline industry after we removed the American Airlines acquisition of US Air but there are mixed results in the literature.
  • How harmful is the Net Promoter Score? Not that harmful when used properly. The NPS is reliable, valid (mostly with historical revenue), and short, but not necessarily better than satisfaction. It’s not meant to diagnose UX problems, tell you what to fix, or be a substitute for qualitative methods.
  • Is the NPS the one number you need to grow? We painstakingly reviewed annual reports and company financial statements from 15 years ago to assess how well the NPS predicts future (not historical) growth. We were surprised to see it can explain 38% of future growth in seven industries (not perfect but pretty good actually). Consider this a “best-case” argument for the predictive validity of NPS.

Benchmarks

  • Benchmarking the User Experience (book). I released my 6th book, Benchmarking the User Experience, an approachable “how-to” guide for measuring the experience of websites, software, and product experiences.
  • 15 metrics for UX benchmarking. Benchmark metrics include study-level (SUPR-Q, SUS, UMUX-Lite) and task-level metrics (completion, time, perceived ease).
  • Should you outsource a UX benchmark or do it yourself? MeasuringU is the industry leader at benchmarking websites, software, and products. Outsourcing a benchmark will generally cost more than doing it yourself but is often worth it to quickly achieve more objective results.

Methods

  • The methods UX professionals use (2018). Popular methods such as usability testing and personas remain popular, and accessibility testing has increased over the last eight years.
  • Practical tips for running a PURE evaluation. This analytic method is like a cognitive walkthrough with a simpler metric rubric. More tips for working with PURE.
  • Predicting UX metrics with the PURE method. With multiple datasets, we found that PURE scores predict task-level ease (SEQ) scores generally well, and to a lesser extent predict study-level metrics (SUPR-Q) well.
  • 5 techniques to make your UX research more effective. To research more effectively, use a research matrix, reduce no shows, clean your data, code verbatim comments, and extract insights from videos.
  • Choosing the right UX testing platform. We review the essential features and costs across some popular UX testing platforms, including our MUIQ.
  • Use a research grid to focus study decisions. Be sure your research is focused on the right outcomes by aligning your methods (the how) with your goals (the what).
  • How to turn user videos into insights. Describe the symptoms of problems, look for the root cause, and provide frequencies and an audit trail.

Metrics

    • Using task ease (SEQ) to predict completion rates and times. The Single Ease Question (SEQ) predicts 44% of concurrently collected completion rates and 28% of task times.
    • Building a UX metrics scorecard. A picture (or dashboard) can be worth a 1000 numbers.

UXscorecard-fig3

  • Are top box scores a better predictor of behavior? Turns out extreme responses might be better than the mean. More research is coming on this in 2019.
  • How similar are UX metrics in moderated versus unmoderated studies? Generally close for completion rates and SUS scores, but users tend to take significantly longer in unmoderated studies.
  • Setting metric targets in UX benchmark studies. Setting targets helps achieve goals, but you want them to be realistic and meaningful. Common targets can be above average, better than a competitor, the industry average a grade, or percentile rank.
  • Is a single item enough to measure a construct? For many simple constructs like ease, intent to recommend, and satisfaction, single items are often sufficient (or even superior) to using multiple items.
  • 10 things to know about the SUPR-Q. Some essentials about this widely used UX measure include its being valid, reliable, compact, backed by a normed database that’s updated quarterly, and it predicts the SUS.

UX Profession

  • How satisfied are UX professionals with their jobs? Generally very satisfied, which hasn’t changed much since 2014. Working more than 60 hours a week reduces satisfaction and making more than 150K increases satisfaction.
  • User experience salaries & calculator (2018): Salaries are relatively stable and are driven mostly by where you live (U.S. California is the highest), your job level (senior level makes two times more than entry level), and company size (smaller company employees report making ~70% of larger employers).

Survey

  • 4 classes of survey questions. You can classify most survey questions as: open-ended, closed-ended (static), closed-ended (dynamic), and task-based.
  • 15 common rating scales explained. There are different ways of classifying rating scales and slight variations can result in different looking rating scales, even though they’re variations on the same scale, including linear numeric, Likert, and Visual Analog scales.
  • How to assess brand affinity & sentiment. Measure awareness and attitude and compare it to the top 50 global brands.

Industry Reports

We conducted eleven in-depth benchmark analyses of eleven industries, which include the SUPR-Q and Net Promoter Scores. Thanks to all of you who have purchased our industry benchmark reports. The proceeds from the sales of these reports fund more original research we post on MeasuringU.

  • Brokerages: An understandable dashboard and security are key drivers of UX. Article | Report
  • Banks: Finding fees and locating branches are common, but not necessarily easy tasks.  Article | Report
  • State government: People struggle to know what different government websites offer. Article | Report
  • Health insurance providers: Poor findability leads to calling customer support. Article | Report
  • Universities: Catering to multiple audiences affects findability (including finding the cost to attend). Article | Report
  • State governments: Major improvements since 2012. Renewing vehicle registration & finding government info dominate reasons for usage.  Article | Report
  • News: People are on alert for biased news, loyalty is low, and finding old articles is a challenge. Article | Report
  • Social media:Social media is ubiquitous and influential, but users still struggle to understand core functionality. Article | Report
  • Flower deliveries: The biggest driver of both poor SUPR-Q scores and low task-completion rates are the hidden fees and lack of price transparency. Article | Report
  • Wireless carriers: Finding plan pricing requires navigating multiple pages and using generally poor search and at times confusing navigation. Article | Report
  • Restaurants: Delivery is growing and selecting a store is a key driver and major pain point in the ordering process. Article | Report

 

We’ll see you in 2019! We have plenty of new articles planned, including how well the SUS predicts growth in the software industry, how accurate historical recommendations are, and several new SUPR-Q industry reports including U.S. airlines and automotive websites.