Search
Exact matches only
Search in title
Search in content
Search in comments
Search in excerpt
Filter by Custom Post Type
How Large Is the Evaluator Effect in Usability Testing

Usability Testing

by Jeff Sauro | March 27, 2018

Browse Content by Topic

Statistics ( 51 )
 Methods ( 50 )
 Usability Testing ( 49 )
 UX ( 49 )
 Survey ( 30 )
 User Research ( 27 )
 Customer Experience ( 26 )
 Usability ( 26 )
 Benchmarking ( 21 )
 Sample Size ( 18 )
 Usability Problems ( 17 )
 NPS ( 17 )
 SUS ( 17 )
 Usability Metrics ( 12 )
 SUPRQ ( 12 )
 Rating Scale ( 12 )
 Qualitative ( 11 )
 Net Promoter Score ( 10 )
 Navigation ( 10 )
 Metrics ( 9 )
 Measurement ( 9 )
 Market Research ( 8 )
 Task Time ( 8 )
 User Experience ( 8 )
 Surveys ( 7 )
 Heuristic Evaluation ( 7 )
 Task Completion ( 7 )
 Questionnaires ( 6 )
 Usability Problem ( 5 )
 Visualizing Data ( 5 )
 Six Sigma ( 5 )
 Mobile ( 5 )
 Reliability ( 5 )
 Mobile Usability Testing ( 5 )
 Questionnaire ( 5 )
 Quantitative ( 4 )
 Analytics ( 4 )
 Credibility ( 4 )
 Confidence Intervals ( 4 )
 Research ( 4 )
 UX Maturity ( 4 )
 Loyalty ( 4 )
 Moderation ( 4 )
 Confidence ( 4 )
 Satisfaction ( 4 )
 Task Times ( 4 )
 Validity ( 4 )
 Rating Scales ( 3 )
 UX Metrics ( 3 )
 Customer Segmentation ( 3 )
 Card Sorting ( 3 )
 Lean UX ( 3 )
 ROI ( 3 )
 Task Metrics ( 3 )
 Expert Review ( 3 )
 Key Driver ( 2 )
 Correlation ( 2 )
 Branding ( 2 )
 Remote Usability Testing ( 2 )
 Excel ( 2 )
 SEQ ( 2 )
 Summative ( 2 )
 Cognitive Walkthrough ( 2 )
 KLM ( 2 )
 Usability Lab ( 2 )
 Marketing ( 2 )
 Unmoderated Research ( 2 )
 UMUX-lite ( 2 )
 Certification ( 2 )
 Eye-Tracking ( 2 )
 SUM ( 2 )
 Personas ( 2 )
 UX Methods ( 2 )
 Tasks ( 2 )
 Data ( 2 )
 Salary Survey ( 2 )
 PhD ( 2 )
 Tree Testing ( 2 )
 Focus Groups ( 2 )
 Findability ( 2 )
 A/B Testing ( 2 )
 IA ( 2 )
 UX Salary Survey ( 2 )
 Problem Severity ( 1 )
 Site Analytics ( 1 )
 Information Architecture ( 1 )
 Contextual Inquiry ( 1 )
 Desktop ( 1 )
 Ordinal ( 1 )
 Crowdsourcing ( 1 )
 Sample ( 1 )
 Five ( 1 )
 Random ( 1 )
 Think Aloud ( 1 )
 Errors ( 1 )
 Trust ( 1 )
 Formative ( 1 )
 Perceptions ( 1 )
 Performance ( 1 )
 Facilitation ( 1 )
 protoype ( 1 )
 Metric ( 1 )
 Prototype ( 1 )
 Task Completin ( 1 )
 Z-Score ( 1 )
 Affinity ( 1 )
 Visual Appeal ( 1 )
 True Intent ( 1 )
 Expectations ( 1 )
 Conjoint Analysis ( 1 )
 Regression Analysis ( 1 )
 Competitive ( 1 )
 Quality ( 1 )
 Task Randomization ( 1 )
 Test Metrics ( 1 )
 SUPR-Q ( 1 )
 Software ( 1 )
 Unmoderated ( 1 )
 Design ( 1 )
 Top Task Analysis ( 1 )
 Effect Size ( 1 )
 User Testing ( 1 )
 Segmentation ( 1 )
 Persona ( 1 )
 Margin of Error ( 1 )

How Large Is the Evaluator Effect in Usability Testing?

By Jeff Sauro . March 27, 2018 . Tags: , ,
Uncovering usability problems is at the heart of usability testing. Problems and insights can be uncovered from observing participants live in a usability lab, using heuristic evaluations, or watching videos of participants. But if you change the person looking for the problems (the evaluator), do you get a different set of problems? The Evaluator Effect It’s been 20 years since two influential papers (Jacobsen, Hertzum,

Read More

Do Novices or Experts Uncover More Usability Issues?

By Jeff Sauro . February 27, 2018 . Tags:
Finding and fixing problems encountered by participants through usability testing generally leads to a better user experience. But not all participants are created equal. One of the major differentiating characteristics is prior experience. People with more experience tend to perform more tasks successfully, more quickly and generally have a more positive attitude about the experience than inexperienced people. But does testing with experienced users lead to uncovering

Read More

8 Ways to Minimize No Shows in UX Research

By Jeff Sauro . January 23, 2018 . Tags: ,
While UX research may be a priority for you, it probably isn’t for your participants. And participants are a pretty important ingredient in usability testing. If people were predictable, reliable, and always did what they said, few of us would make a living in improving the user experience! Unfortunately, people don’t always show up when they say they will for your usability test, in-depth interview,

Read More

Do Observers Affect Usability Test Results?

By Jeff Sauro . November 14, 2017 . Tags:
Many researchers are familiar with the Hawthorne Effect in which people act differently when observed. It was named when researchers found workers at the Hawthorne Works factory performed better not because of increased lighting but because they were being watched. This observer effect happens not only with people but also with particles. In physics, the mere act of observing a phenomenon (like subatomic particle movement)

Read More

10 Resources for Usability Facilitators

By Jeff Sauro . September 26, 2017 . Tags: , ,
Facilitating a usability test is a skill. With enough of the right practice you’ll get better at facilitating and running more effective usability test sessions. A solid foundation in both the theory and practical applications of facilitating a usability test will aid you in becoming a solid facilitator. To help, here are ten resources for both beginners and intermediate usability test facilitators. 1. Read about

Read More

When to Provide Assistance in a Usability Test

By Jeff Sauro . August 29, 2017 . Tags: ,
One of the fundamental principles behind usability testing is to let the participants actually use the software, app, or website and see what problems might emerge. By simulating use and not interrupting participants, you can detect and fix problems before users encounter them, get frustrated, and stop using and recommending your product. So while there’s good reason to shut up and watch users, should a

Read More

5 Interfaces That Benefit From Usability Testing

By Jeff Sauro . March 7, 2017 . Tags:
The fundamental idea behind usability testing is that the interface creator is not the user. We can broaden the idea of an interface to encompass more than websites and software as Don Norman famously illustrated in his book The Design of Everyday Things. An interface is the point where people and systems interact. An interface can be words, images, light switches, door handles,  or complex

Read More

UX & Net Promoter Benchmarks for Entertainment Websites

By Jeff Sauro . January 24, 2017 . Tags: , ,
Content is king. Whether it’s for books, movies, audio books, news sites, or entertainment websites. When you have good content people will come and stay. But if people can’t find the content or there’s too much friction in the experience you’ll likely lose your audience even with killer content. An increasing number of consumers now subscribe to a premium content service like Netflix, Hulu, or

Read More

10 Golden Rules Of Facilitation

By Jeff Sauro . January 11, 2017 . Tags: ,
Facilitation is a valuable skill for measuring the user experience. A good facilitator ensures sessions run smoothly, make participants comfortable, and extract the right data for even the most difficult scenarios, stakeholders, or participants. Joe Dumas and Beth Loring wrote a great guidebook that is an essential read for anyone interested in facilitating a usability session. Even though it's almost a decade old, it's still

Read More

Does Thinking Aloud Affect Where People Look?

By Jeff Sauro . January 4, 2017 . Tags: ,
Having participants think aloud is a valuable tool used in UX research. It's primarily used to understand participants' mental processes, which can ultimately uncover problems with an interface. It has a rich history in the behavioral sciences that dates back over a century. Despite its value, it's not without its controversy. Some research has shown that depending on the activity, having participants think aloud can

Read More