What To Do When You Can’t Run a Usability Test

Jeff Sauro, PhD

In an ideal world, users would be involved in every stage of product development, including requirements gathering, iterative prototype testing and post release testing.

However, there are a lot of reasons why testing with users doesn’t happen.

Among the most common are:

  • Time: Running moderated test sessions takes time to plan and conduct. A developer, product manager or single user researcher with too many priorities often has very little time to spend on planning, recruiting, and moderating. Unmoderated testing solutions from companies like MeasuringU and UserTesting.com take less time to collect data but still require thoughtful time for set up of tasks, the product and analysis.
  • Difficulty finding qualified users: Physicians, engineers, IT admins, CFOs, attorneys, and small business owners are all users we’ve needed to test. They are also users that have few opportunities to take an hour (or more) out of their day for a usability test.
  • Having an interface for users to test: if an app doesn’t exist yet, or you’re considering a new version, or the application crashes and can’t be tested outside a firewall, it makes testing very difficult. What’s the point of spending time and money on a test only to have users sit there while you attempt to troubleshoot problems with the tech support team?
  • Cost: It takes money to find, facilitate and analyze users. It takes money to have a stable app ready to develop, or to pause A/B testing. Even reduced cost usability testing takes some budget and if it hasn’t been allocated, a few hundred dollars can be difficult to scrape up, even in large profitable companies.

While there are ways of mitigating each of these reasons, sometimes testing with users just doesn’t happen. Three alternatives we use to understand the user experience without testing users are:

  1. Heuristic Evaluation: When you’ve watched enough people attempt to use an interface, you notice patterns in behavior that design elements will illicit. For example, there are rare cases where having a form with a reset button is a good idea. It’s usually just clicked by accident. Having 2-3 experts in both usability and the domain can often identify 30-50% of the issues users will encounter. Heuristic Evaluation was the original discount usability method, devised especially for situations where it’s difficult to test with actual users.
  2. Surveys: Ask qualified participants to reflect on their most recent experiences, indicate the major problems they are having , and explain what tasks they were doing. This provides insight into low hanging fruit. For example, if 30% of respondents said that they have problems browsing the mobile website because of a light-box that won’t close, it’s something to examine more closely. Where possible, you can also ask participants to visit a website or have them open a product and try to complete the tasks as if it were a usability test. Interestingly enough, these self-reported usability problems often uncover up to half of the same issues uncovered in a traditional moderated study. You can even have users answer a standardized usability questionnaire like the SUS or SUPR-Q. I don’t recommend comparing the results from questionnaires collected outside of a usability testing with those collected during a study. While this is an open research question, the results are usually disparate enough to make comparisons difficult.
  3. Keystroke Level Modeling : When productivity and efficiency are important goals with an interface, you can measure how long it will take skilled users to complete tasks without errors. We used a modified KLM technique successfully while I was at Oracle where it always seemed like at least three of the four common roadblocks prevented us from testing with users! KLM works especially well when you want to answer “what if” scenarios before embarking on an expensive development effort only to find out that the new prototype takes users twice as long to complete the same tasks. Who wants to spend twice as long filing their expense report?

None of these methods are substitutes for conducting a usability test. Even a few users will uncover most of the obvious issues with an interface. And even if you can conduct a usability test, using Heuristic Evaluations, Surveys and KLM are all great methods to complement and reinforce the data you get from usability testing.

    Your Cart
    Your cart is emptyReturn to Shop
    Scroll to Top