6 Things You Didn’t Know About Heuristic Evaluations

A Heuristic evaluation is a process where someone trained in usability principles reviews an application (a website or software). She compares the website against a set of guidelines or principles (“Heuristics”) that tend to make for more usable applications.

For example, if while completing a task a user gets a message that says “Error 1000xz Contact System Administrator” this would violate a Heuristic: “Error messages should be expressed in plain language (no codes).”

Maybe you already have heard-of and use Heuristics Evaluations. Here are six things you might NOT know about this popular usability method:

1. An effective Heuristic Evaluation needs to have multiple evaluators (3 to 5) not just one because different evaluators tend to find different problems.
2. Heuristics Evaluations should be done prior to and in addition to user-testing, not instead of user-testing. If you have easy access to a few experts then you can identify a lot of low-hanging problems prior to subjecting them to users. What’s more, you can then check your expert effectiveness by seeing how many HE problems are encountered by users during the user-test.
3. Rolf Molich, one of the co-creators of the method said “Heuristic Evaluations are 99% bad” at UPA 2009 on a panel with Jakob Nielsen and Chauncey Wilson on Heuristic Evaluations. His reaction was in large part because HE is being used instead of user-testing and only loosely based on any heuristics.
4. Double experts are better than single experts: Being an expert in both usability principles and the domain[pdf] make for better, more focused heuristic evaluations. So if you’re testing a financial application you’d have better results by having a usability expert either with an accounting background or well versed in the lexicon and flow of accounting and finance software.
5. There is not a single set of Heuristics. Nielsen’s 10 famous heuristics are the ones most used but there are also others:
   1. Bastien and Scapin created a set of 18 Ergonomic criteria [paid link]
      
   2. Gerhardt-Powals 10 Cognitive Engineering Principles
   3. Connell & Hammond’s 30 Usability Principles [inverted pdf]
   4. Smith & Mosier’s 944 guidelines for the design of user-interfaces (from 1986)
6. There is controversy over just how effective Heuristic Evaluations are in finding problems as compared to user-testing. There is even controversy over how you determine the effectiveness of a usability method. This was one of the key criticisms in Gray and Salzman’s 1998 influential paper Damaged Merchandise: A simple count of problems isn’t a reliable or valid measure of effectiveness.