Usability measurements involve human performance and because human behavior is inherently error prone, reaching the goal of 6σ isn’t necessary to proclaim success. Manufacturing companies that are considered producing “high-quality” products are usually somewhere between 4σ and 5σ. The benchmarks and targets that we set in our tests will necessarily need to be more forgiving than in manufacturing. I prefer focusing on the movement of the sigma from one version to the next. Remember, going from 2σ to 4σ is an improvement of 50 times! I also consider the usability of a product to be the sum of a variety of measures of usability—more than just task completion. You can take any list of usability metrics and then derive a sigma value for each. Some measures reveal often overlooked but crucial aspects of what users truly perceive when using a product. The following are some measures and their target benchmarks found to be indicative of a product’s usability: task completion, task time number of critical errors.
Task Completion Sigma
By far the most commonly measured and reported usability metric is task-completion. Unfortunately it is often one of the only metrics analyzed. A common benchmark for the percent of users that should complete a task is 90% or 90 out of 100 users that attempt a task should be able to complete it. There are at least two problems with this benchmark:
- For “experienced” users–users that complete the task at least on a weekly or monthly basis, 90% is much too low.
- Task completion by itself tends to only be a good indicator of a really unusable product. That is, if novice users are only able to complete 50% or 60% of the tasks, is it really necessary to investigate further?
The Bottom Line
In a sense, task completion then is a good preliminary test for detecting egregious usability problems or for first time or novice users. I’d continue to use 90% as goal for novice(never or rarely completed the task) and expect 99% for experienced(complete the task weekly) users.
