1. Completion rates are the fundamental usability metric: A binary measure of pass and fail (coded as 1 or 0) provides a simple metric of success. If users cannot complete a task, not much else matters with respect to usability or utility.
  2. Easy to understand: They are easy to collect and easy to understand for both engineers and executives. You don’t need to be a statistician to understand what it means when only 20% of your users can complete key tasks on an application.
  3. They are popular: Completion rates are used in approximately 86% of summative usability tests and around 79% of formative tests.
  4. Can be collected at any stage of development: Completion rates work well for low-fidelity prototypes through full fidelity benchmark testing.
  5. Have pre-defined success criteria: Prior to testing, determine what things must happen to consider a task successful. Focus on the outcome of the task (did the user get to the right page, submit the right information) not necessarily the path to completion (although that is important too).
  6. Context Matters: What makes a good task completion rate depends on the context. Where the consequences of failure are high (loss of money, life, reputation etc.,) you’ll want high completion rates.   Past studies or other tests using a similar interface in a similar field can provide guidance on what’s acceptable.
  7. The average task completion rate is 78%: If you have absolutely no clue as to what a good benchmark for a completion rate is, use 78%. This is the average from a database of almost 1200 software and website tasks across several products and domains. As you get more information, update the benchmark.
  8. Compute a Confidence Interval around Completion rate using the adjusted Wald confidence interval. It is accurate for any sample size.  For example, if 1 out of 5 users complete a task, the 95% confidence interval is 2% to 64%. While the interval width is wide (over 60 percentage points), its highly improbable that the completion rate will ever exceed 70%.
  9. To ensure a task-completion rate is at least 70%, plan on testing at least 8 users: If all 8 users complete the task, you can conclude with 95% confidence that the completion rate exceeds 70%  (to ensure a 90% completion you plan on testing a minimum of 30 users).
  10. Around 14% of users who fail a task still rate it as super easy to do: The correlation between task success and task difficulty ratings is around .5. This means that task-ratings only predict around 25% of completion rates. The explanatory power isn’t higher because some users don’t know they failed the task and others complete the task but still think it’s difficult.

 

Related Post