It would be nice if all users completed tasks in a usability test. If they did you wouldn’t have to worry about what to do with their task times if they are unable to complete the task. But then again, if all your users had no problems completing tasks, you wouldn’t worry about improving usability. So what do you do with the tasks times of users who don’t complete a task successfully? Do we include them or exclude them from the average task time?
First, if you are going to record task time don’t throw away the task times of users who fail the task scenario. After reviewing the video and further consideration maybe they did complete the task or maybe you made a coding error. Keeping these failed times then allows you to report the following three task time measures, all of which can help establish benchmarks and aid in diagnosing problems.
- Average Task Completion Time: Include only users who completed the task successfully.
- Mean Time to Failure: The average time users are spending on the task before they give up or complete the task incorrectly.
- Average Time on Task: The total duration users are spending on your task
Report all average takes times using confidence intervals and for small sample sizes (under 25) the geometric mean is the better estimate of the middle value than the mean or median.
For most cases the average task completion time is all you would report (excludes the failed task times). However, if your tasks contain a lot of failure and thus you’re excluding many users from the average time be careful you’re not painting too rosy a picture of efficiency. If you have 12 users attempt a task and only 4 complete it, the mean time to failure and the total time on task would likely be as informative as the average time for just those lucky few who made it through the task successfully.
