While it might seem like having users think aloud while they complete a task will increase the time on task—the literature is actually mixed. Some studies report no difference, some show longer task times for thinking aloud and others report faster task times for thinking aloud. It has been hypothesized that the reason users can perform faster while they think aloud is because this vocalization allows them to think more clearly through their approach for completing the task. It’s like explaining a problem to someone when the actual process of explaining helps solve it (see Berry and Broadbent (1990) and Lewis 2006 p. 1282).

Probing

The key issue is whether you probe (ask for follow-up explanations after the user makes comments). Probing will increase the task time as users take longer to explain what they meant unless you are able to somehow subtract the probing time from the total task-time. If you need to probe, it doesn’t necessarily render the task time metric useless. If you consistently probe users while they think aloud at different stages of testing then task time can still be used to compare the designs iteratively. There’s a tradeoff between the information gains of probing (keeping in mind that people don’t always know why they do things)

Usability Lab Rules?

 
Photo adapted from Editor B

and the increase in variability of time measurements, which is partly dependent on whether the study is primarily to find and fix problems or obtaining a benchmark.

Having it Both Ways: Retrospective Probing

As is often the case with my testing I want to make the most of user time. I want both a reliable benchmark of the website or application’s usability and I want to know what the problems are and why.  So I ask my users to complete the tasks as quickly as they can without rushing. I don’t say anything about thinking aloud. Some users will express their thoughts as they complete tasks and others are silent. When I see a problem that I want to know more about, I wait until after the task is complete and ask the user to elaborate more on what they did and why. I find this retrospective approach gets me the biggest bang for my user Buck (or Euro).

References

  1. Berry, D. C., and Broadbent, D. E.  (1990).  The role of instruction and verbalization in improving performance on complex search tasks.  Behaviour & Information Technology, 9, 175-190.
  2. Lewis, J. R. (2006). Usability testing. In G. Salvendy (ed.), Handbook of Human Factors and Ergonomics (pp. 1275-1316). New York, NY: John Wiley.