The Think Aloud (TA) method is one of the most distinguishing characteristics of usability testing. The method involves having participants speak their thoughts as they attempt tasks on an interface.
Think Aloud as a method for understanding human attitudes and behavior has been around for over a century. While we often think of the Think Aloud method as a single method, there is substantial variation in how TA is implemented.
In an earlier article, we discussed the differences between four different types of think-aloud studies: concurrent TA (with a moderator), two retrospective types (one with video and one from memory), and unmoderated TA (no moderator).
One of the more recent manifestations of TA is unmoderated thinking aloud. It was introduced with the proliferation of remote UX evaluation and tools, including our MUIQ® platform, which allows participants to record their screen, audio, and webcam without the need for an attending moderator.
In unmoderated TA, participants receive instructions on how to think aloud and are then asked to think aloud during a task. In many cases, instructional videos or repeated digital prompts during the tasks remind participants to continue talking. No moderator is present to prompt or probe the participant. This feature is included in our MUIQ unmoderated testing platform.
The advantage of not having a moderator is that there can’t be any unintentional influence on task performance. However, the disadvantage is that participants may potentially be more likely to give up (drop out).
Higher dropout rates not only mean a study takes longer to fill. It may also affect the representativeness of the study if participants who drop out are different in some meaningful way from those who stick it out.
In turn, this can lead to concerns about the non-representativeness of the findings. Before we investigate any potential effects of study dropout, we first wanted to understand whether asking participants to think aloud in an unmoderated usability study increases the likelihood of dropping out. We aren’t aware of any published data that provides dropout rates, so of course, we decided to conduct a study.
Dropout Rates from Four Experiments
To better understand how asking participants to think aloud may affect participation/dropout rates we used data from four studies.
Between March and July 2022, we collected data from 314 online U.S.-based panel participants across four studies. For each website, participants were randomly assigned to one of two conditions: a think-aloud or a standard task-based condition. Both conditions required participants to download a browser extension to record their screens and clicks (part of MUIQ).
In the Think Aloud condition, participants were asked to also share their microphones, instructed to think aloud, and given a short video demonstrating how to think aloud during the task. Participants, in their study invitation, were told they would need to share their screens and record audio. Participants received the same compensation regardless of the condition to which they were assigned.
Each study focused on a different industry (Real Estate, Airlines, Restaurant Reservations, and Hotels). Each participant attempted one task on one website (Zillow, United, OpenTable, or Tripadvisor)—a between-subjects experimental design. The tasks for each of the industries/websites are shown below.
Task Descriptions
Zillow: Search for a single-family home in Denver, CO, between $325,000 and $400,000, with three bedrooms and two bathrooms. Make sure the home is close to a good school district.
United: You want to fly from Denver, CO, to Los Angeles, CA, with your significant other. Arrive in L.A. on September 8th and leave on September 12th. Select the earliest, nonstop flight on both of your travel days. Select the fare that allows you to choose your seat ahead of time. No checked bags. Choose seats next to your partner.
OpenTable: Make a reservation for four people at a sushi restaurant in Denver, CO, tomorrow any time after 5:00 pm. Make sure the restaurant you select is not at the lowest or highest price point. Of the restaurants that fit these criteria, look at their overall rating, customer reviews, and photos to select the one that is the most appealing to you.
Tripadvisor: Look for a hotel in Denver, CO, from September 19th to the 22nd for you and your spouse that costs less than $200 per night. Make sure the hotel has a fitness center, conference rooms, free parking, and has more than 500 reviews. Of the hotels that meet these criteria, look through the hotel reviews and photos to determine which best suits your preferences.
The OpenTable, Zillow, and Tripadvisor studies took participants approximately 11 minutes to complete, and the United study took approximately 16 minutes to complete. In addition to the task, participants were asked some initial demographic and prior experience questions. After the task, participants answered some post-study questions, including the SUPR-Q® and the UX-Lite®.
Results
Table 1 shows the differences in the dropout rates between the Think Aloud condition (TA) and the non-Think Aloud condition (non-TA). Table 2 shows the results of the N−1 two-proportion tests for each study and overall. Across all four studies, asking participants to think aloud significantly increased the dropout rate. Overall, the dropout rate more than doubled, from 19% for non-TA conditions to 50% for TA conditions (p < .0001). At the study level, the dropout rates were statistically different (p ≤ .01), except for the study on United Airlines (p = .37).
| Website | TA Start | TA Complete | % Dropout | Non-TA Start | Non-TA Complete | % Dropout |
|---|---|---|---|---|---|---|
| Zillow | 45 | 26 | 42% | 28 | 26 | 7% |
| United Airlines | 34 | 22 | 35% | 32 | 24 | 25% |
| OpenTable | 60 | 28 | 53% | 25 | 19 | 24% |
| Tripadvisor | 63 | 24 | 62% | 2 | 22 | 19% |
| Total | 202 | 100 | 50% | 112 | 91 | 19% |
Table 1: Differences in dropout rate (% Dropout columns) between Think Aloud (TA) and non-Think Aloud (non-TA) conditions.
| Website | z | p | d | 95% Low | 95% High |
|---|---|---|---|---|---|
| Zillow | 3.20 | 0.001 | 35.1% | 14.9% | 50.4% |
| United Airlines | 0.90 | 0.37 | 10.3% | −11.9% | 31.3% |
| OpenTable | 2.46 | 0.01 | 29.3% | 6.7% | 48.1% |
| Tripadvisor | 3.75 | 0.0002 | 43.4% | 22.0% | 59.8% |
| Total | 5.51 | < .0001 | 31.7% | 21.2% | 41.1% |
Table 2: Significance of N-1 two-proportion tests and 95% confidence intervals around differences for comparison of TA and non-TA for each study and overall totals.
There was some variability among the studies, with the dropout rate being low for the Zillow study non-TA condition (7% dropout) and high for the Tripadvisor TA condition (62%). The ten-point gap between the dropout rates for United Airlines (lowest dropout rate for TA condition minus the highest dropout rate for the non-TA condition: 35% − 25%) was not statistically significant. It’s unclear whether there’s anything about the United Airlines task or website that may explain the smaller gap between conditions.
Summary and Discussion
Does asking participants to think aloud increase study dropout rates? The short answer is yes, definitely.
Key Findings
Across four online studies with 314 participants, our key findings were:
Asking participants to think aloud doubles dropout rates. Participants who begin an online study and are asked to think aloud are more than twice as likely to drop out compared to participants who are asked to share their screen only and not think aloud or share their webcam (50% vs. 19%). Note that this difference may be an underestimate, as all participants were told they would have to record audio (even participants who ultimately weren’t in the TA condition).
There is variation in dropout between studies. There is some variation in the relative dropout rate across the studies. While there were statistical differences in dropout rates between most conditions, in the United Airlines study, the difference of 10 points (35% to 25%) wasn’t statistically significant. This suggests factors such as the task or website may influence different dropout rates.
Key Takeaways
The key takeaways for UX practitioners are
High dropout means more resources required for data collection. The consequence of having a high dropout rate means you should plan to invite more than roughly double the number of prospective participants you need to take a study if you ask them to think aloud. For example, if you need 20 participants in a TA study, plan to invite at least 40.
Random assignment increases confidence in these results. As far as we know, this is the first quantitative examination that assigned participants randomly to either TA on non-TA for unmoderated usability studies. The random assignment better allowed us to control for confounding variables.
Moderator influence ruled out. One of the common confounding factors in earlier studies on thinking aloud was the influence of a moderator—how a moderator, either in place of or in addition to the process of thinking aloud—may affect results. This influence is ruled out when conducting unmoderated studies.
Future Research
What causes differences in dropout rates across studies? While this analysis was intended to estimate the dropout rate from asking participants to think aloud, it doesn’t address the question as to why more are dropping out. It’s likely some combination of the perceived additional effort (without additional compensation), potential privacy concerns, and technical problems with microphones and browsers. A future analysis can investigate the cause further. We’ll also more closely examine differences in total task time when participants think aloud.
Do differences in TA and non-TA dropout rates affect generalizability? In a future analysis, we’ll further examine whether the higher dropout rate contributes to a generalizability problem—that is, whether participants who think aloud are in some systematic way different than people who don’t.
Acknowledgments: Thanks to Dylan Atkins for contributing to the data collection and analysis.
