In an earlier article, we showed that only about 9% of panel participants will eventually complete a study in which they are asked to think aloud. That is, if you need ten usable think-aloud videos, expect to invite around 111 participants.
On the surface, this means you’ll need to plan for a lot of people to ultimately get a modest sample. But a larger concern is whether the people who do participate are systematically different from those who don’t.
One of our motivations for analyzing unmoderated think-aloud study data in detail was to understand the generalizability of the findings from think-aloud studies (and because nothing systematic has been researched on this topic). However, in our earlier analyses on study-level metrics and task-level metrics, we analyzed only those who ultimately participated from a general panel. In this article, we’ll look at the demographic variables we collected from the roughly 50% of people who were unwilling to participate.
Soliciting Think-Aloud Participants
Between May 2021 and April 2022, we conducted five surveys asking online panel participants from the US and UK about prior experience with websites and software products. Across the surveys, we also asked participants whether they would be willing to participate in a follow-up think-aloud study that, for additional compensation, would involve sharing their microphone and webcam (asked separately in the survey). The percentage of participants willing to participate who agreed to each request appears in Table 1.
| Survey No. & Country | Total | % Willing to TA (share webcam/mic) |
|---|---|---|
| 1 (UK) | 100 | 37% |
| 2 (US) | 202 | 46% |
| 3 (US) | 303 | 52% |
| 4 (US) | 507 | 50% |
| 5 (US) | 104 | 49% |
| Total | 1216 | 48% |
Across the 1,216 participants in all five surveys, a bit less than half (48%) reported being willing to share their webcam and microphone, a necessary step in participating in a think-aloud study. We’ve already reported that those in the UK were significantly less willing to participate than people in the US (UK: 37%, US: 49%).
We now shift our focus to the 627 participants who were not interested in participating in a TA study, comparing them with those who did participate.
Demographic Analyses
We collected basic demographic data across the five studies, but some studies had more data than others. Table 2 shows the demographic data we had across the five studies that were collected at least twice. Age was collected the most often (in four of the five studies).
| Survey Number | Gender | Age | Education | Income |
|---|---|---|---|---|
| 1 (UK) | Y | |||
| 2 (US) | Y | |||
| 3 (US) | ||||
| 4 (US) | Y | Y | Y | Y |
| 5 (US) | Y | Y | Y | Y |
| Total | 2 | 4 | 2 | 2 |
We compared the demographic distributions with the data from the 221 participants who agreed to Think Aloud, as we described in our earlier articles on time and task metrics. We started with gender differences (Table 3):
| Gender | Completed TA Study | “No” in Screener | Diff |
|---|---|---|---|
| Female | 43.9% | 51.1% | 7.2% |
| Male | 53.8% | 47.2% | −6.6% |
There was about a 7% shift in the distribution of men and women. Women were slightly more represented in the No group (51%) than in the group that completed the study (44%), but the difference was not statistically significant (χ2(1) = 2.5; p = .11).
Table 4 shows the frequency distribution of age from the US Census and between those who completed the TA studies and those who refused in the screener. In some studies, age brackets did not align with those used in the final TA studies, so only ages that aligned were included (leaving out 309 respondents from the screener).
| Age | US Census 2021 | Completed TA Study | “No” in Screener | Completed— “No” Diff | “No”/ Completed Ratio |
|---|---|---|---|---|---|
| 18–24* | 13% | 14% | 31% | −17% | 2.2 |
| 25–34 | 14% | 25% | 33% | −8% | 1.3 |
| 35–44 | 13% | 31% | 19% | 12% | 0.6 |
| 45–54 | 12% | 15% | 7% | 8% | 0.5 |
| 55–64 | 13% | 12% | 4% | 8% | 0.3 |
| 65+ | 17% | 2% | 4% | −2% | 1.9 |
Comparison of “Completed” and “No” distributions with the US population shows that 65+ was underrepresented in both samples, probably due to low participation of that age group in online surveys. “Completed” was closer than “No” to the census in four of the other five age groups (the exception was 35–44). It is conceivable that the younger age groups were overrepresented in the panel. Due to larger percentages of younger groups declining to participate in TA, the percentage completing TA sessions grew closer to the census percentages. It’s important, however, not to make too much of this. The percentages might be similar, but the process that the “Completed” group went through makes it implausible that this group is experientially similar to the US population at large.
The difference in “Completed” and “No” distributions is statistically significant (χ2(5) = 45.43; p < .001). Interestingly, younger participants were more likely to decline to participate in the TA studies. Roughly 64% of the “No” group were 34 or younger compared to 39% “Completed” from the same age range. The largest percentage point difference was for the youngest cohort, where more than twice as many 18–24-year-olds were in the “No” group compared to the “Completed” group (31% vs. 14%).
This finding is somewhat surprising. We expected more representation from the youngest cohort because thinking aloud requires interaction with more technology. Instead, we found the opposite.
Table 5 shows the differences in education between the samples, with no statistical difference (χ2(3) = 1.9; p = .59).
| Completed TA Study | “No” in Screener | Diff | |
|---|---|---|---|
| High school diploma/GED | 13% | 16% | −3% |
| Some college (no degree) | 28% | 26% | 2% |
| Bachelor's degree | 42% | 39% | 3% |
| Graduate/Professional Degree | 17% | 20% | −3% |
Table 6 shows the differences in income between the samples. While some larger nominal differences are in the lower income group, the differences aren’t statistically significant ((χ2(3) = 3.74; p = .29).
| Income | Completed TA Study | “No” in Screener | Diff |
|---|---|---|---|
| $0–$24k | 31% | 46% | −15% |
| $25k–$49k | 19% | 35% | −15% |
| $50k–$99k | 32% | 39% | −7% |
| $100k+ | 18% | 19% | −2% |
Discussion and Summary
An examination of the demographic differences between those who completed a TA study compared to those who weren’t willing to participate revealed:
Most demographic variables didn’t reveal significant differences. This is often the case in UX research. We collect demographic information because it’s relatively easy, clients usually want to know the demographic composition of samples, and occasionally the differences are statistically significant. In these analyses, the differences in gender, education, and income were not significant. It’s possible with larger sample sizes that differences will emerge, but the size of the differences will likely be modest.
Country matters. Willingness to participate in TA was significantly lower in the UK than in the US (at least, for the online panels we used).
Age matters. Of all the demographic data collected in UX research, differences due to age are often statistically significant. We found significant differences among age groups in this study, but not in the way we expected. Among participants 34 or younger, 64% declined to participate and 39% completed a TA session. More than twice as many in the 18–24 group declined to participate (31%) compared to those who completed (14%).
Other variables might matter. We had only demographic data to work with for these analyses, so that’s where we started. Some other variables that might matter are difficult to collect in surveys (e.g., psychological individual differences such as field dependence/independence). Others, however, would be relatively easy to collect (e.g., self-reports of product experience, tech savviness, sensitivity to privacy). In particular, sensitivity to privacy might account for the differences we found for the UK (presumably more sensitive to privacy) and the US (presumably less sensitive to privacy), and it may also account for a part of the age group differences. We can explore these variables in future research.
