Have you taken a survey for a company without an incentive?
I mean surveys where you have no clear chance of winning a prize, getting a discount, or receiving any clear compensation for your time?
If you did, what motivated you to take it? Were you just curious, maybe killing time? Did you have a more positive or negative attitude toward the product or company that you wanted to express?
Empirical customer research requires people. Especially for online survey research, which usually requires a lot of people. Surveys remain a popular method for UX research, too.
Researchers need to rely on both a mix of volunteer and paid participants for surveys and other online research, such as unmoderated UX studies. We’ve discussed ways to improve surveys, considerations for paying, working with online panels, and things that make surveys painful.
But what motivates people to take these surveys for free? Are participants who volunteer their time more likely to be satisfied or dissatisfied with the company or product? Is it only the really angry or really happy people who bother to fill out surveys?
You can find plenty of advice online for getting people to take your survey, but very little on what motivates people to take them for free.
To better understand the reasons people take unpaid surveys and to see whether their attitudes were more systemically positive or negative, we conducted our own research across two online studies.
The first study, conducted in August 2017, asked participants whether they had taken an unpaid survey and then asked them to select the reason that best matched their motivation. Participants were recruited (325 responses were collected) from online advertisements for desktop and mobile.
The second study was conducted in November 2018 using a U.S. paid panel of 525 participants. In both data sources, participants could select multiple reasons using a set of predefined options (that we derived from online sources), including an “Other” option. The results of the predefined categories are shown in Figure 1.
The most common reasons selected were that people liked to express their opinion (42%), wanted to change the way things are done (33%), and found the surveys interesting (32%). Notably, only 13% reported having a problem (a negative experience). In contrast, more than twice as many reported a good experience (29%). Only 5% selected “Other,” suggesting that these reasons may sufficiently account for people’s motivation to take free surveys (at least for the sample we used).
This initial analysis suggests respondents may have a more positive than negative bias when taking unpaid surveys. To further investigate differences in sentiment between paid and unpaid participants, in the November 2018 study we also asked participants to answer a 7-point sentiment question (from 1 = Not at all favorable to 7 = Extremely favorable) toward the company/product they took a paid and/or unpaid survey for. Some examples of unpaid survey companies include:
- Bradburn Apartments
- Time Warner/Spectrum
- Community Care Behavioral Health
- Six Flags Great America
- NJ transit
- Papa John’s
Examples of paid survey companies include:
- Capital One
- Burger King
- Mechanical Turk
About a third (32%) of the participants reported having taken an unpaid survey for a brand or product and 38% reported having taken a paid survey (other than the current survey). Interestingly, the average sentiment was about 10% higher for paid survey participants (5.5 vs. 5) and the difference was statistically significant (p < .01).
The distribution of sentiment is shown in Figure 2. While both show a positive bias, there is a difference in the distribution between paid and unpaid respondents. Significantly more paid respondents selected the most favorable sentiment of 7 (top box) compared to unpaid respondents (31% vs. 19%). The unpaid survey respondents also had more unfavorable responses (14% selecting 3 or less) than paid respondents (8% selecting 3 or less).
Furthermore, 92 participants (about 16% of the sample) reported having taken both a paid and an unpaid survey. When we looked at just those participants (to control for between-participant variability), a similar pattern was seen. Companies from paid surveys were still rated higher (5.6 paid vs. 5.4 unpaid), but this smaller difference wasn’t statistically significant.
While there was a wide variation in companies identified by participants, some companies appeared multiple times on both unpaid and paid surveys. For example, 10 respondents reported taking an unpaid survey for Walmart and 7 reported taking a paid survey for Walmart. Even at this small sample size, the same pattern emerged. There was a relatively large difference of 1 point (bordering on statistical significance) with participants again rating the paid experience higher than the unpaid (5.4 vs. 4.3; p = .125).
It’s unclear whether this higher sentiment toward paid surveys is because people are getting paid (a sort of halo effect) or whether this is indeed evidence that unpaid respondents are just a bit more likely to be less satisfied (or even dissatisfied) than paid participants. It’s also possible that some of the aggregate attitudes were affected by the survey company that administered the surveys. For example, several respondents reported the company they took the paid survey for were Swagbucks, Mechanical Turk, or Opinion Outpost (all companies that administer surveys). It could be that respondents aren’t differentiating between the companies that administer the surveys and the company the survey is for. But even when we looked at a common brand such as Walmart we saw the same pattern of slightly higher attitudes from paid respondents. A future analysis can examine this more.
Summary and Takeaways
Across two studies investigating the motivations for why people take unpaid surveys, we found:
Interest, affecting change, and intrinsic motivation are key motivators. The primary reasons participants report taking unpaid surveys are because of interest, wanting to express their opinion, and hoping to effect changes at the company.
There’s a slight positive bias for all surveys. In general, participants tend to have a slightly favorable attitude toward the company or product for both unpaid and paid surveys.
Paid survey attitudes are higher than unpaid ones. Somewhat surprisingly, we found the sentiment towards companies in paid surveys was higher, both in the mean and extreme responses than unpaid surveys. This pattern held when we examined the same participants and looked at the same company (Walmart) despite the possible influence from paid survey providers such as Mechanical Turk.
There’s a possible selection bias. It could be that our two methods for collecting participants (online ads and paid panels) may bias the types of participants who joined our studies. It may not, for example, be representative of participants who take unpaid surveys (as they may be less likely to take online paid surveys). Future research can examine this in more detail.
|UX Measurement Boot Camp : Three Days of Intensive Training on UX Methods, Metrics and Measurement Aug. 7th-9th 2019|