In exchange for their time, users are compensated.
One drawback to these professional users is that there are some who are in it just for the money.
Consequently, they may not take your study seriously and “speed” through the questions to receive their remuneration.
For open-ended comments and feedback, it’s pretty obvious who’s not taking your test or survey seriously. Terse comments such as “it’s easy” and “everything is great” are good indications of a cheater.
For multiple choice rating scale questions though, detecting disingenuous answers is not as obvious because all responses to rating scales are usually acceptable.
Detecting Cheaters: Please Select a 3
Adding just a single question to your survey is usually sufficient for detecting the most egregious cheating users. A question like “please select a 3 for this question” would easily tell you whether someone is even half-paying attention.
Of course this only nets the respondents who are completely blazing through your survey. Some professional cheaters have caught on to this trick and look for such questions.
Detecting Cheaters: Conflicting Responses
Another option is to include two versions of the same question but with completely opposite wording. For example: “I enjoyed using this website” and “I did not enjoy using this website.”
If a respondent agrees or disagrees to both questions then they are probably not taking your questions seriously. There is a chance users will just misread the question even though they were genuinely answering your questions.
Many questionnaires like the System Usability Scale(SUS) already have questions worded both positively and negatively.
It is possible for respondents to agree to two statements such as “I think that I would like to website frequently” and “I thought there was too much inconsistency in the website.” The responses are only somewhat conflicting. However, it is unlikely that a respondent would agree to all 10 items which contain five positive and five negative items.
Looking for all 5’s, 4’s, 2’s or 1’s in the SUS questionnaire can also be a way for detecting these cheaters (as suggested in Albert et al 2010).
How common are cheaters?
I looked at five datasets that contained a speeding question (e.g. “Select a 3 here”) and 55 datasets using an online administered SUS (see Table 1 below).
|# of Cheaters||Total Sample Size||%|
Table 1: Number of respondents in five online surveys that answered simple
validation questions incorrectly (e.g. “select a 3 here”).
Across the five datasets, the percentage of respondents that answered the cheating question incorrectly range from a low of 2.2% to a high of 20%.
Of the 2646 users tested, 317 (12%) answered the simple question wrong. This number is pulled a bit higher due to one large study[pdf].
To detect cheaters using the SUS, I looked for users that answered with all the same values except 3 (3 is an acceptable neutral response for all SUS items). Across the 55 online SUS datasets, only 7 contained a user that answered the same value for all 10 items.
In fact, of the 950 total users, only 7 (0.74%) responded with all 1’s, 2’s 4’s or 5’s. This method is detecting less than 10% of the actual cheaters.
Expect 10% of users to Cheat
When conducting your next online survey or usability test, anticipate somewhere around 10% of the responses from online professional users to be disingenuous. If you’re hoping to have 100 usable responses for a study, plan on obtaining 110 and having to throw-out around 10.
Of course the percent of users that cheat depends on a number of factors including how long your survey is, the quality of the panel and even things like the day of the week and time of day. But if your survey is long and complicated, don’t be surprised to see that number hit 15-20%.
“Select a response” items are simple and effective
Detecting cheaters is best done with a simple “Select this response” or a conflicting question. Using responses to semi-conflicting items like those in the SUS, are detecting only a fraction of the cheaters.
Unfortunately, cheaters either answer too haphazardly or have caught on to this detection method and avoid picking the same response to all items.
Conflicting questions are also viable but be careful how you word the item. You may inadvertently trip up users who didn’t see the “NOT” or just misinterpreted the item.
How many cheaters do you see and how do you detect cheaters in your online surveys?