{"id":49,"date":"2010-12-07T22:00:00","date_gmt":"2010-12-07T22:00:00","guid":{"rendered":"http:\/\/measuringu.com\/cheat-survey\/"},"modified":"2021-01-28T06:29:27","modified_gmt":"2021-01-28T06:29:27","slug":"cheat-survey","status":"publish","type":"post","link":"https:\/\/measuringu.com\/cheat-survey\/","title":{"rendered":"How Many People Cheat In Online Surveys?"},"content":{"rendered":"
<\/a>Remote user research has increased the demands for people willing to take surveys, provide website feedback and participate in usability tests.<\/p>\n In exchange for their time, users are compensated.<\/p>\n One drawback to these professional users is that there are some who are in it just for the money.<\/p>\n Consequently, they may not take your study seriously and “speed” through the questions to receive their remuneration.<\/p>\n For open-ended comments and feedback, it’s pretty obvious who’s not taking your test or survey seriously. Terse comments such as “it’s easy” and “everything is great” are good indications of a cheater.<\/p>\n For multiple choice rating scale questions though, detecting disingenuous answers is not as obvious because all responses to rating scales are usually acceptable.<\/p>\n Adding just a single question to your survey is usually sufficient for detecting the most egregious cheating users. A question like “please select a 3 for this question” would easily tell you whether someone is even half-paying attention.<\/p>\n Of course this only nets the respondents who are completely blazing through your survey. Some professional cheaters have caught on to this trick and look for such questions.<\/p>\n Another option is to include two versions of the same question but with completely opposite wording. For example: “I enjoyed using this website” and “I did not enjoy using this website.”<\/p>\n If a respondent agrees or disagrees to both questions then they are probably not taking your questions seriously. There is a chance users will just misread the question even though they were genuinely answering your questions.<\/p>\n Many questionnaires like the System Usability Scale<\/a>(SUS) already have questions worded both positively and negatively.<\/p>\n It is possible for respondents to agree to two statements such as “I think that I would like to website frequently” and “I thought there was too much inconsistency in the website.” The responses are only somewhat conflicting. However, it is unlikely that a respondent would agree to all 10 items which contain five positive and five negative items.<\/p>\n Looking for all 5’s, 4’s, 2’s or 1’s in the SUS questionnaire can also be a way for detecting these cheaters (as suggested in Albert et al 2010<\/a>).<\/p>\n I looked at five datasets that contained a speeding question (e.g. “Select a 3 here”) and 55 datasets using an online administered SUS (see Table 1 below).<\/p>\nDetecting Cheaters: Please Select a 3<\/h3>\n
Detecting Cheaters: Conflicting Responses<\/h3>\n
Semi-Conflicting Responses<\/h3>\n
How common are cheaters?<\/h3>\n
# of Cheaters<\/th>\n | Total Sample Size<\/th>\n | %<\/th>\n<\/tr>\n |
---|---|---|
2<\/td>\n | 25<\/td>\n | 8%<\/td>\n<\/tr>\n |
1<\/td>\n | 30<\/td>\n | 3.3%<\/td>\n<\/tr>\n |
6<\/td>\n | 269<\/td>\n | 2.2%<\/td>\n<\/tr>\n |
72<\/td>\n | 360<\/td>\n | 20%<\/td>\n<\/tr>\n |
236<\/td>\n | 1962<\/td>\n | 12%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n |