{"id":243,"date":"2014-12-16T22:45:00","date_gmt":"2014-12-16T22:45:00","guid":{"rendered":"http:\/\/measuringu.com\/speeder-research\/"},"modified":"2021-01-28T06:29:59","modified_gmt":"2021-01-28T06:29:59","slug":"speeder-research","status":"publish","type":"post","link":"https:\/\/measuringu.com\/speeder-research\/","title":{"rendered":"How Speeders Affect Online Research"},"content":{"rendered":"<p><a href=\"https:\/\/flic.kr\/p\/9rD99Q\"><img decoding=\"async\" style=\"float: left; padding: 10px; margin-right: 20px;\" title=\"Speeding Through a Study\" src=\"https:\/\/measuringu.com\/images\/slow-down.jpg\" \/><\/a>Speeders are survey participants who finish too quickly, even impossibly quickly.<\/p>\n<p>How much do they affect the quality of online research?<\/p>\n<p>This question has increasing relevance as online research proliferates, including <a href=\"https:\/\/measuringu.com\/blog\/unmoderated-things.php\">unmoderated usability studies<\/a>, since an increasing amount of data comes from <a href=\"https:\/\/measuringu.com\/paid-testers.php\">paid panel participants<\/a>.<\/p>\n<p>With in-person studies, we see each participant&#8217;s engagement level. With data collected remotely, we need another way to determine whether participants have engaged with the study or whether&#8211;as may happen when they are in it strictly for the money&#8211;they have rushed through it.<\/p>\n<p>We use three methods to weed out suspect respondents:<\/p>\n<ul>\n<li>cheater questions<\/li>\n<li>open-ended prompts<\/li>\n<li>duration tracking<\/li>\n<\/ul>\n<p><b>Cheater Questions<\/b>: Traditionally, &#8220;select this response&#8221; questions, scattered throughout a survey, have exposed participants who pay no attention, answer haphazardly, or give &#8220;straight-line&#8221; responses. Because more sophisticated paid participants have learned to look for such questions, <a href=\"https:\/\/measuringu.com\/blog\/cheat-survey.php\">these questions<\/a> now tend to catch only the most egregiously reckless responders. Cheater questions have become the CAPTCHA of surveys&#8211;intended to keep out the robots but not doing the job very well. Instead, these questions often create false positives.<\/p>\n<p><b>Open-Ended Prompts<\/b>: Examining responses in open-ended comments can provide insight into the quality of responses. Unfortunately, examining open-ended responses can be laborious and subjective. What&#8217;s more, participants can still provide a marginally useful answer, such as &#8220;no comment,&#8221; and not be excluded this way.<\/p>\n<p><b>Duration Tracking<\/b>: Online survey software tracks the time that participants spend in a study. This data can help flag speeders. This method is attractive because it&#8217;s more objectively applied and harder for a disingenuous participant to work around.<\/p>\n<p>At MeasuringU we use all three methods when filtering our responses, since each method, on its own, has flaws. Duration tracking tends to be one of the more universally applied methods for winnowing out poor-quality responses, so we wanted to understand what effect speeders have on the quality of the findings. First we looked for extant research on the effects of speeders.<\/p>\n<p>The subject of speeding on data quality was examined in a recent paper, &#8220;<a href=\"http:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/9781118763520.ch11\/summary\">The Impact of Speeding on Data Quality<\/a>,&#8221; by researchers from Germany, led by Robert Greszki. They examined responses to two large datasets (1100+ and 1500+). Participants were asked about their attitudes toward the heads of governments in the US and Germany. Both surveys used paid participants. The median survey-completion time was between 27 and 33 minutes.<\/p>\n<h2>Defining Speeders<\/h2>\n<p>As a first filter, the researchers removed the participants deemed to have completed the survey much too quickly. The fastest respondents took between 2 and 6 minutes. These durations were considered impossibly fast for a survey designed to take 30 minutes.<\/p>\n<p>For the remaining respondents, the next question was How quick is too quick?\u00a0 Participant attentiveness waxes and wanes throughout a study. Instead of looking at total study time, the researchers focused on response time per page (most likely 1-3 questions per page) with both surveys having around 50 pages of questions (yikes!).<\/p>\n<p>They defined &#8220;typical time&#8221; as the median response time per page. Speeders were defined using three buckets: those who completed the studies 30%, 40%, and 50% faster than the median time. The latter group contained participants who answered the questions in less than half the time of the median. This group, contained between 3% and 11% of responses. In comparison, the 30% group had 20%-25% of the sample.<\/p>\n<p>To my surprise, the researchers found no significant differences in response patterns across all speeding categories and across a number of questions\u2014even such politically charged questions as the president&#8217;s handling of the economy. They concluded, using a number of criteria, <b>that excluding speeders didn&#8217;t impact the results in a substantial way.<\/b><\/p>\n<h2>Our Data<\/h2>\n<p>Our experience with speeders is that they add noise to the data and likely respond haphazardly. But we never examined the data closely to see if the fastest responses systematically bias the results.<\/p>\n<p>To see how well these findings applied to user research data, we examined 10 datasets from some recent <a href=\"https:\/\/measuringu.com\/services\/muiq\">unmoderated<\/a> usability studies and surveys we conducted. Participants in all studies were obtained from paid panel respondents (Americans of various ages and both genders). These studies focused primarily on websites of retailers and software manufacturers. The median sample size for the studies was 253, with a range of 104 to 1763.<\/p>\n<p>Following the delineation by the Greszki et al research, we identified participants who completed the studies in less than 50% of the median time. For example, if the median time to complete the study was 500 seconds, 50% speeders were those who took 250 seconds or less.<\/p>\n<h3>Study-Based Speeders<\/h3>\n<p><a href=\"https:\/\/measuringu.com\/services\/muiq\">Unmoderated<\/a> usability studies include a mix of task-based questions and study-based questions. Surveys include only study-based questions. We selected task- and test-based metrics to look for patterns for the fastest respondents. We made 84 comparisons on questions ranging from perceived task difficulty to brand favorability. We found 9 differences to be statistically significant ( p &lt;.05). Given the large number of comparisons, however, we&#8217;d expect chance alone to yield 8 significant differences, so this finding alone isn&#8217;t compelling. See <a href=\"http:\/\/www.amazon.com\/gp\/product\/0123849683\/ref=as_li_ss_tl?ie=UTF8&amp;tag=meausallc-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0123849683\">Chapter 10 in Quantifying the User Experience<\/a> for more discussion on making multiple comparisons.<\/p>\n<h3>Task-Based Speeders<\/h3>\n<p>The surveys Greszki et al examined enabled them to focus on the page-level response time. Our study data isn&#8217;t broken out by page, but we do have task-based data. We looked at 30 tasks across 7 of the datasets which contained task-based metrics of task-ease (<a href=\"https:\/\/measuringu.com\/blog\/seq10.php\">SEQ<\/a>), confidence, and, for a subset, completion rate.<\/p>\n<p>As a first pass, we removed the fastest participants &#8220;superspeeders&#8221; for each task (those who took less than 25% of the median time), keeping those that took between 25% and 50% of the median time. The percentage of speeders for each task then ranged from 4% to 16%, which represented between 14 and 294 people.<\/p>\n<p>We made 74 comparisons between speeders and nonspeeders, of which 20 were statistically significant (p &lt;.05). Given this number of comparisons, we&#8217;d expect 7 to be statistically significant just from chance alone, suggesting that these differences might mean something (because the number of statistically significant differences [20] is greater than what we&#8217;d expect from chance [7]). Digging deeper into the differences, we found that 10 had higher scores for speeders and 10 had lower, suggesting no pattern. Again, this includes a combination of task-completion rate, task ease, and confidence.<\/p>\n<p>With smaller sample sizes in each group (some tasks had only 14 speeders) the power to detect a difference (if one exists) is low. Therefore, using a meta-analytic technique, we ignored the significance levels and looked at the raw differences for each task metric. The results are shown in Table 1.<\/p>\n<table class=\"articletable\" cellpadding=\"3\">\n<tbody>\n<tr>\n<th>Speeder Score<\/th>\n<th>Completion Rate<\/th>\n<th>SEQ<\/th>\n<td valign=\"top\"><b>Confidence<\/b><\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center;\" align=\"center\">Higher<\/td>\n<td style=\"text-align: center;\" align=\"center\">7<\/td>\n<td style=\"text-align: center;\" align=\"center\">18<\/td>\n<td align=\"center\" valign=\"top\">13<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center;\" align=\"center\">Lower<\/td>\n<td style=\"text-align: center;\" align=\"center\">7<\/td>\n<td style=\"text-align: center;\" align=\"center\">12<\/td>\n<td align=\"center\" valign=\"top\">17<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center;\" align=\"center\">% Higher<\/td>\n<td style=\"text-align: center;\" align=\"center\">50%<\/td>\n<td style=\"text-align: center;\" align=\"center\">60%<\/td>\n<td align=\"center\" valign=\"top\">43%<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center;\" align=\"center\">Mean Difference<\/td>\n<td style=\"text-align: center;\" align=\"center\">-7%<\/td>\n<td style=\"text-align: center;\" align=\"center\">.12<\/td>\n<td align=\"center\" valign=\"top\">-.1<\/td>\n<\/tr>\n<tr>\n<td align=\"center\" valign=\"top\">Median Difference<\/td>\n<td align=\"center\" valign=\"top\">-1%<\/td>\n<td align=\"center\" valign=\"top\">.04<\/td>\n<td align=\"center\" valign=\"top\">-.05<\/td>\n<\/tr>\n<tr>\n<td align=\"center\" valign=\"top\">95% CI Around Difference<\/td>\n<td align=\"center\" valign=\"top\">\u00a0(-.16 to .16)<\/td>\n<td align=\"center\" valign=\"top\">(-.17 to .17)<\/td>\n<td align=\"center\" valign=\"top\">(-.41 to .41)<\/td>\n<\/tr>\n<tr>\n<td align=\"center\" valign=\"top\">Total Tasks<\/td>\n<td align=\"center\" valign=\"top\">14<\/td>\n<td align=\"center\" valign=\"top\">30<\/td>\n<td align=\"center\" valign=\"top\">30<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><b>Table 1:<\/b> Difference in task-level metrics between speeders and nonspeeders.<\/p>\n<p>For example, 14 tasks had completion data; of those, 7 showed speeders with higher task-completion rates and 7 had lower task-completion rates. For task difficulty, 18 of the 30 tasks we examined had speeders rating the task slightly easier. For confidence, 17 of the 30 tasks showed speeders rated confidence lower. Notice how the percentages hover around 50% in the &#8220;% Higher&#8221; column, suggesting speeders are just as likely to rate higher or lower than the rest of the responses.<\/p>\n<p>Table 1 shows a subtle but not statistically significant pattern in responses for the speeders. In general, speeders have a slightly lower completion rate (1% lower), rate task ease slightly higher (.12 points higher) and confidence slightly lower (.1 point lower).<\/p>\n<p>Our data also shows that <b>speeders do not substantially affect the responses<\/b>. We would have expected the task-completion rates, at least, to be a lot lower for speeders, as task completions provide objective answers. Surprisingly, there isn&#8217;t much of a pattern in the responses.<\/p>\n<h2>Super-Speeders<\/h2>\n<p>To see whether this non-pattern in responses holds for the most egregious respondents (most of whom would likely be cut due to impossibly fast times) we looked at the extreme group of speeders initially excluded (&#8220;superspeeders&#8221;). These took 25% or less of the median task time. If the median time was 100 seconds, superspeeders took 25 seconds or less to complete the task. The percentage of superspeeders for each task ranged from 6% to 19%, similar to the participants per task of the speeders.<\/p>\n<p>Superspeeder data (Table 2) reveals a subtle but not statistically significant pattern, slightly different than the speeders. In general, superspeeders have the lower completion rate we expected (25% lower but not statistically different than 0), rate task ease slightly lower (.05 points lower\u2014in the opposite direction of the speeder group), and confidence slightly lower (.44 points lower).<\/p>\n<table class=\"articletable\" cellpadding=\"3\">\n<tbody>\n<tr>\n<th>SuperSpeeder Score<\/th>\n<th>Completion Rate<\/th>\n<th>SEQ<\/th>\n<td valign=\"top\"><b>Confidence<\/b><\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center;\" align=\"center\">Higher<\/td>\n<td style=\"text-align: center;\" align=\"center\">1<\/td>\n<td style=\"text-align: center;\" align=\"center\">17<\/td>\n<td align=\"center\" valign=\"top\">13<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center;\" align=\"center\">Lower<\/td>\n<td style=\"text-align: center;\" align=\"center\">13<\/td>\n<td style=\"text-align: center;\" align=\"center\">13<\/td>\n<td align=\"center\" valign=\"top\">17<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center;\" align=\"center\">% Higher<\/td>\n<td style=\"text-align: center;\" align=\"center\">7%<\/td>\n<td style=\"text-align: center;\" align=\"center\">57%<\/td>\n<td align=\"center\" valign=\"top\">43%<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center;\" align=\"center\">Mean Difference<\/td>\n<td style=\"text-align: center;\" align=\"center\">-25%<\/td>\n<td style=\"text-align: center;\" align=\"center\">-.05<\/td>\n<td align=\"center\" valign=\"top\">-.44<\/td>\n<\/tr>\n<tr>\n<td align=\"center\" valign=\"top\">Median Difference<\/td>\n<td align=\"center\" valign=\"top\">-23%<\/td>\n<td align=\"center\" valign=\"top\">.02<\/td>\n<td align=\"center\" valign=\"top\">-.35<\/td>\n<\/tr>\n<tr>\n<td align=\"center\" valign=\"top\">95% CI Around Difference<\/td>\n<td align=\"center\" valign=\"top\">\u00a0(-.35 to .35)<\/td>\n<td align=\"center\" valign=\"top\">(-.38 to .38)<\/td>\n<td align=\"center\" valign=\"top\">(-.76 to .76)<\/td>\n<\/tr>\n<tr>\n<td align=\"center\" valign=\"top\">Total Tasks<\/td>\n<td align=\"center\" valign=\"top\">14<\/td>\n<td align=\"center\" valign=\"top\">30<\/td>\n<td align=\"center\" valign=\"top\">30<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><b>Table 2:<\/b> Difference in task-level metrics between superspeeders and non-speeders.<\/p>\n<p>The lower task completion rate provides some evidence that these superspeeding participants are answering the task-verification question wrong more than the general population of respondents. Future analysis can look to see if this pattern holds.<\/p>\n<h2>Conclusions<\/h2>\n<p>Our research corroborated the results of the earlier study, which indicated no substantial impact of speeders on the results of survey data. We extended that finding to task-based usability data and user-research surveys. The patterns that did emerge could be attributed to chance. Typically, we would throw out data from superspeeders, but our data suggests that doing so would have little impact on the results, (other than, perhaps, showing lower task-completion rates).<\/p>\n<p>We will continue to examine the effects of speeding, looking for a linear relationship between the quality and duration of the response. For now, we can conclude and recommend the following:<\/p>\n<ul>\n<li>Speeders (50% of the median task-time) and superspeeders (25% of the median task time) account for between 5% and 20% of the responses<\/li>\n<li>Responses from speeders and superspeeders appear to have little detectible impact (and no consistent pattern) on the metric averages (including task completion, perceived ease, and confidence)<\/li>\n<li>Nothing indicates that that participants with fast <b>overall study times<\/b> have different responses from the average-speed responses, suggesting that we can&#8217;t use study-completion time as an effective filter. (Exception: a participant spending 2 minutes to answer 50 pages of questions can safely be ignored.)<\/li>\n<li>Effects of speeding are more likely to be detected at the task level, page level, or question level as opposed to the study level, which suggests that you would watch for fast responses as opposed to fast responders.<\/li>\n<li>When filtering data for quality, use multiple methods for examining data quality (cheaters, speeders, and open-ended comments)<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Speeders are survey participants who finish too quickly, even impossibly quickly. How much do they affect the quality of online research? This question has increasing relevance as online research proliferates, including unmoderated usability studies, since an increasing amount of data comes from paid panel participants. With in-person studies, we see each participant&#8217;s engagement level. With [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":1629,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[43,27],"tags":[45,28],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/243"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=243"}],"version-history":[{"count":0,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/243\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/1629"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=243"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=243"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=243"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}