{"id":66,"date":"2011-04-19T23:30:00","date_gmt":"2011-04-19T23:30:00","guid":{"rendered":"http:\/\/measuringu.com\/neutral-option\/"},"modified":"2021-01-28T06:29:30","modified_gmt":"2021-01-28T06:29:30","slug":"neutral-option","status":"publish","type":"post","link":"https:\/\/measuringu.com\/neutral-option\/","title":{"rendered":"Survey Items Should Include A Neutral Response: Agree, Disagree, Undecided?"},"content":{"rendered":"<p><a href=\"http:\/\/www.flickr.com\/photos\/winstonavich\/191723300\/sizes\/m\/in\/photostream\/\"><img decoding=\"async\" style=\"float: left; padding: 10px; margin-right: 20px;\" title=\"Neutral Options are nice to have\" src=\"https:\/\/measuringu.com\/images\/shifter.jpg\" border=\"0\" \/><\/a>Few things tend to generate more heated debate than the format of response options used in surveys.<\/p>\n<p>Right in the middle of that debate is whether the number of options should be odd or even.  Odd numbered response scales include a neutral response whereas even ones do not.<\/p>\n<p>Research generally shows that including a neutral response will affect the distribution of responses and sometimes lead to different conclusions.<\/p>\n<p>However, it is less important when assessing usability as you&#8217;re usually more concerned about comparisons over time or against a benchmark than the percent of users who agree to statements.<\/p>\n<h3>The Consequences of Neutral<\/h3>\n<p>The impetus behind not using a neutral response option is that researchers are concerned that having a neutral point attracts respondents who actually slightly lean toward a favorable or unfavorable response.  Having a neutral response would then mask these sentiments.<\/p>\n<p>With an even number of options respondents are forced to decide whether they think favorably or negatively toward an item.<\/p>\n<p>Some early research by Presser and Schuman (1980) found that typically between 10-20% of respondents chose the neutral option when it was provided compared to the same survey when it wasn&#8217;t. Their research was conducted using more politically sensitive questions:<\/p>\n<ul>\n<li>Tendencies on political issues: Liberal to Conservative<\/li>\n<li>Federal Funding for Schools<\/li>\n<li>Penalties for Marijuana Use<\/li>\n<\/ul>\n<p>In fact, Presser and Schuman did find that the biggest shifts came for the political tendency scale (liberal to conservative).  Then as now, there is a certain social acceptability to being &#8220;middle-of-the-road&#8221; politically.<\/p>\n<p>Such neutral options provide an easy out for respondents who are less inclined to express their opinion, but potentially mean a large proportion who favor or oppose a topic aren&#8217;t counted.<\/p>\n<p>Despite the major shifts seen when including or excluding neutral options, Presser and Schuman found that the distribution of responses for the items didn&#8217;t change significantly.<\/p>\n<p>In other words, if there were 20% choosing somewhat-liberal without the neutral option, approximately 20% would still chose somewhat-liberal with the neutral option. The total numbers would be less, but when reporting out the proportions, they found researchers would essentially draw the same conclusion\u2014even on these sensitive topics (or at least the ones they studied with the subjects in their study).<\/p>\n<p>However, later research by Bishop (1987) did show that a researcher, or pollster, would draw different conclusions about the proportion of respondents who favor or oppose an issue based on the inclusion of a neutral response. His research also included politically sensitive topics:<\/p>\n<ul>\n<li>Social Security Benefits<\/li>\n<li>Defense Spending<\/li>\n<li>Nuclear Power<\/li>\n<\/ul>\n<p>Bishop concluded that the type of question (and the type of opinion it elicits) does matter so one should carefully consider the context and the consequences of neutral options. A view also expressed by Parasuraman (1986).<\/p>\n<p>If you find yourself summarizing the proportion of respondents who favor or oppose an item, having a neutral response may matter (as will the <a href=\"https:\/\/measuringu.com\/blog\/scale-points.php\">number of response options<\/a> and labels).<\/p>\n<h3>So what about attitudes about usability?<\/h3>\n<p>There are a number of standardized questionnaires that measure attitudes toward the satisfaction and usability of applications (e.g. software, hardware and websites). The most popular questionnaires and their number of response options are:<\/p>\n<ul>\n<li>System Usability Scale (<a href=\"https:\/\/measuringu.com\/sus.php\">SUS<\/a>): <span style=\"font-weight: bold;\">5 points<\/span><\/li>\n<li>Post-Study System Usability Questionnaire (<a href=\"http:\/\/drjim.0catch.com\/PsychometricEvaluationOfThePSSUQ.pdf\">PSSUQ<\/a> <span style=\"font-size: small;\"><span style=\"color: #ff0000;\">[pdf]<\/span><\/span>): <span style=\"font-weight: bold;\">7 Points<\/span><\/li>\n<li>Software Usability Measurement Inventory (<a href=\"http:\/\/sumi.ucc.ie\/whatis.html\">SUMI<\/a>): <span style=\"font-weight: bold;\">3 Points <\/span><\/li>\n<li>Questionnaire for User Interaction Satisfaction (<a href=\"http:\/\/lap.umd.edu\/quis\/\">QUIS<\/a>): <span style=\"font-weight: bold;\">9 Points<\/span><\/li>\n<\/ul>\n<p>All four standardized usability questionnaires have an odd number of response options implying that the authors of these questionnaires believe that a neutral response is legitimate. An earlier version of the QUIS did contain an even number of items (10) but now contains an odd number.<\/p>\n<p>It seems reasonable that users will genuinely have a neutral attitude about some items that ask about usability or the general satisfaction with a system. Such attitudes typically aren&#8217;t controversial like building more Nuclear Power plants so there&#8217;s less of an incentive to hide beliefs in the neutral-zone.<\/p>\n<h3>How common are Neutral Responses?<\/h3>\n<p>While I&#8217;m not aware of any research on how neutral responses affect the distribution of items in usability questionnaires, I do have plenty of data on the number of users who did chose the neutral response using the System Usability Scale (SUS), the Single Ease Question (SEQ) and the popular Net Promoter Question.<\/p>\n<p>Below is a graph of the percentage of users choosing the neutral response &#8220;3&#8221; from 2,052 users across several dozen websites who responded to the 10 item SUS.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/SUS-neutral-responses.png\" \/><br \/>\n<span style=\"font-weight: bold; font-size: small;\">Figure 1: Percent of 2,052 respondents who chose the neutral &#8220;3&#8221; option for each of the 10 items on the SUS items (items shown below). Error bars are 95% confidence intervals and numbers along horizontal axis are the 10 items in the SUS.<\/span><\/p>\n<ol>\n<li><span style=\"font-size: small;\">I think that I would like to use this system frequently.<\/span><\/li>\n<li><span style=\"font-size: small;\">I found the system unnecessarily complex.<\/span><\/li>\n<li><span style=\"font-size: small;\">I thought the system was easy to use.<\/span><\/li>\n<li><span style=\"font-size: small;\">I think that I would need the support of a technical person to be able to use this system.<\/span><\/li>\n<li><span style=\"font-size: small;\">I found the various functions in this system were well integrated.<\/span><\/li>\n<li><span style=\"font-size: small;\">I thought there was too much inconsistency in this system.<\/span><\/li>\n<li><span style=\"font-size: small;\">I would imagine that most people would learn to use this system very quickly.<\/span><\/li>\n<li><span style=\"font-size: small;\">I found the system very cumbersome to use.<\/span><\/li>\n<li><span style=\"font-size: small;\">I felt very confident using the system.<\/span><\/li>\n<li><span style=\"font-size: small;\">I needed to learn a lot of things before I could get going with this system.<\/span><\/li>\n<\/ol>\n<p>We can see the range is responses that are neutral range from 5% to 22%. Item 5 &#8220;I find the various functions in the website well integrated&#8221; garners the most neutral responses. Perhaps this is because the item doesn&#8217;t <a href=\"https:\/\/measuringu.com\/blog\/sus-websites.php\">apply as well to websites<\/a>.<\/p>\n<p>On the other hand, respondents are less equivocal on item 4 &#8220;I think that I would need the support of a technical person to use the website.&#8221;  Again, most users probably don&#8217;t think they&#8217;ll need to call tech-support when using a website (fortunately!).<\/p>\n<h3>Single Items<\/h3>\n<p>For single item analysis, I have data from 484 users who attempted tasks across various websites and consumer and business software using the <a href=\"https:\/\/measuringu.com\/blog\/single-question.php\">Single Ease Question<\/a> (SEQ)&#8211;a 7-point rating scale with a neutral option.<\/p>\n<p>On average only 9% of users selected the neutral point (shown in the graph below). One reason is that the putative neutral point is not really neutral in the minds of users. Something also seen in the 1994 meta-analysis by Nielsen &amp; Levy where they found the mean response was a 3.6 on 5 point scales, significantly higher than a 3 (Nielsen &amp; Levy 1994).<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/NPS-SEQ-neutral.png\" \/><br \/>\n<span style=\"font-weight: bold; font-size: small;\">Figure 2: Percentage of respondents selecting the neutral option on the Net Promoter question (NPS)<br \/>\nand the Single Ease Question (SEQ).<\/span> <span style=\"font-weight: bold; font-size: small;\">Error bars are 95% confidence intervals.<\/span><\/p>\n<p>The question used to compute the Net Promoter Score (How likely is it that you&#8217;ll recommend this product to a friend?&#8221;), also shown in the graph above, generated similar results as the SEQ.  There are 11 response options with a neutral response. Of the 2925 responses, 8% choose the neutral option of a 5.<\/p>\n<h3>Why neutral doesn&#8217;t matter much<\/h3>\n<p>There are at least three reasons why you shouldn&#8217;t concern yourself too much about the inclusion or exclusion of neutral options in usability questions and probably questions about customer loyalty:<\/p>\n<ol>\n<li><span style=\"font-weight: bold;\">Items are summed, averaged or combined<\/span>: Analysis on questionnaires are averaged or summed across items. Having four response options or five won&#8217;t matter when you take the average. Even on single response questions like the SEQ the mean is reported. For the Net Promoter Question, 4&#8217;s, 5&#8217;s and 6&#8217;s are all considered Detractor&#8217;s, not neutral, so having an even or odd number probably won&#8217;t change scores much.<\/li>\n<li><span style=\"font-weight: bold;\">Relative Comparisons are more meaningful<\/span>: responses to questionnaires aren&#8217;t terribly valuable by themselves. You need to compare the scores to something meaningful.  In the case of SUS for example, <a href=\"https:\/\/measuringu.com\/sus.php\">a score of a 68 represents an average score<\/a>.  Comparing it to a prior version, a benchmark or competitor provides meaning to those numbers and it doesn&#8217;t matter if the questionnaire did or did not have a neutral response on the total score as long as you use the same questions.<\/li>\n<li><span style=\"font-weight: bold;\">The effects of changing response options are usually modest<\/span>: In general the effects of usable or unusable applications tend to outweigh the much smaller effects of scale points, labels, scale directions, neutral responses and <a href=\"https:\/\/measuringu.com\/blog\/extreme-items.php\">poorly written questions<\/a>.<\/li>\n<\/ol>\n<p>So the next time you find yourself debating whether to include a neutral response option in a survey item, consider that the time is probably better spent finding a meaningful comparisons like an industry benchmark.<\/p>\n<p>In fact, it&#8217;s probably as important WHY users are selecting the options they are (Dumas 1998). If you do need to summarize the &#8220;percent who agree&#8221; tread lightly on interpreting those figures as many factors will affect the percentage (and can be exploited by the disingenuous pollster or researcher).<\/p>\n<h4>References<\/h4>\n<div style=\"margin-left: 40px;\"><span style=\"font-size: small;\">Bishop (1987) &#8220;Experiments with the Middle Response Alternatives in Survey Questions&#8221; <span style=\"font-style: italic;\">Public Opinion Quarterly<\/span> (1987) 51(2): 220-232<\/p>\n<p><\/span><span style=\"font-size: small;\">Dumas, J. (1998b). Usability Testing Methods: Subjective Measures: Part II &#8211; Measuring Attitudes and Opinions. October issue of Common Ground, The newsletter of the Usability Professionals&#8217; Association, 4-8.<\/p>\n<p>Presser and Schuman (1980) The Measurement of a Middle Position in Attitude Surveys <span style=\"font-style: italic;\">Public Opinion Quarterly<\/span> (1980) 44(1): 70-85<\/p>\n<p>Parasuraman, A. (1986).  Marketing research.  Reading, MA: Addison-Wesley.<\/p>\n<p>Nielsen, J. &amp; Levy, J. (1994). Measuring usability: Preference vs. performance. Communications of the ACM, 37, 4, 66-75.<br \/>\n<\/span><\/div>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Few things tend to generate more heated debate than the format of response options used in surveys. Right in the middle of that debate is whether the number of options should be odd or even. Odd numbered response scales include a neutral response whereas even ones do not. Research generally shows that including a neutral [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":11042,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"field_56e41332a1ae5","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"","_tribe_ticket_capacity":"","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[9,43],"tags":[10,45],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/66"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=66"}],"version-history":[{"count":0,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/66\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/11042"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=66"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=66"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=66"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}