{"id":513,"date":"2019-01-09T04:33:06","date_gmt":"2019-01-09T04:33:06","guid":{"rendered":"http:\/\/measuringu.com\/nps-order\/"},"modified":"2023-04-25T14:30:51","modified_gmt":"2023-04-25T20:30:51","slug":"nps-order","status":"publish","type":"post","link":"https:\/\/measuringu.com\/nps-order\/","title":{"rendered":"Does Changing the Order of the NPS Item Affect Results?"},"content":{"rendered":"
<\/p>\n
Public officials don\u2019t care much about what the general public thinks.<\/p>\n
Voting is the only way ordinary people can have a say in government.<\/p>\n
How much do you agree with those two statements?<\/p>\n
If the order in which those items were presented were switched, would it affect how you responded?<\/p>\n
While most UX and customer research doesn\u2019t involve sensitive topics, does the order in which items are presented in common metrics like the Net Promoter Score matter? To investigate this, we need to first understand order effects in surveys.<\/p>\n
There is a well-documented order effect in surveys. That is, the order in which questions or items are presented in a survey can<\/em> have an effect on how people respond.<\/p>\n The causes of the order effect include being primed by earlier items and a desire for respondents to be consistent (or appear consistent) in responses and avoid cognitive dissonance.<\/a><\/p>\n If people agree to one item, it may push them to agree to another item to appear consistent, even if they don\u2019t necessarily feel that way. Other reasons may be subtler; for example, thinking about more specific items. Even negatively or positively worded items may activate different thoughts and hence affect responses.<\/p>\n In the seminal book by Schuman and Presser<\/a>, where I pulled the above examples that were tested in a study, they describe the order effect as \u201cnot pervasive\u2026but there are enough instances to show that they are not rare either\u201d (p. 74).<\/p>\n The published literature on order effects indicates that it\u2019s not always clear how results will differ when the order is changed and whether there always is an effect. The effect depends at least on the topic being addressed with more sensitive topics being susceptible to the influence of earlier items. Tourangeau and Rasinski (1988)<\/a> describe the order effect as being \u201cdifficult to predict and sometimes difficult to replicate.\u201d<\/p>\n One manifestation of the order effect is something called a general-specific order effect. That is, asking a general question (how satisfied are you overall with life?) versus a specific question (how satisfied are you with your marriage?…with your job?) can have an effect on both the correlation between items and the mean responses.<\/p>\n For example, Kaplan et al. (2012)<\/a>\u00a0analyzed the general satisfaction of military recruiters in one study (n = 6,000) and job satisfaction in another (n = 88), reversing the order of general and specific items. They found lower mean scores and higher correlations when the general items came before specific items<\/strong> (e.g. general: \u201cAll in all, I am satisfied with my job\u201d and specific: \u201cRating the number of hours I work\u201d).<\/p>\n They recommended putting general items first to \u201cobtain the most unbiased estimates of means, standard deviations and correlations between the content of their scales.\u201d<\/p>\n In another study, Auh et al. (2003) <\/a>examined people\u2019s satisfaction with their hair care provider (n = 191). They also found slightly (~6%) lower mean scores when general satisfaction and loyalty items were first<\/strong>. However, in contrast, they found explanatory power was the same or higher when the attribute items (e.g. \u201cMy hair care provider cuts my hair evenly\u201d) came before the general ones (e.g. \u201cOverall satisfaction\u201d).<\/p>\n The authors suggest being consistent and, in some domains\u2014for example, when asking about computers or expensive household appliances\u2014to ask specific attributes first, then overall questions second (which may be more natural for respondents). These two studies from the literature show some similarities (there is a general-specific order effect) but don\u2019t show the same patterns.<\/p>\n There isn\u2019t much research specifically examining order effects in UX or CX. For example, Jim Lewis recently examined the effects of order<\/a> on usability attitudes and didn\u2019t find an order<\/strong> effect when alternating the SUS, UMUX, or CSUQ.<\/p>\n For the Net Promoter Score, there is at least one report<\/a> that if you ask the \u201cwould recommend\u201d question used to compute the NPS early in a survey, the score will be higher than if you ask it later in the survey but no data is given on the size of the effect.<\/p>\n For the SUPR-Q<\/a>, which incorporates the Likelihood to Recommend item, Heffernan<\/a> found that their Net Promoter Score dropped if they asked it after the other seven SUPR-Q items. It\u2019s unclear though whether this effect was a result of the presentation format (one at a time) and\/or from high attrition on their web survey.<\/p>\n We\u2019ll need more data to decide.<\/p>\n To look for evidence of an order effect with the NPS we conducted our own study to see whether scores change based on when questions are asked in a survey. In November 2018, we surveyed 2,674 respondents online and asked them to think about a product or service they most recently recommended. We then asked them how likely they would be to recommend the same product to another friend or colleague (the LTR item).<\/p>\n We varied when we presented the LTR item. It either appeared early in the survey (after about 40 questions about demographics and social media usage, and other Likelihood to Recommend questions about other brands) or later in the survey (after 72 items). We created several batches of surveys. Each batch alternated the order of when participants would see the LTR item with 1,015 participants receiving it early and 1,657 seeing it later in the survey.<\/p>\n We found little evidence to suggest an order effect with this setup. The mean responses differed only slightly (by .1 point) and not statistically significantly between the two groups (p = .9). In this data, the mean LTR value was 8.8 vs. 8.9.<\/p>\n In examining the percentage of respondents that selected each number on the 11-point LTR scale we also saw very similar percentages as shown in Table 1, suggesting there was little movement in both the mean or extreme responses.<\/p>\n\n In examining the open-text responses of the companies people recommended, we found six appeared at least 30 times. I compared the mean LTR response by company and again found no statistical difference or even discernable pattern based on when the LTR item appeared (see Table 2). Half the companies had higher LTR scores when the item was later in the survey and half had higher scores when the item appeared earlier in the survey.<\/p>\n\n In another study we looked at the effect of the SUPR-Q items on the Net Promoter Score (mean of the LTR item). Data was collected from 501 participants in October and November 2018. Participants in the study were reflecting on one of six brokerage websites<\/a>\u00a0or a recent experience on one of six restaurant websites<\/a>. Participants either were asked the Likelihood to Recommend item before the seven SUPR-Q items or after. All eight items were presented on the same page but their order on the page was swapped for roughly half the respondents across the 12 websites (see Figure 1).<\/p>\nGeneral vs. Specific Order Effect<\/h2>\n
Order Effects in UX\/CX<\/h2>\n
Study 1: Early vs. Late NPS<\/h2>\n
\n\n
\n\t \nLTR<\/th> Early<\/th> Later<\/th>\n<\/tr>\n<\/thead>\n \n\t 0<\/td> 1%<\/td> 1%<\/td>\n<\/tr>\n \n\t 1<\/td> 0%<\/td> 0%<\/td>\n<\/tr>\n \n\t 2<\/td> 0%<\/td> 0%<\/td>\n<\/tr>\n \n\t 3<\/td> 0%<\/td> 0%<\/td>\n<\/tr>\n \n\t 4<\/td> 0%<\/td> 1%<\/td>\n<\/tr>\n \n\t 5<\/td> 3%<\/td> 3%<\/td>\n<\/tr>\n \n\t 6<\/td> 3%<\/td> 2%<\/td>\n<\/tr>\n \n\t 7<\/td> 9%<\/td> 7%<\/td>\n<\/tr>\n \n\t 8<\/td> 15%<\/td> 16%<\/td>\n<\/tr>\n \n\t 9<\/td> 16%<\/td> 17%<\/td>\n<\/tr>\n \n\t 10<\/td> 52%<\/td> 52%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n \n\n
\n\t \n<\/th> N<\/th> LTR Early<\/th> LTR Later<\/th> Diff<\/th> p-value<\/th>\n<\/tr>\n<\/thead>\n \n\t Amazon<\/td> 110<\/td> 9.4<\/td> 9.3<\/td> 0.1<\/td> 0.793<\/td>\n<\/tr>\n \n\t Walmart<\/td> 42<\/td> 9.3<\/td> 8.7<\/td> 0.6<\/td> 0.298<\/td>\n<\/tr>\n \n\t Ebay<\/td> 42<\/td> 9.4<\/td> 8.9<\/td> 0.5<\/td> 0.403<\/td>\n<\/tr>\n \n\t Target<\/td> 40<\/td> 9.3<\/td> 9.4<\/td> -0.1<\/td> 0.823<\/td>\n<\/tr>\n \n\t Spotify<\/td> 33<\/td> 8.6<\/td> 9<\/td> -0.4<\/td> 0.395<\/td>\n<\/tr>\n \n\t Hulu<\/td> 30<\/td> 8.7<\/td> 9<\/td> -0.3<\/td> 0.447<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n Study 2: NPS Before and After SUPR-Q<\/h2>\n