In UX research, both studies and surveys contain a lot of questions. Getting those questions right can go a long way in improving the clarity and quality of the findings. For example, we’ve recently written about how to make survey questions clearer.
And while there are many stories of how the change of a single word in a survey question can lead to different results in social and political research, we are less sure about the impacts on typical UX and CX surveys, where questions are less sensitive or politically charged.
Nevertheless, multiple small impacts due to wording can have a cumulative deleterious effect. In this article, we call attention to nine words that may cause trouble. These aren’t necessarily problematic words, but before you use one, you should double-check that it’s the best word to unambiguously capture the concept you intend to measure.
Three Absolutes: Ever, Always, and Never
When writing survey questions or response options, watch for keywords that make statements absolute. Ever, always, and never are three such absolute words to watch for.
The potential problem with questions phrased with absolute words is that respondents will be more likely to agree or disagree to items phrased in the absolute. That may not be helpful.
Would you ever consider purchasing a license for accounting software?
Taken literally, most people would find it hard to rule out every point in the future—even the distant future—when they or their organization may consider purchasing accounting software.
Of course, some respondents might read ever less literally, meaning just now or in the near future. The challenge is that you don’t know how people may interpret it literally without testing it first. Adding a time frame or removing ever may increase clarity and response consistency. If necessary, you can pilot test both versions independently to see whether the response distributions differ.
I always shop for the lowest price detergent.
There’s likely a time when a frugal shopper might need to purchase a specific detergent that’s not the lowest in price (perhaps because of an emergency). Using in general or typically may distinguish the typical frugal shopper from the literal question reader.
Customer service never responds on time to my requests.
If you get a timely customer service response one out of ten times, technically you need to disagree or even strongly disagree with a statement that may be demonstrably false but doesn’t capture the spirit of the concept (unresponsive customer service).
Again, never is a strong word that may capture the sentiment of some extreme responders, but it may not reflect the feelings of people less likely to agree with the absolute wording. This effect is similar to research we conducted on variations of the System Usability Scale (SUS) that found significant changes to response patterns when we rewrote items to be much more extreme.
Keep in mind this doesn’t mean you can’t use the words ever, always, and never. There may be times when you want to identify the extreme responders to these more absolute statements. In fact, researchers use this technique in Item Response Theory when generating more difficult items (items fewer respondents will agree with). For example, the SUPR-Qm contains the item “I would never delete the app,” but only because this standardized questionnaire contains numerous items that contribute to its overall score, many of which are less extreme.
If a respondent is participating in an interview instead of completing a survey, you can imagine him asking for clarification: “You mean like never-ever respond to my requests, or just about never?” In a sense, surveys are standardized conversations with respondents. Avoid these absolutes unless they’re associated with a research goal.
Four Modal Verbs: Could, Should, Might, and Would
In 3rd grade, if a student asks her English teacher if she can go to the bathroom, the teacher would respond, “I don’t know, can you?” There is a difference between can and may, even though we often use these words interchangeably (much to the chagrin of English teachers).
But what gets used loosely in conversation can be interpreted more literally in surveys. Watch for the following modal verbs, which have subtle differences in meaning: could, should, might, and would.
Could implies something is possible or impossible: Could you recommend the product?
Should implies judgment: Should you recommend the product?
Would implies definite future action: Would you recommend the product?
Might implies permission and possible future action: Might you recommend the product?
These subtle wording differences do not necessarily affect survey question responses. For example, when we tested the difference between will and would in different versions of the LTR item, we found no statistical (or really any observable) differences. We have not, however, tested all possible variants (we have plans, of course). To reduce the possibility of misinterpretation when using one of these four words, make sure you’ve chosen the most appropriate one for your research goal.
You
When using you, be clear whether you refers to just the respondent or includes a whole household.
English has the unfortunate problem of having the same word for the second person singular (just you) and second person plural (all of you). In a conversation, this ambiguity can be cleared up with you guys, you all, or y’all. Harris (2014) makes the point to be clear to the respondent who the you is in the question:
Do you have a subscription to Amazon Prime?
Respondents may wonder whether it counts when they use a parent’s or partner’s subscription. Depending on the research question, it may be better to ask more inclusively (with appropriate response options):
Do you or anyone else in your household have a subscription to Amazon Prime?
If the focus is on the respondent only, then you can clarify that as well:
Do you (not including anyone else in your household) have a subscription to Amazon Prime?
Our
Our seems like an innocuous enough word, but Peterson (2000) suggests that its use may subtly influence the respondent due to the sponsorship bias that can occur when respondents know the company they’re rating is conducting the survey.
Overall, how likely are you to recommend our company to a friend or colleague?
To
Overall, how likely are you to recommend ACME Corp to a friend or colleague?
The latter comes across as more objective and may remove bias. If you find yourself using our, consider replacing it with the company or product name.
Summary
Small changes in wording can subtly change how participants respond to questions, often in unintended ways.
The three absolute words of ever, always, and never may garner agreement from both extreme responders who interpret the questions literally and respondents who interpret the words more loosely, with no way to determine which is which during analysis. Unless there is a compelling need to use them, avoid absolute words in survey questions.
The differences between could, should, would, and might are similar but connote slightly different meanings, so when using them, choose the word carefully.
Two words that seem straightforward but can be tricky are you and our. Clarify the use of you (you alone or your household/family), and avoid the potential bias of our company by using the name of the company or product.
You can use these words when they’re appropriate, but don’t use them thoughtlessly. In the absence of data to help you understand their impact on responses (and their ability to capture the intended concept), you (including everyone in your household) should watch out for these nine words.
