It can be particularly daunting knowing that subtle word changes may lead to unanticipated responses. The good news is that you don’t have to start from scratch each time. Instead, you can follow a few steps to help you make decisions and, ideally, reduce the likelihood of errors. Here are seven steps to writing a survey question.
1. Start with the concept/construct.
Concepts and constructs are what you intend to measure. They can be concrete or abstract. There’s a subtle difference between these two terms. Typically, constructs (like satisfaction and delight) are more abstract and multi-faceted than concepts (like age and income). Here are some concepts/constructs that are commonly included in UX/CX surveys:
- Current user of product/service
- Intent to recommend
- Intent to use
- Intent to purchase
Concepts that may seem simple can turn into more complex constructs (e.g., gender).
2. See whether a question already exists.
Chances are you’re probably not the first researcher to be interested in a particular concept, so others may have already written appropriate questions that you can find in published research. This step is especially important when you’re measuring abstract constructs such as usability, appearance, trust, and delight. Our advanced research platform, MUIQ®, includes several commonly used questions and standardized questionnaires for just this reason.
When you can, adopt or adapt an existing question and its response options, especially if it’s both reliable and valid. However, the more you deviate from the original item, the more you risk deviating from what you intend to measure. (For some guidance, see “Can You Change a Standardized Questionnaire?“)
For example, if you’re interested in measuring task-level ease, the SEQ® is a reliable and valid measure. If you want to measure trust, two items from the SUPR-Q® standardized questionnaire tap into trust and credibility. To measure delight, consider using two items to measure the expectation and surprise components of delight. For voice quality (intelligibility, naturalness, prosody, social impression), consider the MOS-X2.
If the question you need doesn’t exist, continue with the following steps.
3. Determine the type of question.
If you have a concept in mind and you’ve ruled out using an existing question, it can be helpful to first determine what type of question content is most appropriate for the concept/construct. Knowing the type can help with the development of typical response options and reveal common pitfalls to avoid. Survey questions fall into four types of content:
- Attributes: Typically, demographic-like questions describing attributes of the respondents (e.g., age, income, education) or descriptions of other people or things (e.g., employees at a company).
- Behavioral: Self-reports of past or current behavior, which usually include a reference period (e.g., last six months)—don’t confuse these with observing behavior.
- Abilities: An assessment of a person’s knowledge or skill, including the ability to complete usability tasks (e.g., finding information).
- Thoughts/Sentiments: Questions that ask for a respondent’s attitude, judgments, or preferences. Satisfaction, brand, and ease questions are common examples. A lot of concepts fall into this group.
4. Brainstorm ideas.
To brainstorm ideas, come up with different words, phrases, or ideas to describe the concept or construct you want to measure (by yourself, if necessary, but ideally with a group.) Be concrete, and take the perspective of the people who will use the data you collect.
For example, say you’re interested in how well people understand a new product. Specific, concrete questions might concern the product’s features, its pricing, how well it integrates with other products, its ease of use, the discoverability of its features, how quickly users complete tasks, or its trustworthiness. If necessary, you can break these concepts down into even more detailed questions. (For example, to what extent is a trust rating due to attitude toward the brand reputation or to an actual positive or negative experience?)
5. Interview the target respondents.
Earlier, we cited the importance of using the respondent’s language when writing questions. This step provides the opportunity to validate the words and phrases you might have come up with from brainstorming. For example, do respondents understand the terms you’re planning to use? Do they use different terms from what you expected? And what are you missing or misrepresenting?
6. Craft the question.
With ideas and terms refined about the concept, it’s time to craft the questions.
Despite calling them, well, questions, survey questions can be both questions and phrases, so we often refer to them using the broader term item. Earlier, we described the anatomy of a survey question as having a stem and one or more response options. The stem must contain a request for an answer and may include four other parts (introduction, topic information, instructions, opinions of others), which aren’t always used.
But how do you write the question? Saris and Gallhofer (2007) provide some guidance and suggest three ways to turn a concept into a question.
Use a simple or direct request.
To create a simple or direct request, invert the subject and auxiliary verb of a statement. For example, if you’re interested in someone’s age, the statement “I am 25 years old” inverts to “Are you older than 21?” Some other examples are
“Overall, that task was easy to do” becomes “Overall, was that an easy task?”
“I would recommend that product” becomes “Would you recommend that product?”
“I would need the help of a technical person” becomes “Would you need the help of a technical person?”
Form an indirect request.
Instead of asking a question, present one or more statements and then have people rate them with agreement (Likert-type) items. For example, the UMUX-Lite uses two agreement items (Figure 1).
Use wh- requests.
Finally, you can use the classic “wh” questions (e.g., who, what, when, and which) and pose the question directly to the respondent.
- What is your age?
- Who is your cell phone provider?
- Where do you reside?
- Why did you select the number you did?
7. Select the response option(s).
Selecting the best response options requires careful consideration. It’s a common stumbling block for survey developers, as there are many options to consider. We recommend starting with the goal of the questions (what are you going to do with the data and how will you present it) to guide the decision. We’ve put together a decision tree to help narrow the choice for response options. Table 1 has a few examples of selecting the right response options based on the goal.
We’ve identified four key categories to help you get started:
- Discovery: When discovery is the primary goal, use open-ended items.
- Measurement: When collecting measurements, the key distinction is whether the required data is categorical or not.
- Categorical: Categorical data can be ordered (e.g., age groups) or unordered (e.g., streaming video services), allowing selection of multiple items when choices aren’t mutually exclusive or restricting selection to one item when choices are mutually exclusive.
- Not categorical: Noncategorical question types can be classified as ranking, allocation, or rating. Of these types, rating scales are the ones most commonly used in UX and CX research.
For example, if you want to know how often respondents use a product or service, you’d craft a frequency question (Goal > Measurement > Categorical > Ordered > Frequency > Frequency Item), such as the one shown in Figure 2.
If your goal is to understand the relative importance of attributes or activities where participants can allocate amounts such as points, dollars, or percentages to attributes or activities, you could use a fixed-sum question (Goal > Measurement > Not Categorical > Allocation > Fixed Sum Item), similar to the example in Figure 3 from MUIQ.
Researchers can avoid writer’s block when writing survey questions by following seven steps (and sometimes all you need are the first two):
- Start with the concept/construct.
- See whether a question already exists. (If it does, you’re done; otherwise, continue.)
- Determine the type of question.
- Brainstorm ideas.
- Interview the target respondents.
- Craft the question.
- Select the response options.
It isn’t always easy to write survey questions, but it can be easier when you have a set of steps to fall back on.