User experience research has many methods.

Some methods, such as usability testing, are used frequently and have been around for decades. Other, more recent additions, such as click testing, are useful variations on existing methods.

Selecting the right UX method means narrowing down the available methods to those that will address (1) the primary research goal and (2) the logistics of the resources needed to implement the method. That is, the approach involves methodological considerations (what addresses the question best) and logistical considerations (what is realistic given the time and budget).

To help with the selection process, we developed a taxonomy of common UX research methods (Figure 1), covered in detail in an earlier article.

Following a taxonomy is the first—but far from last—step in selecting the right method. Here are four ways to help narrow down the methods to use for your next research project.

 

1.    Development Phase

Although there isn’t an official product design book with canonical phases of product development, there have been attempts to break the process of product development into broad phases.

We’ve use a three-stage process, Plan → Design → Release, similar to Rohrer’s (2014/2022) three stages of Strategize → Execute → Assess. Martin and Hannington (2012) provide a more granular set of six phases that break down into substages, but these still align with a three-stage process. Figure 2 shows these models along with several key UX research methods appropriate for different phases.

Regardless of how the development process is outlined, in selecting a UX research method, generative methods tend to happen earlier in the process. These include observational studies and interviews that help identify problems and generate ideas on how to solve those problems for users.

For example, while working at Intuit, we used a type of contextual inquiry called Follow-me-Home to identify problems experienced by small business owners, which informed the requirements and design of the QuickBooks Point of Sale product.

Once they have a prototype based on the generative findings, researchers can switch to more evaluative UX methods to judge how well the proposed solution solves the identified problems. You can’t evaluate something that hasn’t been generated, so evaluative methods tend to come later in the process. This includes qualitative problem discovery, usability testing, click testing, and tree testing used on rough structures to assess navigation (e.g., how people look for features and content). When a product has been released and is in use, you’re able to evaluate its usage more quantitatively using UX benchmarking, log analysis, and A/B testing. Surveys can be used early (assessing attitudes or defining users) and late (assessing attitudes toward experiences).

 

2.    Analysis Focus: Qual or Quant

Another way to decide on appropriate methods is to consider the type of research question. If the questions lend themselves more to the “why” or looking to describe or explain, then a qualitative method works. If a question tends toward understanding how many (to generalize insights), then a quantitative method is generally better.

Of course, most UX research methods aren’t strictly qual or quant but can collect both qualitative and quantitative data. For example, surveys often collect closed-ended rating scale data to generate quantitative metrics and open-ended questions for qualitative comments. Usability testing can identify what problems users encounter in a formative usability evaluation (qual) and, in a task-based UX benchmark study, how long users take to complete tasks and successful completion rates (quant).

Some methods are mostly qualitative (interviews and observation), while others are mostly quantitative (A/B testing or click-stream analysis). Figure 3 shows methods by their qual or quant focus (on the x-axis) and attitudinal to behavioral focus (on the y-axis), adapted from Rohrer (2014/2022).

Look for certain keywords in research questions that suggest approaching the question with qual or quant data. While these words can appear in both types of research questions, some tend to appear more in one than the other. Words such as preference, time, and which tend to suggest quantitative methods. Words such as why, explain, and describe tend to be better addressed with qualitative methods. Some words, such as discover, understand, and expect, can be answered using a mix of both. Don’t use these words as prescriptions but as a guide to how you’ll answer research questions.

 

3.    Data Collection Type

A more logistical consideration is how, if at all, you can collect data (empirical or analytic) and what type of data you can collect (attitudinal or behavioral).

Figure 1 shows that most UX methods are considered empirical; that is, you’re collecting data (through observation or other data collection) from people (users, customers, stakeholders, etc.). But it can be difficult and expensive to find enough of some types of participants (e.g., early-stage startup founders, investor relations managers at Fortune 500 companies).

Empirical methods can be further broken down into either a behavioral or attitudinal focus, as shown in both Figure 1 and the y-axis of Figure 3. Formative moderated usability studies and ethnographic research tend to be more behavioral (seeing what people do). Surveys that ask about satisfaction or ease with prior experience (e.g., retrospective benchmarks) collect attitudinal data.

An alternative to empirical methods is to use analytic methods that, because they are evaluative, also typically happen later in the development process (when there’s either a proposed or an existing interface to analyze for potential problems). Analytic methods include heuristic evaluation, cognitive walkthroughs, keystroke-level modeling (KLM), and practical usability rating by experts (PURE).

 

4.    Time and Money

A final and often determining factor in choosing a method to use is time and money. Some methods tend to take longer and cost more and, in some cases, require specialized software, equipment, or training. You will likely get rich insights and quotes from conducting in-depth interviews of prospective users, but recruiting, scheduling, and compensating may be costly. Synthesizing the data may also take weeks. In contrast, surveying the same group will likely be quicker and cheaper to administer, but the cost would be less depth and little ability to follow up on and probe interesting comments.

Later in the development phase, you may need to understand why a certain feature isn’t being used. An eye-tracking study may provide more precise data on whether people even notice an element (or if it’s at least in their visual field). However, eye-tracking is both expensive (equipment to buy or rent) and time-consuming to properly execute and analyze. Alternatively, a formative usability study with five to ten participants thinking aloud may uncover a mismatch in mental models for a fraction of the cost of an eye-tracker and may be more than adequate to help you understand the issue.

 

Examples of Common Research Questions and Associated Methods

Table 1 shows examples of some common research questions and how researchers can select different methods based on the four considerations of development phase, analysis focus, data collection type, and time and money.

For example, to find out whether there are problems in an interface of an existing app or prototype, we can run a usability test to look for problems (qualitative, empirical) with a few participants. This test would likely take a few days to weeks to set up, recruit participants for, and conduct. Alternatively, we can have two or three evaluators conduct a heuristic evaluation to look for problems that may cause problems for users (qualitative, analytic). This evaluation can likely be conducted in a few days, assuming evaluators are available.

For another example, if we want to know where people look for information or a feature in a piece of software, product, or website, we might use several methods depending on whether there’s a prototype (usability test, click test), a working product (usability test, observation), or just an early concept/image (tree test, click test).

When the question is how to organize content and features (early in the development process), you could conduct a card sort to understand how people start from a set of features and organize them into groups (empirical, behavioral, quantitative analysis guiding qualitative interpretation, generative). If it’s later in the development process and you have an existing structure, you could run a tree test (empirical, behavioral, quantitative) to assess findability, search time, and perceived ease of navigation.

If you’re interested in a deeper dive into the UX methods taxonomy and its use, check out the Essential UX Methods course at MeasuringUniversity™.