Qualitative research is often used as a catch-all phrase to mean not to expect any “hard numbers” from research findings.
While qualitative research is the collection and analysis of primarily non-numerical activities (words, pictures and actions), it doesn’t mean you can’t apply a structured approach to your research efforts.
Usability testing is often characterized as a qualitative activity. Summarizing findings from watching participants in a usability test generates a lot of utterances, actions and images.
In reality, usability testing is (or at least should be) a mixed-method approach: both qualitative and quantitative data are collected.
Here are seven steps to help structure your next qualitative research effort. These have been adapted from Johnson & Christensen 2012 to focus on the type of qualitative research more typical in User Experience such as usability tests or contextual inquires (where an interface is usually involved).
You don’t need to follow these steps linearly, or even include them all in your research, but having these steps should both help structure your next project and help focus the discussion the next time you’re working with someone who proposes a qualitative research approach.
- Determine Research Questions: Focused questions are at the heart of actionable qualitative research. In fact, they are at the heart of good quantitative research as well and play a key role in Lean UX thinking. Are users not using the mobile app because of usability, security concerns or something else? How do users make decisions about how to invest: do they ask a friend, use a financial advisor, or research on their own?
- Design the Study: Getting input from users instead of just internal discussions is an essential first step. With research questions defined, the “What” of the study has been established. Now think Who, When, Where and How. There are logistical advantages and challenges to collecting the data you need. It’s a matter of trading them off.
Who: For in-person sessions, figure out who will be attending: both the type of participant and the facilitators. All the usual guidelines for recruiting participants apply here. It’s often valuable to have a product stakeholder participate along with the facilitators. Back when I worked at Intuit, we’d have both a UX researcher and a product manager attend in-home sessions with users. It was the PM who would ultimately decide what goes into products, so having them see and hear users first hand was impactfulWhere: Determine if you will collect data at participants’ homes, in a coffee shop, in a company conference room, or in a usability lab.
When: Will this occur during one week or over different seasons, buying periods, or product releases? Is it during working hours, weekends, or after work? Days and times impact both the types of participants, their attitudes and potentially your findings.
How: Work out the details of what the participants will do, if anything, and what you need to have ready to collect data. When attending a remote location, we come equipped with a notepad and portable usability lab (basically a laptop, webcam and software).
- Collect Data: The qualitative researcher should assume the role of an unobtrusive observer and have little impact on the settings being observed—whether it be watching participants use existing products at home or in a more controlled lab environment. Qualitative is often used synonymously with small samples, but one can take a qualitative approach to larger sample sizes (more than 50 participants) just as one can take a quantitative approach to small sample sizes (less than 10).
- Analyze Data: Most qualitative research studies generate a lot of data. Creating a system for coding actions and notable quotes helps speed through the process of turning utterances into actionable insights.
- Generate Findings: What was learned from engaging users? This step involves synthesizing the copious amount of notes, videos and artifacts. As many of the responses from participants will be open-ended, there will be a need to identify patterns. For example, when we were interviewing users about why they didn’t pay their credit card bill on their mobile phone, we didn’t ask users if they had security concerns. Instead, many of them voiced the concern in their own words and stories.
- Validate findings: One of the best ways to validate findings is to triangulate using other methods, including surveys or additional sources. One weakness of qualitative research is that it is hard to establish external validity, that is, to provide corroborating evidence that the findings aren’t just the opinion of the researcher. Every researcher, of course, does bring with her biases on the problems with a product or what deserves emphasis in the interview.One approach to minimize this researcher bias is to include a section on the interviewer or principal investigator’s background and how it might influence their conclusions. Having recordings of sessions and detailed notes helps other interested parties come to their own conclusions and can help validate findings. Including verbatims along with the interpretation also helps others see how the conclusions were drawn.
- Report : We usually deliver a power point with backup notes or an appendix with more detailed findings and verbatims. While information comes in sequentially from each participant, we find reporting the data in an inverted pyramid by issue works best. We start with the most important findings, and then note the number of participants that supported these findings and some good quotes to support what we concluded. We also provide confidence intervals around the issue and insight frequency so readers have some idea about the prevalence of an issue in the larger user population.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Fall 2020: Delivered Online|