While the term Lean UX has faded in popularity, many of the lean concepts remain, and they are even incorporated into other frameworks such as Agile.
One of the core tenets of Lean UX is to focus on decisions and not deliverables. In project-based UX research, decisions often take a back seat to deliverables. Research deliverables, however, are only an intermediary step towards the ultimate goal—usually, an improved user interface created through iterative formative evaluations.
UX research in general (and usability testing in particular) can get bogged down with logistical details. Complicated screeners, bloated test plans, endless reviews, and PowerPoint presentations—all can get in the way of observing people using a product and uncovering problems that, if fixed, would improve the experience.
One Lean UX solution, to improve the speed from planning to problem fixing, is to have a regular schedule of participants lined up for testing. This has come to be known in many organizations as Rolling Research (RR). As the name suggests, the schedule of testing rolls continuously, like a conveyor belt on a UX research assembly line. This regular schedule acts as a forcing function, so product teams and researchers need to be ready with something to test—usually a prototype or a concept and some questions or hypotheses.
At MeasuringU, we have had the opportunity to work with a variety of clients in different sectors on RR programs. While there aren’t any rigid rules, RR programs generally have three things in common:
1) A regular cadence (e.g., monthly, quarterly).
2) A unifying theme (e.g., product area, type of user).
3) A research problem or product that is ongoing and/or multifaceted.
Apart from these three rules, however, RR programs can vary in many ways, and they range from following teams doing Agile work on a specific product to divisions of a corporation doing exploratory research on a concept under development. If you’re thinking of starting a RR program or are looking for ways to improve an existing one, we’ve found these 13 tips helpful.
1. Start with a plan for people and the process. RR involves a tightly coordinated effort among researchers, the product team, and stakeholders (the people). Creating a process map that lays out who needs to do what by when is a great way to organize the cadence and ensure that collaborators with different perspectives know what is expected from them at each stage of the project. Figure 1 shows an example of a visualization of the process for a monthly cadence.
You’ll also need a plan appropriately structured for qualitative methods (broader questions typically with smaller sample sizes) or quantitative methods (often needing software and more defined research questions and metrics).
2. Create milestones with dependencies. Priorities shift and are not perceived the same way by the different members of a RR program. Engineering’s deadlines may not align with Production’s, both of which could be out of sync with the research team. When creating a map like the one in Figure 1, ensure you mark and clarify the deadlines that must be accomplished by different team members, and be sure to leave enough time between their completion and the next step to avoid scrambling.
3. Help define and refine the research questions. You can’t test everything and answer every question with a few people in a few days. One of the biggest mistakes we see in UX research in general, which carries over to RR, is trying to do too much in addition to dealing with the ambiguities of vague research questions. “Do people understand a design” is a broad question. Understand how to do something? Comprehend functionality? Actually use something? Having examples to help define and refine research questions ensures you’re not just running participants for the sake of it.
4. Set clear and realistic expectations. A regular, rapid schedule benefits product teams by providing feedback consistently and quickly, but this rapid turnaround can also become your greatest adversary without proper planning. To prepare for the inevitable ups and downs, you will want to think carefully about what you can accomplish in a short working window. Work backward from your deadlines and be open to alternative means of preparing and receiving the results (see #12). Quick decision-making and reprioritizing for RR when things don’t go as expected is essential to gain usable insights with each round of testing.
5. Regulate the recruit. In a race against time, coming up with a reasonably standard recruit profile is a good idea. Select those who either can be hired quickly (i.e., gen-pop frequent users of common applications) or can be booked in advance based on the theme of the program even if the specific project is still being finalized.
6. Set the sample size. Most of the research questions tackled in RR focus on qualitative discovery, where interfaces and problems aren’t well defined or understood. For discovery, you can uncover the most common and some less-common issues with 5–15 participants. For answering research questions that require some precision (“Do at least half of existing users think or do…”) or preference questions (“Do users prefer option A or B”), you’ll need more participants—generally more than 20—and you’ll likely need to consider an unmoderated component. We have plenty of tables you can use as look-up guides for sample size planning.
7. Don’t be afraid to use quant. While most of our RR engagements focus on qualitative problem-discovery sessions or in-depth interviews, some research questions are better answered with quantitative methods. In particular, preference questions, where people click, or “how many” questions that need to be generalized can often be answered quickly with automated quantitative methods rather than qualitative 1:1 sessions. We routinely answer research questions with images or prototypes using our MUIQ® platform, especially for international participants.
8. Know how to find the right participants. One of the biggest impediments to UX research in general and RR in particular is finding enough of the right participants. For general consumer profiles, this is rarely a problem, but for more niche profiles (e.g., advertising managers, IT decision makers, social media influencers), it can roll an RR program right off the tracks. To find more specialized recruits, you need to identify panels and professional recruiters to help you, or you can build up an internal list of customers (an internal panel). The more lead time recruiters have on difficult profiles, the better the match will be.
9. Don’t skip the intake and kick-off. RR programs often have the same people involved, but they just as often have new teammates or people unfamiliar with the process. Ensuring everyone is on the same page about what the program can and can’t do is essential to smooth operation. Although it can be repetitive to the old-timers, it’s important to make sure the new blood is familiar with the goals and capabilities of the program.
10. Schedule frequent debriefs. Unlike large benchmark studies, where you have time to pilot and make tweaks as data collection takes place over weeks, most RR data collection happens in a few days. Plan to meet at the end of each day and, if possible, after each participant. You may need to pivot your research questions, change tasks, or modify questions based on research as it’s occurring. This is an extension of the RITE method.
11. Build in flexibility. Prototypes change. Product directions change. Things change a lot. Things change quickly and unexpectedly. Such changes can make it difficult to come up with a comprehensive plan that can be resilient in a time crunch. Our advice is to build in a few clear pivot points, likely after the first one or two sessions, to review all the results, make sure they still align with the goals, and provide an opportunity for changing aspects of the study to get the kind of impact you want. To build in a pivot point, don’t cram all your participants into one or two successive days; create a breather day for review and potential changes.
12. Get creative with the deliverables. To get the most out of your engagement, keep an open mind for how you may present the results. Rather than expecting detailed decks, consider scheduling regular debriefings, asking for top-line results via email, or having teammates attend moderated sessions so that you can get insights in real time. These activities will get you actionable insights and keep the momentum going throughout the program.
13. Make it easy to observe. Set up live sessions where stakeholders (product managers, designers, and developers) can easily drop in. We find that using the same link and password for each Zoom session makes it easy for people to follow along. Additionally, we post recordings with transcripts the same day; some teams find hosting watch parties helpful, or they may have some time at the end of the day to quickly view sessions to focus on problem areas. In our daily emails, we also call out particularly insightful videos, helping our stakeholders narrow in on problems or comments that succinctly address their questions.
