When conducting a usability test, gathering input on a design, or testing new features, we strongly recommend the 1:1 moderated session over group sessions.
However, we usually aren’t convincing others to run 1:1 sessions.
Instead, we spend more effort convincing others of combining the traditional in-person 1:1 session with other methods.
We find we can generate more insights at the same or less cost using a mix of remote moderated and unmoderated testing, surveys, and inspection methods.
Gathering input from a user in a face-to-face session provides rich feedback without the biases introduced by multi-person sessions. A skilled moderator with the right tasks can gather immediate feedback on what the users says and what they don’t say or do.
Slight hesitations, long pauses and puzzled looks provide cues for the moderator to probe deeper into potential problem areas. However, those benefits come with some costs.
One-on-one in person sessions can be:
- Time consuming. For every user, you need at least a facilitator’s time and possibly a note-taker as well. With typical sessions lasting an hour along with transitions and late arrivals, it’s hard to run more than five users in a day. Running concurrent sessions means higher costs and the problem of merging notes.
- Harder to recruit for. It’s not easy to get people to come to a facility during the workweek in a particular city. This is especially difficult when you’re recruiting specific profiles like small business owners, accountants or lawyers.
- Biased. Sample bias is ubiquitous in research (even academic research) so there’s no avoiding it. However, participants who have the time and are able to come to a lab in the middle of a workweek may be less representative than those who might have a few minutes in the evening.
- More expensive. Getting people to come in person means higher honorariums costs ($100+) for one hour of time.
- Harder to schedule. You’ve usually got three schedules to work with: the user’s, the facilitator’s, and the client’s, not to mention fixed lab times. Scheduling can be a thankless job.
Despite the drawbacks of in-person moderated testing, we still use it and love it. We simply don’t rely solely on it for gathering insights. Instead, we use a mix of surveys, moderated (remote and in-person) and unmoderated testing to maximize our usability testing budgets.
Here are five ways to maximize usability testing budgets:
- Facilitate fewer sessions. If you have the budget and time to conduct thirty 1:1 sessions, run half the users and apply the remaining funds and time to using additional research methods to triangulate on your questions. This will also increase the confidence in your findings.
- Survey. Surveys are one of the most cost-effective ways of gathering information from users. They aren’t a substitute for a usability test, but they can quickly gather data on general preferences and behavior. We often use a survey to screen participants for a usability test AND to gather data on general behaviors. The in-person time is too valuable to waste on things such as the websites that users have been to, demographic questions, or even attitudes about product features. Gather those ahead of time, follow up with the participant if necessary, and summarize those surveys in the report.
- Use remote moderated sessions. We can usually recruit more easily, with less lead time and have a lower honorarium for participants who can participate from their home or office. As an added bonus we’ve found we can recruit more difficult to find users like doctors and lawyers and such.
- Conduct unmoderated sessions. With the same tasks and interface you can test dozens or hundreds of users for the same recruiting costs as 20-30 in person sessions. This is especially the case when the interface is a website or any prototype that can be hosted remotely. What’s more, services like Usertesting.com and now MUIQ, offer the ability to record the screens of users in unmoderated sessions. Having a video record of where the users get stuck or distracted is just about as good as the lab itself.
- Use Heuristic Evaluations. It’s often not requested by clients or specified in project documents, but having even a single expert review an interface with a set of tasks can reveal about 30% of the issues users encounter in a usability test. This can be especially helpful when users are hard to find. What’s more, Heuristic Evaluations tend to uncover legitimate problems that aren’t encountered during a session because of time constraints, but are nonetheless valuable for development teams.
