But detecting these small differences (e.g. 2%–10% changes) through behaviors and attitudes has generally not been feasible from traditional lab-based testing due to the time and costs of recruiting and facilitator costs/time.
With unmoderated testing, organizations can now collect data from hundreds to thousands of participants quickly and from around the world to differentiate real change from random noise.
It’s also opened a more ecologically valid window into the typical online users’ experience (usually at home on their own computer with many distractions instead of in a controlled lab). To take full advantage of unmoderated testing though, you need a technology platform that ideally presents tasks and collects data for you.
In 2008, Tullis, Albert, and Tedesco’s book Beyond the Usability Lab listed four commercial options for unmoderated studies. A decade later, only two of the original platforms featured in their book still exist (Loop11 and UserZoom), but others have emerged (e.g. UserTesting, Validately, and TryMyUI to name a few).
These newer platforms offer the capability to collect data quickly, primarily from small-sample think-aloud videos, but they really aren’t meant for larger scale studies. Consequently, we weren’t satisfied with the capabilities and flexibility available in any of these unmoderated platforms, so we built a new one: MeasuringU Intelligent Questioning (MUIQ) platform. The MUIQ platform provides the key features we found we needed to conduct rich research from prototypes to fully functional websites on desktop and mobile devices for both small and large sample sizes.
The following are the main platform features we’ve found important for conducting effective unmoderated studies, especially UX benchmarking studies. Choosing the right platform for your unmoderated needs will require balancing features, price, and flexibility.
- Task-based: This capability displays a taskbar along with a website (Figure 1) and only some platforms provide this feature. An additional benefit is if you can configure the taskbar so it doesn’t interfere with key elements on the page (Figure 2). Being able to display tasks onscreen is probably the most essential feature for effective task-based research.
- Branching and logic: When different segments of a company’s population (e.g., customers versus financial advisers) are included in the same study, you’ll want to display only relevant questions to each participant segment and branch or add conditionals to keep studies smooth and short for each segment.
- Question screen outs: You need the ability to find participants’ answers to specific demographic questions, such as their prior experience with a brand, income, or age, to ensure you’re targeting the right type of participants for your study. This is especially important when using online panels.
- Quotas on questions: To manage the right balance of participants (e.g. 50% experienced and 50% new users), you need to have question-level quotas. This simple feature is surprisingly absent from most unmoderated platforms (but available with MUIQ).
- Screen recording: Having a recording of a participant’s screen for both desktop and mobile web, and ideally mobile native apps, is one of the best ways to understand interactions and validate task successes. We love this feature. While you can’t expect to watch thousands of videos, watching those with high task-failure or long task-times can offer immediate insights into problems.
- Participant selection: There seems to be a trend in bundling participants with commercial providers. (For example, UserTesting and UserZoom are increasingly bundling this cost with the platform.) While this provides conveniences, you’ll want the ability to select your own panel or participants both for specialized participants and to avoid oversampled “professional users.”
- Click paths: Knowing where participants click allows for building many different types of visualizations, such as heat and click maps, that show how participants move through a website. Due to browser security restrictions, collecting clicks on all platforms requires a plug-in or embedding code and only works on some browsers (usually Chrome and/or Firefox).
- Mobile task-based: If applicable to your study, you’ll want all the previously discussed features available for mobile web and, as much as possible, for mobile native apps.
- Multiple browser support: As of this writing, Chrome is the dominant browser (around 60% of the desktop market), but not everyone uses it, especially at large businesses (which often don’t have a choice). You’ll want to let participants have a choice of browsers (Firefox, Edge/IE, Safari), even if this means click data won’t be collected.
- Custom & advanced question types: Card sorting, conjoint/max-diff, file-upload, visual analog scales, and integrated standardized metrics like SUPR-Q and SEQ benchmarks address the complex nature of UX research studies that often blend into market research.
A lot of technology is required to deal with constantly changing web browsers and mobile devices (Android and iOS). Consequently, unmoderated testing platforms aren’t free and can be a considerable cost to account for when embarking on an unmoderated test.
For unmoderated studies, a major cost to consider will be either the license fee to use a commercial platform or the cost of the service that includes the platform (as is the case with MUIQ). Prices will differ depending on the platform needed (desktop and/or mobile), features, and the number of concurrently accessing users and studies. Some companies bundle platform access, participants, and professional service all into one fee.
Commercial platforms typically range from around $1,000 for a simple web-based-only study to $100K+ for a one-year license to conduct multiple studies on desktop and mobile devices, along with some participant and professional service and training costs included.
To help with your unmoderated planning, Table 1 provides some idea about the range of costs and comparisons for unmoderated platform options for a few providers. I’ve provided guidance on the features by placing the numbers that correspond to each feature from the list above (e.g. 1 means supports task-based questions). I’ve divided the platforms into three types that highlight their dominant features: survey, task-based, and think-aloud. The more $ signs, the more the cost.
|SurveyMonkey||Survey||$||2, 6||Limited functionality and unable to see what participants are doing. Good for retrospective benchmarks or with simple task instructions (no metrics). Free to $100/month for more advanced features.|
|Qualtrics||Survey||$$||2,3,4,6,7,9||Advanced survey capabilities, question types (no card sorting), and basic screen clicks; no task-based studies or screen recording. Requires annual license >$20K.|
|MUIQ||Task-Based||$$||1,2,3,4,5,6,7,8,9,10||Advanced logic, branching, card-sorting, customer questions and complex randomization. No license is required; includes service fees to program, host, and analyze (starting at 10K). Includes screen videos on desktop, mobile web, and native app.|
|UserZoom||Task-Based||$$$||1,2,3,5,6,7,8,9||Fully featured but requires an annual license that starts at $50K-$100K+. Includes screen videos, think-aloud videos, and card sorting, but no support for conjoint/visual analog scales.|
|UserTesting||Think-Aloud||$$$||1,5,6,8,9||Limited customization options and more difficult to find specialized users. Requires an annual license of $50K-$100K+. Includes think-aloud videos from UserTesting’s panel, which can be very convenient for quick consumer profiles. No click paths.|
|Validately||Think-Aloud||$$||1,5,6,7,8,9||Similar to UserTesting, but limited in the number of available questions and tasks. Requires an annual license, $10K+. Includes think-aloud videos from Validately’s panel or own participants.|
|TryMyUI||Think-Aloud||$||1,5,6,7,8,9||Similar to UserTesting, but limited in the number of available questions and tasks (collects click-paths). Supports one-off testing, < $50 per user (unlike UserTesting and Validately, which require a subscription).|
Table 1 isn’t an exhaustive list. Platforms constantly change features and pricing and there are new entrants (and consolidation) all the time. Having some unmoderated platform is necessary but not sufficient for conducting effective online UX research. Even the best technology isn’t a substitute for well-constructed tasks, questions, and analyses—something we cover in detail at the UX Measurement Boot Camp.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Denver: Aug. 5th-7th, 2020|