A core idea behind usability testing is having real people (users) try to accomplish real tasks on your software, websites, mobile apps, or devices. Through observing both what users do and say we are able to both quantify the experience using metrics and understand the problems in the experience.
The emphasis in a formative evaluation is on finding and fixing problems. Formative evaluations are best done early and often in the interface development lifecycle. While the key outcome of a formative evaluation is a list of problems and severity ratings, we often recommend some metrics to complement the more qualitative insights.
MeasuringU provides full services for each phase of formative usability testing.
- Refining research questions, hypotheses, and metrics. We often recommend a mixed-methods approach for task-based usability testing that involves (where possible) a large unmoderated study and a moderated in-depth qualitative component.
- Defining the tasks, writing the scenarios, and recommending pre- and post-study questions.
- Hosting the testing environment for unmoderated studies or setting up software and hardware.
- Recommending the right sample size. There isn’t a one-size-fits-all number; instead the right number is based on how common the problems are. Sample sizes are usually small (between 5 and 10) and tend to uncover most of the more common issues (not necessarily ALL the issues).
MeasuringU can manage and recruit qualified participants. We assist in developing screeners, scheduling participants, and managing participant honorariums and communication.
Formative usability studies include a detailed description of problems, insights, and often some metrics:
- Problems: We provide descriptions of problems encountered by users and use screenshots, quotes, and video highlights to illustrate problems for development teams to fix.
- Insights: Usability tests aren’t just long lists of problems. We also include areas that worked well for users, and compliments and quotes from favorable interactions.
- Prioritization: Not all issues identified in a study can be addressed. We provide problems and insights with a rating of severity and frequency.
- Metrics: The most common metrics for formative studies are completion rates and the Single Ease Question (SEQ) for task-level ease and an overall perception of the experience (often SUS and SUPR-Q).