Usability testing is expensive.
At least that has been the perception.
While technology has improved and gotten cheaper, it’s the technique that’s become more accessible and accepted.
The discount-usability movement helped emphasize the effectiveness of low-cost, smaller sample sizes to find and fix usability problems early and often in the development cycle for smaller budgets. We’ve found that 96% of organizations that have higher UX maturity report using usability testing regularly.
Earlier I described the essentials needed for moderating a usability test yourself and showed that you don’t necessarily need a big budget to conduct usability testing.
However, not all usability tests can be accomplished with a few participants, a low budget, and a do-it-yourself approach. This is especially the case for larger sample benchmark studies, comparative usability tests, or interfaces with a heterogenous population that may require a lot more time and expertise. For such studies, it may make more sense to hire out part or all of a usability test. When doing so, it helps to understand what drives the cost, which can lead to better planning (and hopefully more testing).
Here is an overview of the typical cost of a usability test based on my discussions with other practitioners and our experience at MeasuringU.
The major drivers of cost are the technology platform, participant recruitment/honorariums, and professional services such as facilitation and analysis. Table 1 shows the variables that tend to increase the cost of a usability test. The more items you check, the more the cost (and they often are multiplicative costs, not additive). I’ll provide more detail on each of these components below.
|Factor that increase cost||Increases cost (Check if applicable)|
|Platform techonology costs|
|Mobile and desktop|
|Large sample sizes|
|Hard to find/recruit participants|
|International study and translation|
|Large moderated study|
|Study design & analysis complexity|
|Many presentation options and revisions|
Platform Technology Costs
For unmoderated studies, a major cost to consider will be either the license fee to use a commercial platform or the cost of the service that includes the platform (as is the case with MUIQ). Prices differ depending on the platform needed (desktop and/or mobile), features, and the number of concurrently accessing users and studies. Some companies bundle platform access, participants, and professional service all into one fee.
Commercial UX platforms typically range around $1,000 for a simple web-based-only study to $100K+ for a one-year license to conduct multiple studies on desktop and mobile devices, along with some participant and professional service and training.
Benchmark and comparative studies require a larger sample size and additional analysis and consequently cost more. And if you need to test the difference between more than two interfaces, (e.g. multiple designs or competitors) the cost increases as it adds to setup and analysis time.
Testing Multiple Device Types: Mobile & Desktop
The mobile experience is often substantially different than the desktop, especially when considering mobile apps. If you want to usability test across multiple platforms (desktop, mobile web, iOS and Android native apps) it takes more time to develop a study and a larger sample size to distribute participants across devices. Mobile studies are often more difficult to recruit for as well.
Larger Sample Sizes
Competitive and comparative studies are often between-subjects studies designs, meaning you need a larger sample size, which increases the participant and recruiting costs. Even standalone studies can require larger sample sizes if there’s a need to test with multiple user segments or personas and to generate more precision around your metrics.
The cost of finding and paying people to participate in your study can range from a trivial cost to a substantial one, especially for larger sample benchmark studies and/or when participants are harder to find.
Moderated Recruitment costs
For hard-to-find profiles in moderated studies, external recruiting firms and some agencies will find participants for most major cities in the U.S. and many international locations. They tend to charge between $100 and $300 to find each qualified participant and charge a project management fee. The more you need, the quicker you need them, the harder to find the participants, the more it adds to the cost of a study. For example, for a moderated study with 20 participants with some qualifications (e.g., work for a large company and influence purchasing decisions), you should expect to pay between $12K and $15K for recruitment costs and honorariums (usually cash or check).
Unmoderated Recruitment Costs
For unmoderated usability studies in which you may need dozens to thousands of participants, you need to consider many sources: online panels, internal customer lists, advertising, and bulk email costs. All of which have their costs.
For online panels, the lowest cost per participant will range from between $7 and $10 (general consumer profile) and go up substantially to $40 to $100 (specialized profiles like financial advisors or IT decision makers). Many panels also have a minimum project amount (usually $1,500) and may charge a project management or setup fee.
Customer lists aren’t necessarily cheaper. To use your own customers—especially if you compensate them—you need to factor that expense into the budget along with a way to distribute money to them. You also need a way to send an email to all your customers using a bulk email provider (CCing 1,000 people will get you blocked by most ISPs) that charge per email sent (often $1,500 to $3,000 for 5,000 to 5,0000 emails). Be sure your customers gave you permission to contact them!
International studies usually also increase cost and can limited recruitment availability. Western European and Asian markets are usually at least twice the cost per complete as U.S. markets (they can be as high as $25 to $75 per participant). You also need to factor in the cost of translating the study and back-translating the responses. The longer the study and larger the sample size, the more the translation costs.
Professional Services Costs
If you’re fully outsourcing a usability study to a research agency like MeasuringU, the bulk of the cost will be the professional service time needed to develop the study plan, to host/program a study (for unmoderated) and/or to facilitate the sessions (for moderated), analyze and compile the results, and to pay for participant recruiting and honorariums.
Costs for each phase can range from $10K to $20K. Outsourcing the analysis can be easier because the data has been collected, whereas outsourcing the collection in a moderated study represents the bulk of the work and can be the most difficult and expensive to outsource.
Agency vs. Contractor
If you hire one independent contractor to conduct all phases of a study (from recruiting to facilitating), it often costs less than an agency (participants’ honorarium costs that will be the same).
While this will keep a lower budget, time to complete the project may increase as most independent contractors don’t have their own space or access to all the technology and recruiting sources you may need (and you have only one person working). If budgets are tight and you can’t conduct the study yourself, a contractor might be the next best option (especially more experienced contractors).
Example Usability Study Costs
While there is no simple formula for how much a usability test will cost, the following are some examples at different price points.
- Independent contractor conducting 5-10 moderated sessions on a general consumer profile
- Agency conducting the analysis phase on a simple unmoderated study
- Moderated study with 5-8 general consumer participants for 1-hour sessions
- Unmoderated study using MUIQ on a single website or prototype on a desktop experience for 50-100 general consumers
- Agency conducting the analysis for a competitive benchmark or multiplatform study
- 20-40 moderated participants for a 1-hour study with three different profiles
- A competitive usability benchmark on 2-3 websites (desktop only)
- Moderating 20 small business owners for a 1.5-hour study on a new product
- Moderating 30 accounting professionals on financial software
- Unmoderated competitive mobile app study on 4 apps from 400 participants with dozens of video recordings analyzed for UI problems
- A comparison study with 8 distinct profiles and in-person presentation to marketing executives
- International unmoderated benchmark study with 5 desktop websites for IT decision makers
- Moderating 10-15 participants locally in 4-5 countries
- Pre/post usability evaluation on desktop, mobile web, and native app with IT decision makers
- Purchasing a commercial unmoderated license to conduct multiple unmoderated usability tests for one year
- Moderating 60-80 specialized participants for 1.5 hours with multiple profiles and detailed analyses and metrics
- Competitive benchmark on 4-5 competitors on desktop and mobile in four countries
Thanks to Chelsey Glasson, Cory Lebson, and Jim Lewis for commenting on earlier drafts of this article.
|UX Measurement Boot Camp : Three Days of Intensive Training on UX Methods, Metrics and Measurement Aug. 7th-9th 2019|