Outsourcing a UX Benchmark vs. Doing it Yourself

Benchmarking is an essential part of a plan to systematically improve the user experience.

A regular benchmark study is a great way to show how design improvements may or may not be improving the user experience of websites and products.

After you’ve decided you’re ready to conduct a benchmark, you’ll need to consider whether to conduct it internally within your company or outsource all or part of it to an external firm like MeasuringU.

Benchmark studies involve many decisions in study design, along with data collection and analysis. If you manage your own benchmark, I’ve included all the relevant details in my book Benchmarking the User Experience.

Here are major factors to consider when deciding whether to outsource or run your own UX benchmark.

• Cost: Outsourcing a benchmark will generally cost more than doing it yourself. You can see the typical cost range for benchmark studies we provide. The cost covers professional services, time needed to design and execute the study, UX measurement expertise, and software/technical services as participant recruiting costs. The more competitors, larger sample size, platforms, and complexity of analysis are the biggest drivers of cost.
• Time: Having an external company conduct the benchmark usually means it will get done faster than doing it yourself. Internal teams can focus on interpretation and buy-in instead of dealing with the minutiae of unmoderated platforms, recruiting, and generating graphs in Excel and slides in PowerPoint.
• Expertise: Depending on the firm you hire, you should get access to a company that has experience conducting benchmarks for similar websites and products as yours and within and across industries similar to yours. Making mistakes with a benchmark study (e.g. the wrong tasks or missing metrics) is particularly pernicious because it has a cascading effect on future benchmark studies. You want to get it right the first time and an external firm should offer that advice.
• Purchasing Testing Software/Licenses: Some external companies have their own benchmark testing software (like MUIQ). This often means you don’t have to commit to expensive software licenses or spend time trying to learn software you may only use once or twice a year.
• Recruiting/Panel Relationships: Finding qualified participants remains one of the challenging parts of benchmark studies (especially for hard-to-find profiles). External companies often have their own panels or existing relationships with vetted national panels and recruiting firms, which allows for faster and targeted recruiting.
• Product Familiarity: Even the most experienced third-party firms are unlikely to know as much about the product or website being tested compared to an internal product team. Just because you outsource a benchmark doesn’t mean you can count on outsourcing all the decisions about which tasks and functionality to include and how to make the right business decisions from the research findings.
• Objectivity: An external company isn’t privy to all the internal debates about what’s working and what’s not. Being free from these political influences means the results are often seen more objectively by stakeholders, which can help with buy-in.
• Company/Stakeholder Buy-in: There can be a “not invented here” attitude in some companies for externally conducted research. This is the opposite problem of the benefit of objectivity and something you’ll have to manage with an external benchmark conducted by a third party.

A good compromise of course is outsourcing part of a benchmark. For example, you can manage the task definitions, participant criteria, much of the study script, and reporting; then you can outsource the hosting, recruiting, and analysis. Using this approach can often provide the best of both worlds (less cost, some objectivity, faster time, and access to software with the buy-in).

Table 1 provides a summary of the pros and cons of conducting your own benchmark versus outsourcing your benchmark. Pluses indicate general positives for selecting internal versus outsourced and the – signs indicate it’s more of a disadvantage.

Whichever route you decide, the earlier and often you begin benchmarking the user experience, the quicker you can understand how changes may or may not be improving the overall design and more effectively manage your UX efforts.

Table 1: Factors to consider when deciding between outsourcing and a do-it-yourself benchmark.