The Importance of Evaluating UX

Jeff Sauro, PhD

Maybe you’ve seen the ketchup bottles.

It’s been a popular meme on LinkedIn. The bottles are intended to be a clever way to explain the difference between UI (user interface) and UX (user experience).

The classic bottle of ketchup is the user interface. The ketchup bottle introduced in 2002 with its characteristic inverted bottle and lid is supposed to be the user experience.

While people spreading the meme have good intentions, this example illustrates why there’s still confusion over the difference between UI and UX. Both bottles provide a user interface and both have a user experience.

The intended message here is that the newer ketchup bottle has a better user experience because it’s focused on a common user problem of ketchup being stuck in the bottle.

And maybe it does have a better user experience. But a user experience is not better just because it’s been redesigned–even if it’s been redesigned for the user in mind.

For example, if you use the upside-down ketchup bottles for a while, goopy ketchup begins to build up in the lid. Even worse, the upside-down orientation puts pressure on the lid so when you open it, ketchup sprays on your clothes!

A user experience is better because improvements to the interface are measurably better. Maybe it allows users to more effectively complete a task in a way that’s quicker or more satisfying, or accomplish something that wasn’t possible before.

Now, I suspect the good folks at Heinz went through several rounds of empirical research to develop the bottle and maybe found it a measurably better experience, despite the messy problem.  Showing the research team and the report is a lot less picturesque than the bottle. But it’s the process that gets to the bottle that makes the user experience better.

So while we all want to have a better ketchup experience, few of us has the responsibility to actually design and evaluate ketchup bottles. However, you can apply a simple framework for evaluating the user experience for whatever you design, be it software, websites, mobile phones, and yes, even ketchup bottles.

Framework to Evaluate and Improve the User Experience

The framework to evaluate and improve the experience for your users is relatively simple and flexible; we use it extensively at MeasuringU. It’s based on the Six Sigma DMAIC methodology that’s been successfully applied across thousands of organizations and disparate products.

  1. Define your outcome measures.

    Your metrics should match your research goals. Findability, completion rates, likelihood to recommend, and purchase rate are all good items to measure. Or maybe ketchup pour accuracy, pour time, and spill factors?

  2. Define the users and what they’re trying to accomplish.

    Who are the users and what are they trying to do? Use a combination of true-intent surveys to understand users and a top-task analysis to separate the many trivial tasks from the vital few. When it comes to ketchup, it’s safe to say it’s about the pour (or squirt). And you’ll want to avoid those few ketchup haters out there in your sample.

  3. Measure the baseline.

    You don’t know if you’ve improved the experience unless you know where it is before changes are made. Measure first using the existing product experience with a sample of users attempting the top tasks and collect the metrics you identified.

  4. Make your changes.

    Implement your new design, feature, branding, or bug fix. Think iteratively here. It’s better to fail fast. I wonder how many iterations Heinz went through before deciding on the specific bottle. Would a skinny top ketchup bottle work better, like the one pictured below?  What new problems does it create?

  5. Measure after your redesign.

    Don’t forget to measure again. It’s like the bread on the evaluation sandwich—you need two slices around the meat! It may seem like your changes are guaranteed to make things better, but did they and by how much? Did you introduce new problems in the process? Here’s where you see whether the new ketchup bottle design’s potential shortcomings are worth the putative improvement in getting the ketchup out of the bottle. If they are, the measures should show it.


Every product, website, or design has a user interface. If it’s used, it has a user experience. What differentiates a good user experience from a bad one is not based on how many awards and accolades it gets (that’s nice of course); instead, it’s based on superior measurable outcomes.

Using the framework of defining metrics, users, and tasks, and measuring before and after changes helps ensure the user experience is quantifiably better.  Now, please someone pass me the ketchup!

    Your Cart
    Your cart is emptyReturn to Shop
    Scroll to Top