To fully measure the user experience, you need to measure both.
UX metrics are influenced by more than an interface.
Users have preconceived notions about companies and this affects both how they think and what they do when they interact with a brand—either in a store or online.
Brand attitudes cast a positive halo or a negative shadow on UX metrics, which we consistently see, especially in our benchmark studies. For example, in our retail benchmark study, the brand attitude was the biggest driver of UX perceptions.
It can take time to shift brand perception for the better, while it seems one incident (e.g. a few poorly chosen words from a CEO or a data breach) can quickly tarnish a reputation.
Knowing how your participants feel about the brand (awareness and attitude) and what they associate with it helps improve the quality of your analysis.
One of the most effective ways to measure brand affinity is to simply ask participants to list which words come to mind for a brand or product (called unaided brand affinity). You can then collate them and quantify their frequency.
To illustrate the power of open-ended brand affinity we asked 2,529 U.S.-based online participants to reflect on their experiences with one of the top 50 brands. A lot of “top brand” lists are floating around the Internet; we used this one.
Here’s the process we used:
- Awareness. Ask participants if they’re familiar with a brand (brand awareness).
- List three words. If participants are familiar, ask them to list three words that come to mind when they think about the brand.
- Code these words and short phrases by combining syntactically similar words (e.g. Athletes and Athlete both become Athletes or banking and banks become bank).
- Count the frequency of the words out of the total number of words. For example, in our analysis, Citi had 136 unique words listed by the participants, 22 of them (16%) were “bank.”
- Assign a sentiment rating. We use a simple score to assess sentiment (negative, neutral, and positive). Assign a 1 for a positive sentiment (e.g. fun), a 0 for a neutral sentiment (e.g., “businessman”), and a -1 for a negative sentiment (e.g. “cheap”). Some words have a more obvious sentiment than others (“reliable” is generally good), whereas others could be neutral or negative (“basic” when applied to Honda could be negative, positive, or neutral). If you want to get sophisticated, have multiple evaluators score the sentiment and reconcile differences. We didn’t in this study.
- Compute a sentiment ratio. Add the scores for all the words and divide by the total number of words used. For example, for Honda, five words and their associated sentiment score are:
- durable 1
- awesome 1
- bad -1
- car 0
- vehicle 0
The sentiment ratio from this selection is 1+1+-1+0+0 = 1/5= 20%. Sentiment ratios when calculated this way can range from -100% (all negative words) to 100% (all positive words). Honda’s sentiment ratio was ultimately 63%.
- Look for negative words. For prominent brands, you may find an abundance of positive words. Look for the negative words lurking in the mix as they’re clues to weaknesses. For example, in our study Amazon has a very positive brand sentiment (91% positive) but we still found a few negative words (e.g. “all-consuming”) that may portend a resistance to Amazon’s dominance in so many industries.
Association for the Top 50 Brands
Below are the most common words associated with each of the top 50 brands. For example, Apple is listed as the most valuable brand. The most common word associated with Apple is “expensive” and the most common negative word is “overrated.” It has a sentiment index of 46%, meaning slightly more negative words are associated with Apple. It‘s also interesting that the most valuable brand has a negative word associated with it.
Across the 50 brands, the average sentiment ratio is 39% with a low of 6% (Louis Vuitton) and a high of 91% (Amazon). The high sentiment for Amazon is likely biased on how we obtained our participants (through online panel sources that have a strong connection to Amazon for payment).
|Brand||Most Common Word||Sentiment Ratio||Negative Word|
|American Express||credit card||15%||not accepted everywhere|
|Philips||light bulbs||55%||lower end|
|Hewlett Packard Enterprise||computer||41%||mediocre|
The term “expensive” was by far the most common negative word associated with these brands, showing up as the most common negative word for 10 of the 50 brands and even the most common word for 5 of the brands (Apple, Mercedes-Benz, Louis Vuitton, Audi, and Porsche). Our sample is a non-probability sample of the U.S. population and likely contains many participants who aren’t consumers of the brands but are nonetheless familiar with them.
While you may not be in the business of collecting brand association information for one of the top 50 brands, you should know what your association is with your customers or prospects and you can use those words and your sentiment index to compare to this list. This can be done at a brand or product level. An easy place to start is using the sentiment ratio as explained in this article. To make that ratio more meaningful, if your ratio is above 39%, it’s above average.
In a future article, I’ll dig into more brand metrics, including favorability, satisfaction, and NPS.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Denver: Aug. 5th-7th, 2020|