An increasing amount of shopping is done online.

Almost $100 billion worth of products were sold online during the two-month Christmas buying season of 2016.

It’s become especially easy for consumers to find the lowest price on a product.

If consumers can’t find needed information, purchase a product easily, or don’t trust the brand or information on a website, they go elsewhere and may tell friends and colleagues about the poor experience.

The quality of the website experience continues to be a key differentiator for online retailers. It’s become increasingly important for retailers to benchmark the user experience and make changes based on the findings.

Benchmarking the Retail Website Experience

To understand the quality of the online retail experience, we collected UX benchmark metrics on ten popular retail websites.

  • Amazon
  • Bed Bath & Beyond
  • Best Buy
  • Etsy
  • Nordstrom
  • Overstock.com
  • Staples
  • Target
  • Walgreens
  • Walmart

A good benchmark indicates where a website falls relative to the competition and is an essential step to take in understanding how any design changes contribute to a quantifiable improvement, which ultimately leads to an increase in website revenue.

A website UX benchmark consists of a tiered approach, as shown in Figure 1.

 

Figure 1: Hierarchy and relationship of data collected in UX benchmark studies.

Figure 1 shows two things. First, overall website attitudes are affected by task metrics, which in turn are affected by interactions on the website. Second, website attitudes are affected by other variables outside the task experience (brand, prior experiences, etc.).

All three levels are important to collect. Our benchmark report collects the outer ring and provides a good comparison set of numbers for all retailers in their benchmarking efforts and usability tests.

The Study

We had 935 participants reflect on their most recent experience from Oct and Nov. 2016 on one of the ten retail websites and answer the 8-item SUPR-Q (including the Net Promoter Score).  Participants also reflected on functionality that contributes to a successful retail experience, including using the cart, product filters, and reviews.

More details on the study are available in the report; here are the highlights of what we found.

Quality of the Overall User Experience: SUPR-Q

The SUPR-Q is a standardized measure of the quality of a website’s user experience and is a good way to gauge website attitudes (the outer layer in Figure 1). It’s based on a rolling database of 200 websites, so scores are percentile ranks and tells you how a website experience ranks relative to the other websites. The SUPR-Q provides an overall score as well as detailed scores for subdimensions of trust, usability, appearance, and loyalty.

In general, the retail websites in this study scored high across all dimensions with the industry average at the 78th percentile. As we saw in 2013, Amazon led the pack and scored at the 99th percentile. It’s not much of a surprise given Amazon’s dominance in the online retail space. Walmart had the lowest SUPR-Q score of the group in the 71st percentile.

Trust

If people don’t trust the brand or information on a website, it affects their likelihood to purchase and recommend. While Amazon had the highest trust scores, Target had virtually the same trust score, despite having a lower overall SUPR-Q score, suggesting a generally high amount of relative trust for this big-box retailer.

While all of Amazon’s scores were high, its lowest score came in the trust subscale. Participants in the study mentioned that sometimes reviews are untrustworthy and it can be difficult to deal with third-party vendors and individual sellers. Two participants articulated this sentiment:

“Some reviews are untrustworthy”

“Dealing with other sellers can be difficult because of inconsistency in the experience”

Similar comments were made about Etsy, which had the second lowest trust scores of the group.

“You never know quality of a product when they all come from different vendors”

“Some sellers are hard to deal with”

Usability

Amazon again had the highest usability scores at the 98th percentile. The usability factor on the SUPR-Q accurately predicts a SUS score; in Amazon’s case, it’s a SUS equivalent score of 85. Both Best Buy and Nordstrom had the lowest standardized usability scores at the 72nd percentile (a SUS equivalent of 77).

Appearance

The visual appeal of a website affects attitude toward the brand. There is even some evidence that the usability of a website affects attitude toward the visual appeal. It’s not surprising to see that Amazon and Etsy both had the highest scores on appearance (as they were among the highest usability scores). Bed Bath and Beyond had one of the lowest appearance scores with participants’ comments reinforcing their ratings:

“[Bed Bath and Beyond has a] cluttered design”

“[The] design is not aesthetically pleasing”

Loyalty/Net Promoter Score

The Net Promoter Score (NPS) continues to be a popular measure of loyalty; for this group, the average NPS was 10%. Amazon again rose above the rest with an NPS of 67%, followed by Etsy with an NPS of 44%. Walmart had the lowest NPS at -35% (more detractors than promoters). Loyalty is highly affected by prior experience and this sample tended to have participants with more experience on Amazon and the least on Walmart (more details on the sample are available in the report).

Browsing vs. Buying

In general, users are looking for products to purchase. If we think about the purchase process using the classic sales funnel, we can break down the activities into browsing, narrowing, and purchasing.

Across the websites, at any given time, 66% of participants report only browsing compared to 17% actually making a purchase. These numbers are virtually unchanged from our 2013 findings. That is, users on retail websites are four times more likely to browse than buy, and this statistic seems to hold both during and before the busy holiday buying seasons.

Digging into the narrowing phase, we found that some 7% of consumers reported looking for sales or deals, 3% comparing prices, and a long tail of tasks including looking at in-store availability (when applicable) and 6% managing a drug prescription (Walgreens and Walmart).

Key Drivers of UX Quality

The SUPR-Q provides a broader measure of the overall website user experience (loosely referred to as UX quality). But the SUPR-Q, like most standardized measures, is not meant to be diagnostic. To understand what elements in the experience are likely affecting SUPR-Q scores, we conducted a key driver analysis on more specific “components” of the retail website experience.

Participants responded to 19 “component” items that asked about shopping and purchasing using a 5-point Likert-type scale and one brand-attitude item on a 7-point scale. Of the 19 items, 9 are key drivers of the SUPR-Q and explain 74% of the variability in scores. The 19 items with key drivers highlighted are shown below. How each website scored on each component is available in the report.

Component key drivers

  • Brand attitude/favorability
  • The cart was easy to find and use.
  • Shipping options were clear and understandable.
  • Products were easy to find.
  • I trust the website with my payment information.
  • Images and names help to clearly distinguish products.
  • Product specifications and customizations are easy to use.
  • Products were listed in a helpful manner.
  • The website had the best value for the products.

Other component items:

  • It was easy to enter my billing information.
  • I am confident that my order was accurate.
  • I knew where I was in the checkout process the entire time.
  • The website clearly confirmed my purchase total and contents.
  • It was easy to sort and filter products according to my needs.
  • Prices are clearly displayed for each product.
  • The website has a good selection of products.
  • The recommended and related products were helpful.
  • It is easy to compare different products.
  • Product reviews and ratings were easy to find.
  • Product reviews and rating were helpful.

Figure 2: Key drivers of the retail website user experience (SUPR-Q).

Figure 2 shows the proportion of SUPR-Q scores for each of the key drivers.  For example, the ease of finding and using the cart (“Easy to find cart”) explains 10% of the SUPR-Q scores—which is twice as important as product specs being easy to use, which explains 5% of the variation in SUPR-Q scores.

The 26% “unexplained” proportion of variance reflects aspects that are unaccounted for with these 9 items and the inevitable variability and measurement error from collecting attitudinal data. However, explaining 74% of the variation in an attitude with just 9 items is excellent in the behavioral sciences. It also shows how the 8-item SUPR-Q captures the effect of online experiences.

Key Drivers by Website

We also conducted a key driver analysis for each individual website to identify the strengths and weaknesses of each. The full drivers by website are available in the report. Some of the website findings were:

Nordstrom has a perceived weakness in how easy it is to compare different products on the website, having a significantly negative impact on SUPR-Q scores. Participant comments also echoed this:

“Can’t compare products and prices side-by-side”

“Difficult to know where within the categories you are”

For Staples, participants found products less easy to find relative to the other websites. The verbatim comments also suggest reasons why the products might be less easy to find:

“Too many ads”

“’Weekly Deals’ and ‘Customer Top Picks’ are distracting and make the site look busy”

Finally on Walmart, the listing of products in a helpful manner was a negative key driver and the verbatim reinforced this:

“Difficult to compare items side-by-side”

“Categories are not well organized and it can be difficult to browse for specific items”