With the COVID-19 pandemic’s effect on consumer shopping behavior (e.g., increased online shopping for delivery or contactless pickup), mass merchant revenues rose dramatically in 2020 and the first part of 2021.
For example, Target reported a $15B sales growth in 2020, higher than its total sales growth over the past 11 years.
For another example, despite reports of Amazon’s alleged mistreatment of employees and ongoing concerns about its effect on small businesses, research has shown that during the pandemic consumers increased their reliance on Amazon.
These examples illustrate how the forces affecting companies in this sector are very complex. Thin margins for digital sales, balancing between online and physical presence, changes in customer expectations regarding convenience, and consumer reactions to negative press all affect mass merchants.
To better understand this domain, we benchmarked the user experience of the following mass merchant websites to see how people use them and how they might be improved:
We collected SUPR-Q® and Net Promoter data, investigated reasons for using the websites, measured users’ attitudes regarding their experiences, conducted key driver analyses, and analyzed reported usability problems.
Benchmark Study Details
In April and May 2021, 462 respondents (U.S. residents) participated in a retrospective study in which we asked people who had used at least one mass merchant website in the past year to reflect on their experiences.
Participants completed the eight-item SUPR-Q (which includes the Net Promoter Score), answered the UMUX-Lite standardized questionnaire, and answered questions about brand attitudes, usage, and their prior experiences.
Quality of the Mass Merchant Website User Experience: SUPR-Q
The SUPR-Q is a standardized measure of the quality of a website’s user experience, and it is a good way to gauge users’ attitudes. It’s based on a rolling database of around 200 websites across dozens of industries, including mass merchants.
SUPR-Q scores are percentile ranks that tell you how a website experience ranks relative to the other websites (so the 50th percentile is average). The SUPR-Q provides an overall score as well as detailed scores for subdimensions of usability, trust, appearance, and loyalty.
The nine mass merchant websites averaged at the 80th percentile. In this study, Walmart had the lowest SUPR-Q (44th percentile) and, Kohl’s had the best (97th percentile).
Usability Scores and Trust
We asked participants to reflect on how easy they thought it was to use and navigate through the websites. Kohl’s had the highest usability score (98th percentile) and CVS had the lowest (48th percentile). Comments about CVS usability include
“If I am looking for something other than prescriptions it isn’t quite as friendly.”
“Hard to find certain products.”
Kohl’s received the highest trust score (99th percentile). The lowest trust scores were for Amazon (28th percentile) and Walmart (36th percentile). Examples of verbatim comments for these low-trust companies include
“I don’t like the corporation in general so I don’t enjoy purchasing from their website.” (Amazon)
“I just don’t think Walmart is a good company.” (Walmart).
More verbatim comments are available in the downloadable report.
Loyalty/Net Promoter Scores
The mass merchant websites all had positive NPS (more promoters than detractors) with an average NPS of 25%. In general, a better user experience is a good predictor of higher loyalty.
Use of Mass Merchant Sites and Mobile Apps
As a part of this benchmark, we asked participants how they accessed online mass merchants. All participants reported using their desktop/laptop computers (this was a requirement for participation in the survey), with 57% also using mobile websites and 40% using mobile apps.
The typical frequency of use was a few times a year for websites (a few times a week for Amazon) and never for mobile apps (the most frequently used mobile app was Amazon).
Components of the Mass Merchant Website Experience
Across the websites, using a series of five-point scales (see Key Drivers), we computed the top-two-box percentages of items that rate components of the mass merchant website experience. Overall, 88% of participants agreed that product images were clear, and 86% indicated the sites were easy to navigate. At the lower end of the ratings, only 50% thought the sponsored ads were relevant, and only 57% reported a personalized shopping experience.
To better understand what affects SUPR-Q scores and Likelihood-to-Recommend (LTR) ratings, we asked respondents to agree/disagree with the following attributes of the mass merchant websites on a five-point scale from 1 = strongly disagree to 5 = strongly agree. We conducted key driver analyses (regression modeling) to quantify the extent to which ratings on these items drive (account for) variation in overall SUPR-Q scores and, separately, LTR (the rating from which NPS is derived).
- It’s easy to find exactly what I want.
- It’s faster than going to a brick-and-mortar store.
- I can quickly find the brands I want.
- The checkout process is smooth.
- I can find ratings/reviews for products I’m interested in.
- Product images are clear.
- I can find deals and discounts.
- It’s easy to see exactly how much I’ll be charged.
- Shipping costs are clear.
- It’s easy to select sizes and colors.
- It’s easy to find item specifications.
- Product descriptions are clear.
- It’s easy to browse for items.
- It’s easy to search for items.
- The search results are relevant.
- Product recommendations are relevant.
- The sponsored ads are relevant.
- My shopping experience was personalized.
- I can find inspiration for products I might want on the website.
Key Drivers Include Smooth Checkout and Personalized Shopping Experience
SUPR-Q Drivers. The top key driver from the list above was “The checkout process is smooth” (accounting for 17% of SUPR-Q variation). Top two-box scores of this driver ranged from 72–88%. Other significant key drivers included clear product descriptions and ease of finding “what I want.”
LTR Drivers. Of the 19 items in the attributes list, six accounted for significant amounts of variation in the ratings of likelihood-to-recommend (which is the measure from which NPS scores are derived). Consistent with the SUPR-Q results, “The checkout process is smooth” (11%) was one of the two top drivers in this analysis; “My shopping experience was personalized” (11%) was the other.
What to Improve
Key areas of improvement cited by respondents for each website/app include
- Amazon: Attitude toward the company (e.g., “I don’t like the corporation in general so I don’t enjoy purchasing from their website.”)
- CVS: Price (e.g., “Prices aren’t the same online and in-store.”)
- Kohl’s: Refresh site design (e.g., “It could be more modern and aesthetically pleasing.”)
- Macy’s: Filtering difficult (e.g., “Filtering is done after every single selection. That makes filtering take too long.”)
- Rite Aid: Item availability (e.g., “Doesn’t have everything I always look for.”)
- Target: Improve search (e.g., “Sometimes when you search things, it doesn’t really give you what you’re looking for.”)
- Walgreens: Slow site (e.g., “I dislike how clunky and slow the website is in general.”)
- Walmart: Third-party seller concerns (e.g., “I don’t like all the secondary sellers on Walmart that are just listed online. I want to be shopping Walmart only for the most part and have all my stuff backed up easily by them.”)
- Wayfair: Price (e.g., “They do not price match even when they drop the price on an item themselves a day later.”)
Although not necessarily the most frequent issue for all websites, numerous comments were related to finding items (e.g., availability, navigation, search, filtering). For example, Video 1, captured with our MUIQ® platform in an unmoderated picture-in-picture think-aloud session, shows the participant having trouble with Wayfair filters.
Comparison with Results from the 2016 Retail Survey
In 2016, we collected SUPR-Q and NPS data for Amazon, Target, Walgreens, and Walmart (see our 2016 Retail Report and article). From 2016 to 2021, the largest shifts in the NPS were for Walmart, which improved substantially from −35% to 44%, and for Amazon, which dropped from 67% to 22%. These changes were statistically significant (p < .01 using the adjusted-Wald Z-test for comparison of NPS).
For mean SUPR-Q scores, Walmart rose slightly but not significantly (p = .32), but Amazon dropped significantly (p = .003). Examination of the SUPR-Q subscales found no significant changes in usability ratings. For trust ratings, Walmart, Target, and Walgreens were stable from 2016 to 2021, but Amazon declined significantly (Figure 1).
Full details are available in the downloadable report.
Summary and Takeaways
An analysis of the user experience of nine mass merchant websites found that
- SUPR-Q and NPS were mostly above average. The websites had SUPR-Q scores ranging from the 44th to the 97th percentiles (a mean 80th percentile where the 50th percentile is the average for all websites in our current normative database). The average NPS across the websites was 25% (substantially more promoters than detractors). These results show that this sector enjoys above-average UX and loyalty.
- Amazon seems to have a trust issue that affects both UX and NPS. Compared to its ratings in 2016, both SUPR-Q and NPS for Amazon have fallen significantly. Although some of that decline could be caused by regression toward the mean, a comparison of trust differentials for Amazon and three other companies implicates a drop in trust as a contributing factor.
- Mobile app usage was less frequent than mobile web usage. The typical frequency of use was a few times a year for websites (a few times a week for Amazon) and never for mobile apps (the most frequently used mobile app was Amazon). Overall, 40% of respondents reported using mobile apps, and 57% reported using mobile websites.
- Easy checkout experiences drive SUPR-Q and LTR scores. Of the aspects that affect SUPR-Q and NPS, “The checkout process is smooth” was a top key driver for both SUPR-Q and LTR ratings. Ease of browsing, ease of finding “what I want,” and relevant product recommendations were key drivers of the SUPR-Q; knowing exact charges, finding inspiration for other products, and a personalized shopping experience were key drivers of LTR.
Full details are available in the downloadable report.