The pandemic has made already-popular video streaming services seem essential.
The popularity makes sense given the relatively inexpensive subscription fees, the lack of long-term contracts, and the many channels of access (through websites, mobile apps, smart TVs), and there is a LOT of content (albeit distributed across different services).
But no matter how good the content may be, if there is friction in accessing content, then subscribers may decide to switch to a service with a better overall user experience.
To better understand these services, we benchmarked the user experience of the following five U.S.-based streaming entertainment websites and mobile apps to see how people are using these services and where they can be improved:
- Disney Plus (disneyplus.com)
- HBO Now (hbonow.com)
- Hulu (hulu.com)
- Netflix (netflix.com)
- Prime Video (primevideo.com)
We collected SUPR-Q data, including Net Promoter scores, investigated reasons for using the services, measured users’ attitudes toward the services, and analyzed problems users reported having with the websites/apps.
Benchmark Study Details
In June–July 2020, we had 328 respondents participate in a retrospective study in which we asked people who used at least one streaming entertainment service to reflect on their most recent experiences.
Participants completed the eight-item SUPR-Q (including the Net Promoter Score) and the UMUX-Lite standardized questionnaires and answered questions about brand attitudes, usage, and their prior experiences. Participants who reported accessing the services on their mobile devices also answered the SUPR-Qm.
Quality of the Video Streaming Entertainment Website/App User Experience: SUPR-Q
The SUPR-Q is a standardized measure of the quality of a website’s user experience and is a good way of gauging users’ attitudes. It’s based on a rolling database of around 200 websites across dozens of industries, including entertainment services.
Scores are percentile ranks and tell you how a website experience ranks relative to the other websites. The SUPR-Q provides an overall score as well as detailed scores for subdimensions of trust, usability, appearance, and loyalty.
The scores for the five entertainment websites/apps were mostly above average. The average is at the 72nd percentile (scoring better than 72% of the websites in the database). When we ran a similar study in 2017, HBO GO had the lowest SUPR-Q, and Netflix had one of the highest. In this study, HBO Now has the lowest SUPR-Q of the group with a score at the 43rd percentile. Netflix and Disney Plus lead the group with scores at the 99th and 93rd percentiles.
Usability Scores and Trust
We asked participants to reflect on how easy they thought it was to use and navigate the entertainment websites/apps. Netflix has the highest score in the group (99th percentile) and HBO Now has the lowest (20th percentile).
The pattern of trust scores is a little different, with Disney Plus and Netflix scoring high at the 90th and 94th percentiles, respectively, and Prime Video scoring the lowest at the 61st percentile.
Loyalty/Net Promoter Scores
The entertainment websites/apps have an average NPS of about 9%, so there are generally more promoters than detractors across the services. In general, better user experience is a good predictor of higher loyalty.
The correlation between overall SUPR-Q and NPS for the five services was reasonably high (r(3) = .815, p < .10). The correlation between brand attitude and NPS was highly significant (r(3) = .95, p = .01).
SUPR-Qm Ratings of Mobile Apps
Respondents who reported using mobile apps to access their streaming video service completed the Standardized User Experience Percentile Rank Questionnaire for Mobile (SUPR-Qm). The SUPR-Qm is a 16-item standardized questionnaire that provides a single measure of the quality of user experience with a mobile app. The 16 items form an ordered scale, from items that are easy to agree with (“The app is easy to use”) to those that only someone who had an exceptional user experience would agree with (“I can’t live without the app on my phone”).
Overall mean SUPR-Qm ratings can range from 1 to 5. Across these entertainment services, the overall mean was 3.57 (90% confidence interval ranging from 3.47–3.67), statistically above the typical average from our database of dozens of mobile apps (p < .10). There were significant differences among the SUPR-Qm means of the entertainment services, with Netflix having the highest mean (3.95) and HBO Now and Prime Video having the lowest means (3.30 and 3.40 respectively).
Use of Entertainment Sites and Mobile Apps
As a part of this benchmark, we asked participants how they accessed their subscriptions and their top tasks. Most participants reported using their desktop/laptop; fewer than half reported using a mobile website. About two-thirds reported accessing content with a smart TV or mobile app.
The typical frequency of use was a few times a week. The top tasks were watching content or browsing for content (52% and 10%, respectively).
Components of the Video Streaming Entertainment Experience
Across the sites, 81% of participants agreed there was a good selection of kids and family content. Overall, only 58% of users think it’s easy to find new movies and shows. Other potential areas of improvement include loading times and settings management (especially parental controls).
“Some problems with loading and the fact that I can’t skip intros.”—HBO Now user
“Difficult to locate show; nothing arranged logically.”—Hulu user
“It’s a little cluttered—not terrible but not very clean either.”—Prime Video user
To better understand what affects SUPR-Q scores, we asked respondents to agree/disagree with the following attributes of their entertainment website on a five-point scale (from 1 = strongly disagree to 5 = strongly agree.) A key driver analysis uses regression modeling to quantify the extent to which ratings on these items drive (account for) variation in overall SUPR-Q scores.
- The website has good recommendations when I’m looking for something to watch.
- It’s easy to find new shows and movies.
- It’s easy to navigate.
- It’s easy to watch on all devices.
- The website is always fast and responsive.
- I don’t experience long loading times when using the website.
- It’s easy to manage settings on the website.
- It’s easy to manage family profiles on the website.
- It’s offered at a good price.
- The website has a good selection of original content.
- The website has shows and movies you can’t find anywhere else.
- It’s easy to manage parental controls.
- There’s a lot of kids and family content to choose from.
- I don’t have to watch commercials or ads.
- I like it better than cable or satellite TV.
Key Drivers Include Brand Attitude, Watching on All Devices, and Load Times
We often find brand attitude to be a major driver of SUPR-Q scores, and streaming services are no exception, with brand attitude accounting for 12% of the variation in overall SUPR-Q. While notable, this is of limited value because brand attitude itself is more of a lagging than a leading indicator. Knowing that brand attitude affects SUPR-Q ratings doesn’t provide much specific guidance about what improvements to make.
The top key driver from the list above was “It’s easy to watch on all devices” (accounting for 14% of SUPR-Q variation). Overall ratings of this item were moderately high, with Disney Plus and Netflix receiving the highest ratings.
The second biggest driver, accounting for 11% of SUPR-Q variation, was “I don’t experience long loading times when using the website.” As noted above, this component of the streaming entertainment experience had relatively low ratings across the services.
Other items that accounted for a significant amount of variation in the SUPR-Q were “There’s a lot of kids and family content to choose from” and “I don’t have to watch commercials or ads.”
What to Improve
The main areas of improvement for each website/app were
- Disney Plus: Poor selection of content (e.g., “I dislike how the number of shows or movies is slim compared to Netflix, but I imagine the gap to close as years pass.”)
- HBO Now: Dislike the design (e.g., “I think that the interface was not refined, in comparison with Netflix.”)
- Hulu: Too many ads (e.g., “How many ads there are even with a paid service.”)
- Netflix: Autoplay videos (e.g., “I dislike how the website makes videos automatically play, as it is distracting and can make you watch something you don’t want to watch.”)
- Prime Video: Unclear what content is included with Prime for free (e.g., “Hard to tell the difference quickly if a movie is free or not, hard for older people.”)
An analysis of the user experience of five video streaming entertainment websites/apps found that
- Most of these entertainment websites/apps had above average UX. Except for HBO Now (43rd percentile), the services scored at or above the 55th percentile in the SUPR-Q (99th for Netflix). NPS reflects this pattern, and only HBO Now had a negative score.
- Users appreciated having a good selection of kids and family content but reported issues finding new movies and shows. The reported difficulties for finding new movies and shows are especially problematic given that the top tasks were watching content or browsing for content.
- Ease of watching on all devices and loading times are key drivers of UX. In addition to brand attitude, other key drivers were the availability of family content, the presence of ads, and ease of use.
Full details are available in the downloadable report.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Fall 2020: Delivered Online|