Content is king. Whether it’s for books, movies, audio books, news sites, or entertainment websites. When you have good content people will come and stay. But if people can’t find the content or there’s too much friction in the experience you’ll likely lose your audience even with killer content.
An increasing number of consumers now subscribe to a premium content service like Netflix, Hulu, or HBO GO (often called OTT ). While content is delivered via a TV, mobile device, or website, one of the first touchpoints for these services is often a website.
Benchmarking the Website Experience
To understand the quality of the online experience, we examined the websites of five of the more popular entertainment services and collected benchmark metrics to quantify the experience.
A good benchmark tells you where a website falls relative to the competition and is an essential step in understanding how any design changes contribute to a quantifiable improvement. A website UX benchmark consists of a tiered approach, as shown in Figure 1.
Figure 1: Hierarchy and relationship of data collected in UX benchmark studies.
Figure 1 shows two things. First, overall website attitudes are affected by task metrics, which in turn are affected by interactions on the website. Second, website attitudes are affected by other variables outside the task experience (brand, prior experiences, etc.). All three levels are important to collect.
We had 227 participants attempt a task on one of the five entertainment websites: Vudu, HBO GO, Amazon Instant Video, YouTube, and Netflix. Participants were asked to find an action movie, rated PG-13 with at least a 4 star rating. The task was open-ended enough to allow for self-directed exploration, but had specific criteria to allow participants to interact with filters, search features, and recommendation capabilities.
Participants completed tasks using MU-IQ, our platform for recording task-based survey questions. MU-IQ records participants’ screens and collects a plethora of data on where and how long participants spent on pages across the websites. We looked at a subset of the videos to examine the root causes of the metrics.
More details on the study are available in the report; here are the highlights of what we found.
Overall User Experience: SUPR-Q
The SUPR-Q is a standardized measure of the quality of a website’s user experience and is a good way to gauge website attitudes (the outer layer in Figure 1). It’s based on a database of 200 websites, so scores are percentile ranks and tell you how a website experience ranks relative to the other websites. The SUPR-Q provides an overall score as well as detailed scores for sub-dimensions of usability, trust, appearance, and loyalty.
In general, the entertainment websites in this study scored high across all dimensions. Both Netflix and YouTube led the pack and scored above the 95th percentile. HBO GO scored the lowest of the group with a SUPR-Q score of 71%.
This shows that Netflix has resumed its high place in our database since its temporary fall in 2011 after splitting the business and website.
Net Promoter Score
The Net Promoter Score (NPS) continues to be a popular measure of loyalty. YouTube rose above the rest with an NPS of 65%, followed by Netflix with an NPS of 37%. HBO GO and Vudu had NPS scores that were less than half the leaders.
The SUPR-Q and Net Promoter Score provide metrics about the overall experience. They aren’t meant to be diagnostic. To understand what’s driving these higher experience scores we needed to examine the more detailed task metrics and the behaviors we observed (the lower levels in Figure 1).
Task Ease and Success
While participants found the task easier than average on the SEQ, they found the task easiest on Vudu and YouTube. HBO GO and Amazon Instant Video had SEQ scores around 14% lower than Vudu and YouTube.
Participants spent significantly longer on Netflix at 114 seconds compared to a low of 71 seconds on Vudu. Normally task time is a good indicator of efficiency and would suggest that Netflix is a less efficient experience. However, because these tasks are more exploratory, it may suggest that at least part of that time participants are engaging in the content on Netflix.
Participants generally took a browse first approach when looking for content. Figure 2 is one of the visualizations produced from our MU-IQ platform. It shows the navigation path for Netflix participants. The yellow highlighted areas show when participants used the search feature—most of the time was spent browsing. When participants browsed, the importance of filters and categorization became increasingly important. This was evident in their comments.
Figure 2: Navigation path for participants on Netflix. Yellow boxes indicate when participants used the search feature.
A more detailed examination of the participants’ paths and videos revealed a few findings that are driving the differences in the metrics. In general, having more detail on cover art and having the ability to browse by genre were some of the main drivers of ease and frustration.
Details without Digging
Being able to expand additional program info on the search page by hovering worked well for participants. This worked well on Vudu, for example, as shown in Figure 3.
Figure 3: Vudu included the movie details on the cover art hover, which participants liked having.
The details dialog box in Figure 3 shows a brief synopsis of the movie, the age rating, director, and cast, but gives no information on genre, which would have helped these participants who were looking for an action movie.
“Difficult to find categories easily.”
“There didn’t seem to be any of the relevant categories, action, sci-fi, etc. There were categories, but they were less useful than the general more commonly used ones.”
Vudu, Amazon Instant Video, and Netflix show additional details for a movie when hovering over the cover art, while on HBO GO participants must go to a details page to get any info about the program.
“It should tell rating by hovering over movie.”
Figure 4: With HBO GO, participants had to dig to find more details about the movie.
When participants want to know more about a program, they must stop browsing and go to a details page, which slows down the selection process.
“I don’t like the fact that there were no ratings or genre listings easily visible.”
It’s likely this hidden information drove the lower task and overall UX metrics for HBO GO.
Browse by Genre Helps
Websites that had obvious genre filtering options tended to perform better; however, participants having trouble finding the genre filter encumbered their searching, as shown in Figure 4 from Amazon Instant Video.
Figure 5: With Amazon Instant Video, participants had trouble finding the Browse by Genre feature on the left side.
On Amazon Instant Video, participants expected to find movie genres toward the top of the page, or in the top navigation.
“I think they should put the Genres near the top so I wouldn’t have to scroll down so far.”
While it was difficult to find movies by genre on Vudu, Amazon Instant Video, and YouTube, HBO GO made it easier.
Figure 6: On HBO GO, participants used the Browse by Genre feature that was prominently placed on the top navigation.
Netflix offers a browse drop-down menu, which helps to direct users looking for a particular type of program.
Areas of Frustration
Content is king but so is content about the content (ratings, genre, and movie details). The easier participants could find this information, the easier time they had with the website. The most common areas of frustration stated by participants were around navigation, recommendations, removal of ads (for YouTube), and a better search function (especially for Amazon Instant Video).
More details about each site and their UX and NPS metrics are available in the report.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Denver: Aug. 5th-7th, 2020|