Fewer people are relying on print (newspapers and magazines) and even TV for their news.
Despite the evolution of the news “interface” from print and TV to digital, many of the same content sources remain. Traditional TV and print sources of news are also popular online sources. Consequently, we benchmarked the desktop user experience of the following seven news websites:
- Fox News
- Huffington Post
- The New York Times
- The Guardian
- The Wall Street Journal
While users aren’t searching for product prices or booking travel on news websites, they’re still interacting with both the display and form of the content. This interaction involves using search, getting past ads, and scrolling navigation—all of which has an effect on their likelihood to return, recommend, and trust these sources of news.
We recruited 564 participants in July 2017 for a two-part study to measure both existing perceptions and task-based usability. For the perception study, we asked 352 participants to reflect on their most recent experience on one of the news sites they reported visiting.
Participants in the study answered the 8-item SUPR-Q (including the Net Promoter Score) and questions about their prior experience. In particular, we were interested in current users’ attitudes toward the site, whether they felt the news was unbiased and trustworthy, and if it was easy to get up-to-date news.
For the usability study, in a between-subjects design, we asked 212 participants to complete three tasks on one of the news sites.
- Task 1: Go to the website and find the latest article about the Tesla Model 3.
- Task 2: Go to the website and find the latest article about recent healthcare policy.
- Task 3: Go to the website and find the latest article about the Wimbledon tennis tournament.
Participants also answered the 8-item SUPR-Q (including the Net Promoter Score) afterwards as well as questions about their experience completing each task.
Quality of the News Website User Experience: SUPR-Q
The SUPR-Q is a standardized measure of the quality of a website’s user experience. It’s based on a rolling database of around 150 websites across dozens of industries. Scores are percentile ranks and tell you how a website experience ranks relative to the other websites. The SUPR-Q provides an overall score as well as detailed scores for sub-dimensions of trust, usability, appearance, and loyalty.
In the perception study, the scores for the seven news websites have a wide range, from below average to well above average. The perception desktop average SUPR-Q is at the 52nd percentile, meaning this class of websites scores better than 52% of the websites in the database. This average score is somewhat surprisingly low given the high traffic these sites have. BBC leads the group at the 82nd percentile. Fox News has the lowest SUPR-Q score of the group with a score at the 21st percentile.
In the usability study, the desktop average SUPR-Q is at the 46th percentile, very similar to the perception data. Again, BBC is the leader with a score in the 96th percentile while Fox News comes in last at the 10th percentile.
We conducted a similar analysis in 2012 on CNN, Fox News, The New York Times, and The Wall Street Journal, and the results showed scores increased, although only statistically for Fox News. The average SUPR-Q score for this group increased from 43% to 50%. More details are available in the downloadable report.
The average trust score for these sites is slightly below average. The websites with the highest perceived ideological bias (Fox News, Huffington Post, and CNN) have the lowest trust scores at 3%, 11%, and 30% respectively. BBC and The New York Times are the most trusted, both with trust scores at the 77th percentile.
Loyalty/Net Promoter Scores
The 2017 average Net Promoter Score for the news websites is –9% (more detractors than promoters), which is down compared to the 2012 average NPS of 21%. In the era of “fake news” it seems even heavy users of news sites are less inclined to recommend them to their friends.
Why People Use News Sites
Prior experience has an effect on both attitude and usage data. For these websites, CNN and Fox News have the highest reported daily usage (22% and 20% respectively). The other websites are visited between weekly and a few times per month. Participants reported catching up on news (mentioned by 36%) or reading a specific article (32%) as the main reasons for visiting.
Key Drivers of News UX
To understand the key drivers of the news website user experience we conducted a key driver analysis on more specific “components” of the online news experience across both perception and usability studies. We had participants rate their level of agreement to the following 11 items using a 5-point Likert scale (and one 7-point item about branding attitude).
- News categories help me find what I am looking for (categories).
- I can easily search the website to find what I am looking for.
- It is easy to find old news articles that I want to see again.
- It is easy to sort and filter articles.
- Articles and other content are formatted in a clear manner (format).
- The website’s related and recommended articles are helpful (recommended).
- It is easy to save content for later.
- It is easy to share a news article with someone else.
- There are no unnecessary distractions on the website (distractions).
- The homepage contains a good selection of news stories (selection).
- The website contains the most up to date current events (current).
Figure 1 shows the proportion of SUPR-Q scores for seven items identified as statistically significant drivers. Brand Attitude (labeled BrandAtt) is the biggest driver of attitudes toward the UX quality (as it was for the retail analysis). “Format”—the label for “Articles and other content are formatted in a clear manner”—is the next biggest driver of the news website user experience, explaining 12% of SUPR-Q scores. This means a one standardized point increase in satisfaction with the format of the articles and content will increase SUPR-Q scores by 12%. This is a bit more than twice as important as the news categories being helpful (labeled “Categories”) and similar to the homepage contains a good selection of news stories (“Selection”).
The 23% “unexplained” proportion of variance reflects aspects that are unaccounted for with these items and the inevitable variability and measurement error from collecting attitudinal data. However, explaining 77% of the variation in an attitude with just seven items is excellent in the behavioral sciences. It also shows how the 8-item SUPR-Q captures the effect of online experiences.
Biased & Untrustworthy News
We asked participants what problems or frustrations they have with the news sites. The top reasons participants mentioned for disliking CNN, Fox News, and Huffington Post is the perception of bias and untrustworthy news.
- More than a third of participants mentioned they think Fox News is biased.
“They perpetuate false information and have a very biased presentation.” – Fox News user
- 25% of Huffington Post participants said they think the news source presents biased information.
“I think it’s too biased at times and I sometimes get tired of reading between the lines skeptically.”– Huffington Post user
- Slightly fewer CNN users found biased news to be a problem (18%) but it was still the top frustration for users.
“I do my best to take in information and dissect the biases out of it, but it’s hard when that’s the first thing I see in the title.”– CNN user
The perception of bias is, of course, different than actual bias. While these three news websites have generally the highest perceived bias, readers are likely more skeptical about the objectivity of their content—which is probably a good thing in the era of “fake news.”
Readers of news websites with less perceived bias (e.g. The New York Times and The Wall Street Journal) may consequently be more influenced by content as they may perceive it as more objective (whether it is or not).
On a methodological note, we didn’t ask participants about their political leanings, which will likely affect the perception of bias on these sites (those with more liberal leanings who visit Fox News will think it more biased than CNN, for example). In a prior study on social media websites, for example, there were twice as many self-described Democrats as Republicans, which may explain some (but not all) of the lower scores on Fox News. While it’s likely there’s an ideological self-selection (more liberal readers tend to visit the Huffington Post than more conservative Fox News) a future analysis can examine the role of political ideology on perceived bias and UX scores.
Selection of Up-to-Date News
Less than half of participants agreed with the statement “The website contains the most up to date current events” across all websites, except for The Wall Street Journal in which 50% of users agreed or strongly agreed with the statement.
- The Guardian (36%): “I felt it lacking in terms of news compared to other sites.”
- The New York Times (48%): “Content being stale with a lack of full content from the paper.”
Similarly, around half of participants agreed with the statement “The homepage contains a good selection of news stories;” the exceptions being 60% of The New York Times users and 56% of The Wall Street Journal users agreed or strongly agreed with it.
- BBC (53%): “I wish they had more international stories beyond EU information.”
Ability to Find Old Articles
Although overall task completion was high for all three tasks in the usability study, poor search functionality contributed to the lower success rates of Huffington Post and The Guardian.
In the usability study, The Guardian and Huffington Post had the lowest average task completion rates out of all the websites. Neither homepage has an obvious search bar, which users expect to find specific articles.
“It was really difficult to find some things because I couldn’t find a search bar on their website… so it was a little hard to find articles about specific content that wasn’t already on their front page or on a general category like “politics.” – Huffington Post user
“When I first started using the search option, I kept accidentally clicking on ads, and that was not what I was looking for.” – The Guardian user
An analysis of the user experience of seven popular news websites found:
- Loyalty is low. The average Net Promoter Score for these sites is generally lower than the average scores in the SUPR-Q database and lower than our data from 2012. This is likely driven by poor trust and a sense that the news isn’t the most up to date.
- People are on alert for biased news.This is the top reason to dislike news sites. People want to see trustworthy content.
- More diverse and up-to-date topics needed. Less than half the respondents thought the news sources provide the most recent and varied content possible. Some participants mentioned wanting to see more geographically diverse topics.
- The format and story selection matter most. The biggest drivers of the online news experience are the format of the articles and having a good selection of stories on the homepage. Improving (or maintaining) these areas will likely have the biggest impact on perceptions of the UX quality.
- Finding old articles is a challenge. When people aren’t browsing the headlines or following a link, they need to easily find specific articles on any topic.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Denver: Aug. 5th-7th, 2020|