There are some obvious metrics like revenue, traffic and repeat visitors.
But these are outcome measures. They don’t tell you why revenue or traffic is higher or lower.
Key drivers of these outcomes are how the users perceive and interact with your website. Selling a product that has demand or information that is valuable is of course essential. But it’s rare to have a monopoly on products or information on the web.
What differentiates websites is the customer experience.
Most people would agree that the customer experience is important, but what specifically about the experience helps or hinders a website?
I examined the customer experience research in the Marketing and Usability literature and found some consistent themes. A successful website needs to be usable, credible and visually appealing. This will generate positive word of mouth, repeat visitors and ultimately a more successful website.
The trick is effectively measuring these concepts.
Measuring the Website Customer Experience
When people think of measuring website effectiveness they often think of analytics like click-through rates, purchases, bounce rates and time on site. These are important, but aren’t giving us the whole customer experience picture. An effective way to know if users trust your website, think its usable and visually appealing is to ask them.
Unfortunately it’s not as simple as just asking: “Is this website usable?” or “Do you trust us?” There is a science to asking the right question in a way that generates reliable and valid conclusions. The process is called psychometric validation. It involves identifying different ways of asking users about the constructs of interest then refining statements (called items) to identify the ones that best discriminate good websites from bad.
There are many different questionnaires generating hundreds of items with different rating scales that measure different aspects of website usability, credibility, loyalty and appearance.
I picked 75 candidate items and asked several hundred users to respond to them regarding their recent usage of several websites. I then narrowed the list of items down to around 20 which tended to have the best internal reliability and ability to discriminate between good and bad websites. Finally the top 13 items were selected based on how well they clustered together in a Factor Analysis around the construct they were intended to measure: usability, credibility, loyalty and appearance.
The 13 items together create a new standardized questionnaire called the SUPR-Q. It stands for the Standardized Universal Percentile Rank-Questionnaire. Here are the four essential elements that make for a successfully website and how the 13 SUPR-Q items measure them.
Once you have a product or service that people want (utility), few things matter more than usability. Maybe the website offers a stellar service and looks really slick. If users can’t accomplish what they want to do, find the product they’re looking for, or complete their purchase–it’s like it doesn’t even exist. Especially for eCommerce websites, a usable experience means a profitable experience.
The construct of website usability is measured by having users state their level of agreement to these four items.
1. This website is easy to use.
2. I am able to find what I need quickly on this website.
3. I enjoy using the website.
4. It is easy to navigate within the website.
These four items account for 95% of the 10 item System Usability Scale (SUS) and provide an excellent measure of concurrent validity. That means they provide a reliable measure of usability, specific to websites, more efficiently than SUS.
You’ll notice that two of the items specifically reference findability and navigation—salient attributes of a usable website experience. Item 3 taps into a measure of hedonic quality[pdf]. It’s grouped with three more traditional usability items because it appears users tend to have similarly responses to usability and enjoyment. More technically, it means these items all load on the same factor as uncovered in a Varimax Rotated Factor Analysis.
Credibility (Trust, Value & Comfort)
Does the website sell products and collect credit card information? Are you gathering email addresses to build a subscriber base? If users don’t trust your website, which for many companies is synonymous with their company and brand, they won’t give up their information and website growth is impeded.
These five items measure the construct of credibility–which touches on aspects of trust, value, comfort and confidence.
5. I feel comfortable purchasing from this website.
6. This website keeps the promises it makes to me.
7. I can count on the information I get on this website.
8. I feel confident conducting business with this website.
9. The information on this website is valuable.
Are users talking about your website favorably or are they telling their friends to avoid it like the plague? Will they return to the website and purchase more things or at least see what you have to say? These two items touch upon repeat usage from existing customers and net-new usage from new customers.
10. How likely are you to recommend this website to a friend or colleague? (This is the same question used in the Net Promoter Score).
11. I will likely visit this website in the future.
Is your website looking circa 1998 and is the appearance hindering the experience? Users form impressions of your website based on the appearance in just a few seconds.
12. I find the website to be attractive.
13. The website has a clean and simple presentation.
All the items (except #10) use a five point scale from strongly disagree to strongly agree. Item #10 uses the 11 point format familiar to those who ask it as the Net Promoter question.
Figure 1: The SUPR-Q’s 13 items. All but item #10 are presented using a five point agreement scale.The name of the website can be used instead of “this website.”
Using the SUPR-Q
After the initial validation phase I commissioned several studies to gather data on websites. I surveyed current customers of hundreds of websites to compile a database of 4500 responses between summer 2010 and summer 2011.
The websites come from 18 industries including: Travel, Airlines, Wireless Carriers, Retail, News/Information, Government and Automotive websites. It contains a spectrum from highly usable and trustworthy to difficult to use and utterly chaotic websites. See more details on the SUPR-Q and a list of more of the websites in the database.
The SUPR-Q can be administered after a usability test or to current users of a website retrospectively. In total there are over 200 websites in the SUPR-Q database. Each website contains data from between 30 and 400 users.
In addition to having a reliable and valid instrument for measuring websites, the database behind the SUPR-Q provides a relative percentile rank (this is what gives the questionnaire its name). Instead of working with a raw mean, the global score and each component score are normalized against the database of websites and presented as a percentage.
This way, a score of 75% means the website scores higher than 75% of the websites in the database. Because I’ve commissioned the studies and own the data I can also reveal how each website scores on all the attributes. It provides for an interesting analysis and provides many meaningful comparisons for any website across many industries.
For example, Facebook tends to score highly on loyalty (in the 85th percentile) but very poorly on trust (in the 10th percentile). Zappos, Apple and Amazon have the highest percentile ranks (the highest combination of all factors) while state government website and restaurants are at the bottom.
The database includes many well known brands, some lesser known ones and is refreshed on an annual basis so changes are reflected. For example, Netflix was measured in Q1 of 2011 and has one of the highest SUPR-Q scores (including very high loyalty ratings–corroborating other data). This was before its infamous price increase which has no doubt affected its scores on loyalty and credibility. It will be interesting to see if the scores rebound when I survey the customers again in a few months.
The process of scoring is easy. Collect the responses using any survey application (like MUIQ, Loop 11, or SurveyMonkey) then enter the raw responses into the coded Excel spreadsheet. Immediately you get the normalized percentile rank showing where the website scores relative to the 200 others in the database. You can also see up to 100 individual website scores for each of the attributes of usability, credibility, trust and appearance and even filter by industry.
The SUPR-Q is an efficient and valuable tool for benchmarking your website. It provides a sensitive and reliable measure in 13 easy to administer items. You can use the items freely on your next website evaluation (with attribution) but the real value comes from converting a raw score to a meaningful rank. More information can be found at www.SUPRQ.com.
Contact me to find out more about licensing the SUPR-Q for your next evaluation.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Denver: Aug. 5th-7th, 2020|