The SUPR-Q (Standardized User Experience Percentile Rank Questionnaire) is a standardized questionnaire that measures the quality of the website user experience.
It’s an 8-item instrument that’s gone through multiple rounds of psychometric validation and is used by hundreds of organizations around the world. Here’s a list of 10 essential things to know about the SUPR-Q.
1. It’s derived from research and refined across studies.
Instead of starting from scratch and making up items, a more effective way to build a questionnaire is to start with similar items described in overlapping ways. For the SUPR-Q, this involved combing the UX and market research literature to find items from other published reports that addressed similar or complementary aspects of website UX quality. The SUPR-Q was derived from analyzing the format and items in 17 existing questionnaires and identifying 33 candidate items to test. These items were then refined or removed over multiple studies based on how well they performed across several statistical tests.
Data from over 4,000 responses, across three studies, and over 100 website experiences provided the necessary large dataset to refine the SUPR-Q to be the compact, reliable, and valid questionnaire it is now. We continue to examine new items and also validate translated versions.
2. It’s reliable.
Reliability is how consistent people respond to items. The SUPR-Q’s overall measure of UX quality shows high internal-consistency reliability (Cronbach- α = .86). Its subfactors have lower but still acceptable reliability (α = .64 to α= .88). Lower reliability is a natural consequence of having fewer similar items, so it’s expected to have lower reliability scores with fewer items (8 vs 2)—but it’s a small price to pay. The lowest scoring factor is the loyalty factor and its lower alpha is also driven by the different number of scale points in the 11-point likelihood to recommend item.
3. It’s valid.
Validity is the capability of a questionnaire to measure what it’s intended to. There are a number of ways to measure validity and the SUPR-Q excels in multiple tests of validity. It has high content validity (items cover the construct of User Experience based on expert judgment), high convergent validity (the SUPR-Q correlates with the SUS and other questionnaires that measure similar constructs), and discriminate validity (it differentiates excellent and poor websites as good as or better than other questionnaires). By the way, I used the website webpagesthatsuck.com to identify poor performing websites for the analysis—a list you definitely don’t want your site to be on.
4. It measures four sub-constructs of UX.
In addition to a global measure of UX, the SUPR-Q provides measures of usability, appearance, trust/credibility, and loyalty. Part of the process of questionnaire construction is to examine the number of dimensions a questionnaire has. Across the three studies, the use of a factor analysis revealed these four factors.
5. 50 is average.
To make the score as intuitive as possible, SUPR-Q scores are percentile ranks. Percentile ranks make raw data easier to interpret. It’s what pediatricians use to describe the weight and height of infants and toddlers because it’s hard to know if 25 inches is tall or short (especially to sleep deprived parents with crying kids). With a percentile rank, a 50 means 50thpercentile—which is by definition the average. A SUPR-Q score of 35 is at the 35th percentile—below average.
6. It’s backed by a normalized database.
In addition to having a reliable and valid measure of the website user experience, the other advantage to a standardized questionnaire like the SUPR-Q is a normalized (also called norm-referenced) database to compare scores to. The SUPR-Q database contains a rolling list of around 150 websites that we update partially each quarter. This makes even your first measure with the SUPR-Q more meaningful as you can know whether scores are good (above 75%), bad (below 25%), or average (around 50%). Because we collect and maintain the data (we don’t use client data), you can also compare your score to some of the best-known websites (for example, Amazon, YouTube, Netflix, and Target.com). Maintaining regular updates means the SUPR-Q database isn’t free—as the SUS is—but the timely and relevant benchmarks we believe justify the cost.
7. The database is updated quarterly.
Each quarter we collect data from new websites and use that data to update the SUPR-Q database. We’ll often provide a separate report (for example, for hotels, social media, and retail) that provides more detail on the leaders and laggards in an industry. Interestingly, while individual website scores fluctuate (some do more than others after design changes), the overall average scores across the subfactors don’t change much each quarter (usually by only .1 of a point). This suggests the 150 websites in the database provide a reasonably stable measure of website UX quality to benchmark against.
8. It includes NPS computation.
Whether or not you like the Net Promoter Score, many organizations rely on it (or are told to rely on it). For that reason, the 11-point likelihood to recommend item is included as part of the SUPR-Q. This means you not only get a measure of loyalty, but also the NPS for 150 websites. The average NPS is around –7% (a bit more detractors than promoters), by the way.
9. It predicts the SUS.
A decade ago we started benchmarking websites for the SUS, but we know the website user experience is more than just usability. The 2-item usability factor on the SUPR-Q can predict SUS scores quite accurately because they’re highly correlated (r=.87). We wanted to retain as much continuity to existing SUS data so when we created the SUPR-Q we ensured the usability factor correlated highly and the SUPR-Q provides that.
10. It’s not meant to diagnose problems.
The SUPR-Q, similar to most standardized questionnaires, provides a broad measure of the experience, but it’s not specific enough to tell you what to fix on a website. For that, you need to conduct a usability test or expert review. This is what our industry reports provide. As part of its validation process, the SUPR-Q differentiates between websites with poor and superior user experiences. What’s more, we’ve found that SUPR-Q scores correlate well (r = .5) with a detailed guideline review of a website.
|UX Measurement Boot Camp : Three Days of Intensive Training on UX Methods, Metrics and Measurement Aug. 8th-10th 2018|