Positive word of mouth is a critical driver of future growth for consumer software products.
Along with features and capabilities, user experience is a key determinant of how likely users adopt new technology and recommend software.
We’ve found that usability accounts for between 30% and 60% of the variation in Net Promoter Scores. A positive experience leads customers to recommend a product. A negative experience, predictably, causes customers to actively discourage others from buying a product.
In the latter part of 2016 and beginning of 2017 we collected data from 2,164 participants to learn about their attitudes toward the functionality and usability of consumer software. Measuring user attitudes toward the experience is an important step in benchmarking and improving a product.
This was an update to data we collected on a similar set of products in 2014 using a similar methodology and complements our business-focused software analysis and report.
The 17 software products include a mix of popular productivity, storage, and security software:
- Adobe Photoshop
- Adobe Reader
- Dropbox
- Google Calendar
- Google Docs
- Google Drive
- iCloud
- iTunes
- McAfee Antivirus
- Microsoft Excel
- Microsoft PowerPoint
- Microsoft Word
- Mint.com
- Norton Antivirus
- Microsoft OneDrive
- Quicken
- TurboTax
All participants in the study were from United States. A bit over half (55%) were male and tended to be older (76% above aged 30) and educated (90% had at least some college). Participants were asked to reflect on their most recent experiences with one of the software products and answer a number of items, including the System Usability Scale (SUS) questionnaire and the likelihood-to-recommend question (used to compute the NPS). The full details are available in the report. Here are some highlights.
Loyalty
Net Promoter Scores
The most popular method to measure loyalty is the Net Promoter Score. It’s calculated using the 11-point (0 to 10) likelihood-to-recommend question. It’s computed by subtracting the percent of detractors (0-6) from the percent of promoters (9-10).
Across the 17 products the average Net Promoter Score is 23% with a range of -2% to 50%. This is higher than the business software average NPS of -11% and about the same as the 2014 average NPS of 21%. Google Drive edged out TurboTax this year for the highest NPS in the group at 50% versus 45% respectively.
We have three years worth of data from ten consumer software products. The biggest drop we’ve seen is with iTunes, which has an NPS of 1%, continuing a downward trend (28% in 2011 and 14% in 2014). iTunes has never been popular, even with Apple fans, and the verbatim comments reflect this:
“I honestly prefer other media programs but having Apple products forces me to use iTunes for some tasks. If I could I would leave it behind.”
“I feel like iTunes leaves a lot to be desired. It is buggy and frequently crashes, and is not very easy to navigate – UI could DEFINITELY be improved. However, it’s kind of the only game in town if you’re a Mac user.”
Figure 1: NPS scores for iTunes from 2011, 2014, and 2017
Additional Loyalty Measures
In addition to the NPS, we also captured metrics that build a holistic picture of customer loyalty and the potential impact of word of mouth. These include:
- Retro-Recommend Rate: The percent of customers that actually recommended the software in the preceding 12 months. TurboTax has the highest rate with 68% of participants reporting they recommended it to someone in the last year.
- Product Referral Rates: The percent of customers to whom the product was recommended. Dropbox has the highest percentage, 46%, who reported being referred.
- Promoters Actually Recommending: By looking at the percent of promoters that recommended and who were referred to the product by others, you can estimate the value of a promoter. Quicken leads this group with 81% of promoters having recommended it in the last year (a bit higher than its online cousin Mint.com).
System Usability Scale (SUS) Scores
We used the popular System Usability Scale (SUS) to compute the perceived ease of use of the 17 products. SUS is a 10-item questionnaire with possible scores ranging from 0 to 100. The average SUS score from over 500 products (including websites and consumer software) is 68. The average SUS score from this group is 75, with a low score of 68 and high score of 82.
Raw SUS scores were converted to percentile ranks, which translated the average raw score into 75—meaning this group of consumer software products scores above average in perceived usability. (Note: it just happens that a raw score of 75 translates into the 75th percentile.)
The product with the lowest SUS score in this dataset is Photoshop, with a SUS score of 68 (at the 50th percentile). Photoshop has become packed (some would say bloated) with features over its long history and has a reputation for being hard to learn. This is echoed in the verbatim comments where 39% of respondents said the one thing they would improve would be the learning curve and the difficulty to use some features.
“Even though I enjoy it, it is a bit complicated to those who aren’t computer savvy.”
The SUS scores for Quicken, McAfee, and Excel have been relatively stable since 2011. Microsoft Word showed a dip in perceived usability in 2014 (from an “A” to “B” grade) but has since recovered.
Figure 2: SUS scores for Microsoft Word from 2011, 2014, and 2017
Some verbatim comments provide an insight, which might explain some of the fluctuations with Word:
“Microsoft Word is a very good program so I respect it, but for me and most of the people I know it is often more complex than necessary.”
“I prefer the Pages program offered by Apple for my word processing needs. I find it easier to use and much more user-friendly in its design. That said, while I find Microsoft Word to be a bit clunky, I’m only slightly negative towards it.”
Usefulness & Ease
We also reported on the UMUX-LITE, as we did in the business report. It’s based on earlier work harkening back to the Technology Acceptance Model (TAM) but includes only one item each for ease (how easy is the software to use) and usefulness (how well the capabilities meet the users’ needs). In aggregate, it provides a measure of acceptance, or even more broadly satisfaction, or quality of the product.
Figure 3 shows a scatterplot of the normalized version of the two components of the UMUX-LITE scores. Almost all the software products are rated above average for usefulness (above 50%) except for iCloud and about half are also above average for usability. This is an interesting contrast to the businesses software that had similar usefulness scores but worse usability scores. This is also consistent with our earlier research that has shown consumer software tends to be more usable for the mass market compared to specialized business software.
Figure 3: UMUX-LITE scores
We’ll continue to collect UMUX-LITE scores as it’s become an important benchmark for many organizations as part of using a single (succinct) score to quantify software acceptance and satisfaction.
Usability Correlates with Loyalty
We’ve again seen strong correlation between usability and loyalty (r = .56) when correlated at the product level [pdf]. That means SUS scores here explain (predict) around 31% of Net Promoter Scores with this type of software.
Illustrating this strong correlation, Figure 4 shows the SUS scores for detractors, passives, and promoters (computed at the individual level for all responses).
Figure 4: Average SUS scores for promoters, passives, and detractors
This figure shows that on average promoters have SUS scores of around 84. This pattern is consistent with both the business software data and the research we conducted 7 years ago that found that SUS scores above 80 have a good chance of having promoters. The average SUS score for passives is 72. On average, detractors provide average SUS scores of 60.
Summary
Benchmarking is an important step in improving the user experience of software and should include measures of usability and loyalty. Third-party benchmarks like the ones described here complement internal benchmarking programs by providing additional context. It’s a lot easier to know how well your product is doing if you know where it stands relative to the competition or the industry average.
The full list of benchmarks and select verbatim responses are available for purchase in the Net Promoter & UX Benchmark Report for Consumer Software (2017).
