Does the NPS predict growth?
What data is there to support the predictive ability of the Net Promoter Score?
In the original research reported by Fred Reichheld, he showed that the NPS strongly correlated with growth in 11 out of 14 industries.
However, he used historical, not future, growth. While this established a sort of concurrent validity, it left open the question about the predictive ability of the NPS.
We took Reichheld’s original NPS data from seven industries and then collected the same growth metrics for the next two- and four-year periods. We found a reasonably strong correlation between the NPS and two-year growth metrics and the relationship remained relatively strong four years into the future.
The relationship was, however, weaker than what Reichheld reported with the historical data. On average we found the NPS still explained an impressive 38% of changes in future growth (average correlation of r = .61). Companies with higher Net Promoter Scores tended to have higher growth rates in their industries. Companies with lower Net Promoter Scores tended to have lower growth rates in their industries.
But we used the same seven industries and NPS data from Reichheld. Perhaps these are cherry picked and represent a best-case scenario. What about other industries and data that is more recent?
Looking for Growth Predictions
Our first attempt at looking at another dataset was examining the U.S. airline industry in 2013. That analysis immediately revealed the need to account for mergers, acquisitions, and bankruptcies. When one company acquires another, the added revenue accounted for a lot of non-organic growth, potentially distorting growth patterns (something investors care about too). It’s often not possible to disentangle different sources of revenue to compute growth from public filings, meaning some companies in an industry need to be excluded to understand larger patterns between the NPS and growth.
This was the case with American Airlines, which acquired US Air during the period we examined and significantly increased its revenue growth. When we excluded American Airlines, we found a very strong relationship between the NPS and future growth, with the NPS explaining a substantial 92% of the variation in 2014–2016 growth rates.
In another analysis, we systematically examined the consumer software industry. Using our own data from 2013 and 2014, we found that the NPS (and SUS) correlated strongly with two-year future growth rates.
The NPS (and SUS) was able to explain (predict) at least 50% of two-year future revenue growth rates. This relationship is even larger than the relationship we found at the company level from Reichheld’s data. The relationship held (but was weaker) at four years in the future. We also found that relationship still held when removing two of the more influential companies on the regression equation—suggesting a reasonably robust relationship between the NPS and growth in the software industry. In this analysis, we also had to account for mergers and unavailable growth data for some companies.
But can we extend our earlier findings to more industries beyond Reichheld’s industries, airlines, and software? Will the relationship hold if we use another independent data source and different industries, with data that’s not 15 years old?
Data from 14 Industries
To assess the predictive validity of the NPS (correlation with future growth), we have to go back in time to look for historical (but not too old) NPS data. To do so, we purchased the 2013 NPS Benchmark report from the Temkin Group, a well-known source of NPS benchmarks apart from Satmetrix (recently acquired by Qualtrics).
The Temkin report has NPS benchmark data from 10,000 U.S. consumers collected from an online panel in October 2013. Sampling was matched to U.S. Census data to mirror the U.S. population on age, income, ethnicity, and region. Respondents provided NPS data on a “random” sample of organizations they had interacted with in the prior 90 days (meaning not all responses are independent). There were 269 companies, with sample sizes between 100 and 2,500 per company, in 19 industries.
Finding Future Growth
We then started the laborious task of finding future growth metrics for as many of these industries as possible. With the help of research assistants, we scoured financial statements, public records, industry resources, and press releases to generate growth numbers. Not all industries showcase the same growth metrics (for example, revenue) because it can mask real growth by a company. A good example of this is the retail industry, which uses same-store sales as a preferred metric. This way investors can differentiate organic growth for large retailers based on the company’s offerings from growth derived from opening more stores.
Finding the growth data wasn’t an easy task and quite difficult for some industries and some companies. Aside from the expected problems with mergers, the primary challenge we ran into was that many of the companies have significant international operations and don’t break out numbers for just U.S. business (to match the NPS data from the U.S. respondents).
We were able to find growth data on 14 of the 19 industries for the future years of 2013 to 2017 for 158 companies (Table 1). This cross-industry analysis of NPS and growth is likely the largest ever conducted. If there’s a pattern, we should see it here.
Table 1 shows the industries we found growth data for in the Temkin report, the growth metric we used, and the number of companies represented in each industry.
|Industry||Metric||# of Companies|
|Grocery||Same store sales||8|
|Auto dealers||Car sales (numbers)||21|
|Fast food||Same store sales||31|
|Retailers||Same store sales||31|
|Health plans||Premium revenue||10|
We were unable to find sufficient data in five industries for the following reasons:
- Appliances — Unable to find aggregated data for the U.S. market.
- Car rental agencies — Data wasn’t categorized by brands (e.g. Enterprise, National, and Alamo are all owned by the same parent company).
- Credit card issuers — Unable to associate with issuing banks for enough years.
- Hotel chains — Unable to find data for U.S. market only.
- TV service providers — Unable to differentiate separate revenue for enough providers due to massive consolidation in the industry (e.g., AT&T, DirecTV, Verizon, Time Warner, Charter, Bright House).
Similar to our earlier analysis, we found the Net Promoter Score generally had a positive correlation with future growth metrics. There was a positive correlation between the NPS and growth in 11 of the 14 (79%) industries. Interestingly, this proportion is the same that Reichheld reported about the NPS—it was the best or second best predictor in 11 of 14 industries they examined.
The average correlation between NPS and growth was r = .35 (see Table 2). For the 11 industries with a positive correlation, the average correlation with two-year revenue growth was higher at r = .44. This ranged from a high of r = .89 for U.S. airlines, to r = .49 for retailers and r = .50 for investment banking, to a low of r = .22 for grocery stores and r = .26 for wireless providers.
On average, when the NPS is positively correlated with growth, it can explain (predict) about 20% of the variation in future growth metrics at the company level. This is slightly less than the predictive ability of the SAT, which explains 25% of first-year college grades. The average predictive ability here (R-Square) is about half what we found with the Reichheld dataset (R-sq = 38%), which might be considered as a “best-case” dataset.
|Industry||2013-2015 r||2013-2017 r||2013-2015 Rank r||2013-2017 Rank|
The average correlation only decreased slightly for the longer time period of four-year growth (2013–2017). The average correlation was r = .31 (second column of Table 2). All correlation averages used the Fisher r-to-z transformation to account for non-normality.
Figure 1 shows an example of a strong relationship between the NPS and future growth in the U.S. investment banking industry. The size of each bubble represents the amount of 2013 revenue for each company.
Figure 2 shows a general linear pattern between U.S. car sales growth and the NPS. You can see Jeep had explosive growth during this period compared to its average NPS (for the industry).
Figure 3 shows the strong relationship between same-store sales and the NPS with some usual suspects in the lead (Nordstrom and Costco) and two laggards (Kmart and Sears), both of which are teetering on bankruptcy. We were unable to separate international sales from U.S. sales for Lowe’s, Foot Locker, Gap, Old Navy, Staples, CVS, and Walgreens but included them because a substantial amount of their revenue comes from the U.S. market.
The three industries that had a 0 or negative correlation were Internet Service Provider (ISP), PC, and parcel delivery. The lack of correlation here could be from near-lack of competition from regional or national monopoly-like statuses (e.g., Comcast, Verizon, Cox, USPS) that many of these companies have. The PC industry went through significant consolidation and acquisitions during the analysis period (e.g., IBM/Lenovo, Acer, Gateway, and Compaq), potentially masking a relationship between the NPS and growth. We had only three datapoints in the parcel business (UPS, USPS, and FedEx) and all had NPS scores within six points of each other—likely restricting the range for the correlation. And, of course, it could also be that the NPS is not a good predictor of the growth metrics we selected in these industries for these years.
You can download our sources and notes [pdf] to replicate our findings, check our assumptions, and even find the missing data (let us know if you do!).
NPS Rank Predicting Growth Rank
As was the case with our earlier replication and using correlations in general, all it takes is one unusual datapoint (e.g., from an acquisition or product changes) to substantially alter or mask a relationship. This is simply a consequence of using correlations, which work best with linear relationships (especially on data collected in the real world and not in a controlled experiment).
One way to smooth out the fluctuations in growth is to look at the relative ranks of company growth as opposed to raw growth percentages. That is, does a relatively high NPS in one industry predict the relatively high growth of a company? Does the NPS identify the leader and laggard in each industry?
If a company within an industry has the highest NPS, is it more likely to also have the highest future growth relative to its peers?
To find out, we correlated the relative rank of NPS and growth rate within each industry. The Pearson Correlation of the ranks is the same as the Spearman rank correlation rho. Table 2 also shows the correlation within each industry and overall correlation. For the same two-year period, we found a stronger relationship between NPS rank and growth rank. The average correlation was r = .44, and 12 out of 14 industries had a positive correlation with NPS. The ISP and the PC industries had 0 and a negative correlation respectively. For the four-year growth period, the average rank correlation was r = .29. with the wireless industry becoming negative (11 out of 14 remained positive).
Summary and Takeaways
An analysis of 158 companies across 14 industries from an independent NPS data source found:
Net Promoter Scores tend to predict future growth. This analysis found a modest correlation between Net Promoter Scores in 11 of 14 industries for the immediate two-year and four-year future periods. This finding is similar to earlier work where we found strong correlations for the original Reichheld data and for the airline and software industries. This analysis also showed similar (albeit smaller) correlations for the four-year future period (r = .31) and when using relative ranks for the same periods (r = .44 for two-year and r = .29 for four-year growth).
Correlation varies by industry. The correlation between the NPS and future growth ranged from non-existent/negative to very strong for the industries studied. This is an important qualification for the NPS. Even Reichheld’s original data showed it wasn’t universally the best predictor (only 11 of 14 industries). This may especially be the case for industries that are monopolies (e.g., ISP) and have little competition or are going through consolidation or transformation (which he again cautioned about). Companies can use the data shown in this analysis as an additional datapoint to decide whether the NPS should be used as a proxy for future growth.
NPS is a strong growth predictor in the software industry. One of the strongest correlations we found in this analysis was for the software industry (r = .40 after two years and r= .69 after four years). This corroborates our earlier study using a separate dataset that also found that the NPS was a strong predictor of growth. In our earlier analysis, the correlation was even higher (r > .70). This analysis looked at company NPS whereas our earlier analysis was done at the product level (and may account for the higher correlations). This suggests that the NPS may be a good predictor of growth for at least the larger players in the software industry.
NPS is not a perfect predictor. Don’t expect the NPS (or any metric actually) to perfectly predict the future in any time period. Prediction is notoriously difficult. If you spend enough time working with data you learn to not only appreciate the limitations of data but also to not dismiss the data if it isn’t a perfect predictor. Similar to other predictors, such as standardized test scores, this analysis shows that the NPS tends to be a reasonable predictor of future growth at the company level in many, but not all, industries all the time.
Is NPS better than satisfaction? In our earlier analysis, we found little evidence to suggest that the NPS was superior to customer satisfaction. In that analysis, we found it offered similar predictive validity to satisfaction but not always and sometimes it was worse. But what the NPS may lack in superior predictive validity, it may make up for in efficiency and familiarity. The NPS single item question is certainly shorter than most satisfaction questionnaires and because of its ubiquity, most executives are familiar with it. If your company uses NPS, it might be “good enough” as a proxy. You should probably spend more effort on how to improve customers’ attitudes than arguing about finding a perfect measure. We’ll examine how well other satisfaction measures predict future growth metrics in a future article.
Is the NPS the root or fruit of growth? While Net Promoter Scores seem to be a reasonable proxy for future growth rates, it could be that the NPS is just reflecting already successful companies. That is, it could simply be that higher sales generate more recommenders and not that more recommenders generate more sales. This was a point also raised by Schneider et al. [pdf]. In other words, if you increase your growth, your NPS will also likely increase because you’ve increased the pool of people willing to recommend. Is the NPS the root of growth or the fruit of growth? This is more a fundamental issue with measuring attitudes than anything specific with the NPS. That is, if higher NPS scores are simply the fruit of more sales, we’d likely see higher satisfaction scores and other higher loyalty metrics (such as likelihood to purchase). Identifying causality (or at least temporal precedence) can be quite tricky for real-world data such as company growth, which can’t be subjected to the managed conditions of randomized control trials and will be the subject of many future analyses.
|UX Measurement Boot Camp : Three Days of Intensive Training on UX Methods, Metrics and Measurement Aug. 7th-9th 2019|