To make better decisions, you need data. That’s become a truism. But can the process of using the data lead to bad outcomes?
It seems like a hypothetical question, but it doesn’t take long to find a few key metrics that, when tracked, can lead to unwanted outcomes:
On-time departures: On-time flight departures for airlines were once a key measure of airline promptness. Perhaps you’ve been on a plane that backed away from the gate before sitting on the tarmac for an hour. Of course, you’ll be late at your destination, but the flight was considered “on time” because it backed away from the gate on time.
On-time Arrivals: No problem. Let’s track on-time arrivals because of the problem with on-time departures. That’s what really matters, right? But an interesting thing has happened. The same flight, between the same cities, now takes longer than it did decades ago, despite better technology. What’s going on? Turns out airlines are padding their flight times to help improve on-time arrival.
Crime rates: After a long decline, crime is rising in the United States. There has been—and there will continue to be—a lot of pressure on police departments and municipalities to reduce crime, which can result not in actual crime reduction but in the under-reporting of crimes.
Test scores: High-stakes testing (where teachers’ salaries are tied to improved scores) led some educators to change students’ test scores systematically across large school districts.
High school graduation rates: There’s no need to look into high-stakes testing. Even high school graduation rates and other seemingly benign educational reforms can be mishandled.
Quarterly earnings: Business executives are under a lot of pressure to deliver results. Shareholders care less about squishy internal performance metrics than meeting or beating quarterly earnings estimates, something that seems less fungible. Yet even quarterly earnings can be and are manipulated.
The Net Promoter Score is used increasingly as a key measure of business performance, and like these other examples, it has been subjected to manipulation.
Where there’s pressure to achieve an outcome, there’s pressure to manipulate the measure that determines the outcome. These are examples of Campbell’s Law.
Donald Campbell was an influential psychologist known for a highly cited book on experimental and quasi-experimental design. If you’ve heard the phrase “teaching to the test,” then you know Campbell’s Law. More formally, it states, “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” (This is similar to a famous observation from the economist Charles Goodhart—“When a measure becomes a target, it ceases to be a good measure.”)
Campbell’s Law was written in an era when high-stakes tests such as the SAT were playing an increasing role in education policy. But Campbell’s Law applies to any high-stakes measure, including the Net Promoter Score, customer satisfaction scores, quarterly earnings, and the other examples cited earlier.
When pay and promotion are tied to the NPS without accountability, people will inevitably focus on ways to move the needle. That’s a good thing if the movement reflects actual improvement in positive word of mouth (one measure of loyalty). It’s a bad thing if the movement becomes corrupted and unmoored from the underlying customer sentiment.
A consequence of Campbell’s Law is the Sauro and Lewis corollary: Mismanagement leads to mismeasurement. Without proper checks in place, people will find ways to manipulate measurement to their benefit. Such manipulations include changing when the data are collected, deciding who will and won’t be surveyed, and encouraging favorable responses from respondents.
For example, employees at a large bank deleted email addresses to prevent getting low Net Promoter Scores from customer surveys.
Less egregious corruption can also distort measures. For example, I (Jeff) recently received an email from my dealership after I had my car repaired. The customer service manager asked me to take a J.D. Power satisfaction survey (no NPS here, but the same potential for corruption). He also let me know in bold, underlined text that they really need the highest scores:
Now that’s not telling me how to answer or changing my answers outright, but I sure will feel bad about giving anything less than a 10, given the implication that this could affect pay, bonuses, or possibly continued employment at the dealership.
What Do You Do about Campbell’s Law?
As long as there are quantitative measures with consequences, Campbell’s Law will apply. But what does one do about it? We’ve seen a few possible responses for the Net Promoter Score and other high-stakes company metrics that are subject to influence and corruption:
- Stop measuring: Some say throw out the NPS and other measures. Stop measuring customers’ intent to recommend, and some would say don’t measure at all!
- Change the measure: Consider switching from the NPS to customer satisfaction. But that’s like switching from the SAT to the ACT for high-stakes testing. The email from my dealership shows how futile that approach can be as the same problems will just carry over to the other measure.
- Ignore the measure: Some say just ignore the measure. But there are usually good reasons for measuring high-stakes metrics. Airline promptness, graduation rates, testing scores, and crime statistics are examples of metrics that attempt to measure something that matters. NPS, satisfaction, customer effort, and other customer metrics are ways of gauging attitudes, which studies have shown to matter a lot for retention, profit, and growth.
- Increase governance: Implement ways to reduce corruption or distortion.
Reichheld was keenly aware of the deleterious effects of Campbell’s Law. It’s a bit of a surprise, then, that he said in a 2019 WSJ article, “I had no idea how people would mess with the score to bend it, to make it serve their selfish objectives.”
In The Ultimate Question, Reichheld dedicates a fair number of pages towards identifying ways of preventing the corruption of the NPS. (See Chapter 6, “The Rules on Measurement.”)
In this chapter, he identifies a few ways to help reduce the unwanted tampering with the NPS:
- Develop audit procedures to uncover attempts at “gaming the system.”
- Educate customers against bribery attempts or corruption.
- Use third parties to collect the data to help prevent other tampering or influence.
- Use email, for which people may be more honest (people in person or on the phone don’t want to hurt feelings).
- Make scores transparent to the whole company (if something doesn’t look right, someone is more likely to say something).
- Time surveys in unpredictable intervals to reduce planned changes in behavior.
The effectiveness of these approaches is worth their own study and perhaps will be the subject of another article. And while it makes sense to reward good performance in organizations, another question worthy of investigation is whether it’s better to decouple pay from subjective customer measures that, when not corrupted, can provide valuable warning signs of customer service issues.
In short, Campbell’s Law states that the more you use a measure to make important decisions, the more it becomes susceptible to corruption, distorting the thing you intended to measure. Campbell’s Law applies across disciplines such as education, criminology, and business. It affects metrics such as testing scores, crime rates, the Net Promoter Score, and customer satisfaction. While the impacts of Campbells’ Law can likely never be eliminated, they can be mitigated using strategies to minimize corrupting influences.