Our good friends at Net Promoter are in the news again today with a double dose of criticism in one day. First, Bob Thompson at CustomerThink.com (formerly CRMGuru.com) has an extensive article about his findings that there is little evidence that NPS is a good or even valid metric for growth. Second, Lior Arussy at Strativity Group wrote an article on Destination CRM criticizing NPS for its lack of actionability.
Bob Thompson mentioned a white paper I wrote examining NPS and finding serious statistical flaws. This was the result of more than 2 years of research of over 20,000 customer surveys (not 8,000 like Bob says in his article). We started out with the best of intentions: after Fred Reichheld’s book came out (The Ultimate Question: Driving Good Profits and True Growth), everyone was jumping on the Net Promoter bandwagon, and some large companies swore by it and even started compensating executives based on Net Promoter scores. We thought it might make sense to add NPS to our stable of solution for clients but wanted to test it out first. So we took data from three consecutive years of our Spring/Holiday Top 40 Index, and looked at whether NPS was a good measure and whether it had any relationship to future growth, as Reichheld has repeatedly claimed.
We quickly decided a year or more ago that we wouldn’t be offering it to our clients because it had serious flaws. But it took us a few consecutive years of research to feel comfortable publishing a white paper about it. Here’s what we found:
- NPS has a margin of error so high it cannot be taken seriously. In our research, NPS scores had a +/- 10 point margin of error, which means if your NPS is 24, it could actually be anywhere from 14-34. That’s really not useful.
- The scale (lumping people into three categories: promoters, detractors, or neutrals) is misleading and oversimplified. For example, we found that people who answered the Net Promoter question with a “6” were 10 times more likely to buy offline than people who rated themselves a ‘1”, yet they are all lumped together as detractors. Wouldn’t you rather separate out the “6’s” from the “1’s” and work on improving loyalty in the first group?
- NPS vastly overstates detractors. In our research, NPS would have categorized 27% of all respondents as “detractors” or “bad profits.” When you ask more than just one question, you find that only 1% of all respondents are actually likely to communicate a bad experience (you have to ask at least 3 questions to find this out, not just one!). These are your true detractors. Companies are wasting a lot of time and energy if they are overestimating their detractor base by a factor of 27!
- NPS doesn’t predict growth. There is no causal relationship between NPS and future growth, despite repeated claims that NPS is the best predictor of growth.
There are still people still enamored of Net Promoter. You can read CEOs in forums say things like “I don’t care it it’s statistically significant or correct. It’s simple.” I have to admit this blows my mind. HOW can you lead a company successfully with that attitude? In the face of extensive, repeated evidence from impartial academics with no axe to grind like Neil Morgan at Indiana University, researchers like Tim Keiningham of IPSOS Loyalty, and countless consultants, satisfaction experts, and others, it’s a testament to the power of simplicity that this idea has any traction at all since Reichheld’s article about Net Promoter first came out in the Harvard Business Review in 2003 before his book was published.
Meanwhile, Reichheld and the folks at Sat Metrix refuse to acknowledge or refute the criticisms, and it seems from Bob Thompson’s article that Reichheld has stopped even doing interviews in his own defense.
I don’t really expect that this new round of criticism and research will change the minds of true devotees, and perhaps Bob Thompson’s prediction that this will pass in a year is a bit optimistic. Besides, as Bob points out, NPS will just be replaced by the next fad (like the pet rock?) As Bob said, kudos to Reichheld for bringing the idea of loyalty to the forefront and starting the discussion about it. Now it’s time for business leaders to stop looking for easy answers and do the hard work of figuring out how to measure AND influence loyalty.
