FREEDyourMind

Net Promoter White Paper Now Available

I wrote a post a few days ago linking to a white paper I wrote about the flaws we found when we did a multi year study of the Net Promoter metric. You can now download the paper as a PDF on our website if you'd like a copy, or you can email me at larry.freed at foreseeresults dot com.

Audiocast #1 with Evan Schuman: Time Spent Metrics

I recently did a series of audiocasts with Evan Schuman (of StoreFrontBackTalk and eWeek fame) about a variety of topics having to do with measuring the online experience.

The first audiocast is available by clicking here , and it's about what Evan and I both consider to be the deficiencies of the newish Nielsen/Comscore measurement focusing on time spent (instead of just hits and clicks). I did a blog entry about it at the time too, and I haven't changed my mind. Time spent is at best an incomplete metric and at worst a misleading one.

We'll be posting the other segments in the coming weeks, and they include topics like multichannel measurement, where web analytics is going next, and online metrics in general. Let me know if you have any feedback on the content or format either by emailing me at larry.freed at foreseeresults dot com or by leaving a comment.

More Bad News for Net Promoter

Our good friends at Net Promoter are in the news again today with a double dose of criticism in one day. First, Bob Thompson at CustomerThink.com (formerly CRMGuru.com) has an  extensive article about his findings that there is little evidence that NPS is a good or even valid metric for growth. Second, Lior Arussy at Strativity Group wrote an  article on Destination CRM criticizing NPS for its lack of  actionability.

Bob Thompson mentioned a  white paper I wrote examining NPS and finding serious statistical flaws. This was the result of more than 2 years of research of over 20,000 customer surveys (not 8,000 like Bob says in his article). We started out with the best of intentions: after Fred Reichheld’s book came out (The  Ultimate Question: Driving Good Profits and True Growth), everyone was jumping on the Net Promoter bandwagon, and some large companies swore by it and even started compensating executives based on Net Promoter scores. We thought it might make sense to add NPS to our stable of solution for clients but wanted to test it out first. So we took data from three consecutive years of our Spring/Holiday Top 40 Index, and looked at whether NPS was a good measure and whether it had any relationship to future growth, as Reichheld has repeatedly claimed.

We quickly decided a year or more ago that we wouldn’t be offering it to our clients because it had serious flaws. But it took us a few consecutive years of research to feel comfortable publishing a white paper about it. Here’s what we found:

• NPS has a margin of error so high it cannot be taken seriously. In our research, NPS scores had a +/- 10 point margin of error, which means if your NPS is 24, it could actually be anywhere from 14-34. That’s really not useful.

• The scale (lumping people into three categories: promoters, detractors, or neutrals) is misleading and oversimplified. For example, we found that people who answered the Net Promoter question with a “6” were 10 times more likely to buy offline than people who rated themselves a ‘1”, yet they are all lumped together as detractors. Wouldn’t you rather separate out the “6’s” from the “1’s” and work on improving loyalty in the first group?

• NPS vastly overstates detractors. In our research, NPS would have categorized 27% of all respondents as “detractors” or “bad profits.” When you ask more than just one question, you find that only 1% of all respondents are actually likely to communicate a bad experience (you have to ask at least 3 questions to find this out, not just one!). These are your true detractors. Companies are wasting a lot of time and energy if they are overestimating their detractor base by a factor of 27!

• NPS doesn’t predict growth. There is no causal relationship between NPS and future growth, despite repeated claims that NPS is the best predictor of growth.

There are still people still enamored of Net Promoter. You can read CEOs in forums say things like “I don’t care it it’s statistically significant or correct. It’s simple.” I have to admit this blows my mind. HOW can you lead a company successfully with that attitude? In the face of extensive, repeated evidence from impartial academics with no axe to grind like Neil Morgan at Indiana University, researchers like Tim Keiningham of IPSOS Loyalty, and countless consultants, satisfaction experts, and others, it’s a testament to the power of simplicity that this idea has any traction at all since Reichheld’s article about Net Promoter first came out in the Harvard Business Review in 2003 before his book was published.

Meanwhile, Reichheld and the folks at Sat Metrix refuse to acknowledge or refute the criticisms, and it seems from Bob Thompson’s article that Reichheld has stopped even doing interviews in his own defense.

I don’t really expect that this new round of criticism and research will change the minds of true devotees, and perhaps Bob Thompson’s prediction that this will pass in a year is a bit optimistic. Besides, as Bob points out, NPS will just be replaced by the next fad (like the pet rock?) As Bob said, kudos to Reichheld for bringing the idea of loyalty to the forefront and starting the discussion about it. Now it’s time for business leaders to stop looking for easy answers and do the hard work of figuring out how to measure AND influence loyalty.

Psychopathicos

You must check out Ron Shevlin’s great post today on his proposed Psychopathicos Index.

It’s a natural bestseller!

One Step Forward and Two Steps Back

Nielsen/NetRatings recently announced that they will scrap rankings based on page views and begin tracking how long visitors spend on Web sites.

So what does this all mean (and why did I use the title that I did)?

Much of this change is driven by the evolution of sites to Web 2.0, or Rich Internet Applications (such as Ajax and Flash applications). With these new technologies, the concept of a page starts to go away. The web browser running on your pc has been similar to a dumb terminal (yes, a dumb terminal, for those of you that were not around in the 80’s and early 90’s ask your Mom or Dad, or that gray haired guy down the hall) simply presenting to you data from the web servers. With Rich Internet Applications (RIA) your browser and PC start to interact with you without going back to the server on every click. This creates a better user experience, but we lose much of our traditional tracking of page views. This new technology is causing measurement problems with Nielsen/Net Ratings and its competitors along with the clickstream analytic tools.

So, a change in the way we count may be required. Clearly page views is losing its value. But does time spent on a site help us? According to Nielsen/Net Ratings:

“’Total Minutes’ is the best engagement metric in this initial stage of Web 2.0 development, not only because it ensures fair measurement of Web sites using RIA and streaming media, but also of Web environments that have never been well-served by the page view, such as online gaming and Internet applications,” said Scott Ross, director, product marketing for the NetView service.

We are taking a step forward as we understand that page views is not an accurate metric anymore.

But all is not well in the world of metrics. Time spent on a site is not a very good metric in most cases. If you are a retail site and one consumer spends two minutes on your site and another one spends nine minutes, what did we learn? Nothing. We don’t know if the shorter duration visitor accomplished what they wanted and what the retailer wanted. Same goes for the longer duration visit. Did they buy lots of products in that session? Did they leave frustrated because they couldn’t find what they were looking for and went to your competitor to buy? So, when we start shifting our focus to time spent on a website we are taking two steps back.

So what should we care about?  What are the important metrics you need?

There is no doubt that there is value in behavioral metrics. The behavioral metrics that are the most important are those that are financial and transactional. And it is nice to know how many visitors, what pages they viewed and how long they were on our site. But what we should care about is were visitors able to accomplish what they wanted. In other words, what level of satisfaction did a visitor have. And their satisfaction will vary based on what they wanted to accomplish. We also want to know what they are going to do next - are they going to be a long term loyal customer.

So, as Nielsen/Net Rankings tries to shift our focus from page views to time spent on a site — be cautious! The ultimate performance metric is did we satisfy our site visitors.

Welcome to the Bandwagon

As some of you may know, I am not a big fan of the concept of Net Promoter (sometimes referred to as NPS). While measuring positive word of mouth (likelihood to recommend) and negative word of mouth are very valuable and important metrics, the way the NPS score is calculated causes the “one number you need” to lose all precision and reliability–in the best case making it a meaningless number and in the worst case causing it to be misleading, potentially driving you to make BAD decisions. You know the old saying, garbage in - garbage out.

Welcome to the bandwagon of market research sanity - Timothy Keiningham and colleagues. They recently had a paper published in the Journal of Marketing (July 2007) titled, “A Longitudinal Examination of Net Promoter and Firm Revenue Growth.” Let me share with you a few quotes from the paper.

“The clear implication is that managers have adopted the Net Promoter metric for tracking growth on the basis of the belief that solid science underpins the findings and that it is superior to other metrics. However, our research suggests that such presumptions are erroneous. the consequences are the potential misallocation of resources as a function of erroneous strategies guided by Net Promoter on firm performance, company value and shareholder wealth.”

So, lets put that in simple terms: Net Promoter doesn’t work!

Another important quote

“This means that Net Promoter was tied to past growth rates (as opposed to future growth rates). As a result, the data do not help identify whether Net Promoter levels are linked to current changes in revenue growth.”

To clarify – while trying to substantiate his claims on Net Promoter, Reichheld fails Statistics 101. Now I know that he has claimed that executives don’t care about statistics, but I hope they care about accuracy. The data used to support the Net Promoter theory compares revenue growth in a prior period to Net Promoter results in a following period. That would imply, in the best case, that revenue growth drives net promoter. Hmmm… that’s not what Reichheld says.

Let’s look a little further. In the book promoting Net Promoter the biggest case study referenced is Enterprise Rental Car. Believe it or not, they do not ask the questions prescribed by Reichheld, how likely are you to recommend. Instead they ask how satisfied are you. But Reichheld takes the leap to say in his book, and I quote:

“These questions struck pay dirt: the one question at the top of the page: ‘Were you completely satisfied?’ accounted for a startling 86 percent of the variation in customer referrals and repurchases. Those who gave the company a perfect 5 on a 5-point scale - the equivalent of promoter-were three more times likely to return to Enterprise than a customer giving a lower score.”

To put in simple terms - Reichheld “tricks” the readers into thinking Net Promoter works - and it doesn’t. His most profound case study doesn’t even use the “ultimate question.” A true embarrassment to all of those that have drunk the Net Promoter cool-aid.

So welcome to the bandwagon of Market Research Sanity - Timothy Keiningham and colleagues.

  Comments

1.    
         Paula Thornton      
          
         June 27th, 2007      |       5:09 pm      
      

   Fabulous piece! This will be a key reference for future discussions.

        
2.    
         Paul Schwartz      
          
         July 3rd, 2007      |       7:40 pm      
      

   I see NPS being used more and more. I have used it with my clients with limited results. The best use of it is to ask the follow-up questions (”what was the reason for your score?”) and segment those answers by Promoters and Detractors. This will at least give you insight into why customers would OR would not recommend you. Great post Larry!

        
3.    
         The Great Customer Advocacy Hoax      
          
         July 28th, 2007      |       1:56 pm      
      

   […] See Larry Freed’s blog for more discussion on this study. […]

And the Punch Line Is . . .

A funny thing happened to me on the way to …. answering the phone.

I had rented a car from Enterprise Rental Car a week or so ago. I received a call from someone representing Enterprise that had a few quick questions for me. Hmmmm, sounds like a survey, sure, I’m game. Then I remember, isn’t Enterprise Rental Car one of Reichheld’s biggest case studies in his book on Net Promoter? Sure is.

So, the individual on the phone proceeds to ask me a couple of questions. Now, I read Reichheld’s book, and while there were some interesting things in it, and some things I didn’t agree with, I was excited to experience the “magic” first hand.

Now before I go any further, I had a great experience with Enterprise Rental Car.

So, I wait with baited breath…and here comes the first question.

And the question is (and I will paraphrase - since I don’t have recording equipment on my phone, oh we miss the Nixon days)…”Overall, how satisfied were you? Completely satisfied, somewhat satisfied, ” and so on. Ok, maybe the magic comes in the next question, and here it comes,…”How likely are you to call Enterprise again? …”.

And you can imagine my shock. I answered the questions, hung up the phone, and quickly dug out Riechheld’s book from behind my bookcase. I quickly flipped through the book and yes, my memory was right, Enterprise Rental Car is the dominant case study in his book.
So what gives? Let me summarize the FACTS.

• Reichheld’s books says Satisfaction surveys don’t work.
• Reichheld’s book says the one question you need to know is “Are you likely to recommend…”.
• Reichheld’s book uses Enterprise Rental Car as the big case study to show how his Net Promoter works!
• Enterprise Rental Car does not ask the “recommend” question, but rather asks a satisfaction question and a retention question.

And the punch line is….Reichheld’s book is a fraud.

How can we believe him, when his biggest case study goes against everything he says?

Makes you wonder about the business books of today!

  Comments

1.    
         The ONE Question To Ask Customers « Marketing ROI: Whims from Ron Shevlin      
          
         February 15th, 2007      |       5:50 pm      
      

   […] in marketing, marketing ROI, banking, marketing strategy, customer loyalty, marketing measurement. trackback Larry Freed really tees off on Fred Reichheld and the Net Promoter Score concept on his blog,calling the book a fraud. I think Larry is too harsh — but I would assert that “likelihood to refer” is not the ONE question to ask. […]

        
2.    
         Sarah      
          
         February 16th, 2007      |       1:37 pm      
      

   Oh, that’s too funny. So the Enterprise RentACar survey is not one question, and it is not “how likely are you to recommend?” So do they still use NPS? Or do they think that is NPS?

        
3.    
         Glenn (Customer Service Experience) Ross      
          
         February 19th, 2007      |       12:22 am      
      

   Okay, you’ve got your copy handy, and I don’t, but I thought Fred mentioned that Enterprise didn’t follow Fred’s method. Didn’t he say that Enterprise’s survey was home grown and had evolved over time?

   Regards,

   Glenn

        
4.    
         Larry Freed      
          
         February 19th, 2007      |       2:03 am      
      

   Reichheld makes the “assumption” that if they were completely satisfied they would be a promoter. Now, we all know what assumptions can lead to. But why would you base your biggest case study in the book on an assumption? Why would you take shots at satisfaction programs and then base your biggest case study on a company that asked a satisfaction question?
   -Larry

        
5.    
         Raghu      
          
         February 21st, 2007      |       7:29 am      
      

   Well, I agree with you Larry that those Enterprise C-SAT questions sound contrary to what you would expect after knowing Reichheld’s reference to them in his book.

   However, I still do think the most important customer facing question is - “would they refer the company / product / service to friends, family etc”.

   After running the world-wide customer service network & organization for a leading direct to consumer company that you often quote as an ACSI winner and now running the global operations (including CS) of another B2C venture, it is the same question that we have always focused on over the years, maybe with different connotations.

   Focusing on just the satisfaction aspect in surveys and compiling the positive response rates with number of people who were happy with service (ex: 80%, 85% etc like ACSI) etc only gives you so much. It can only make you feel happy in a myopic way with where you are today, and is hardly actionable…..

   What is clearly more important is the number of people out of all responses “who were actually neutral or even dis-satisfied” which I consider to be the inverse correlation (and hopefully less than 10%) of the positives. Driving that expressed neutrality or customer dissatisfaction rates to zero using a six sigma customer centric process is probably more meaningful….

   So in essence, if you truly want to have your customers be “Evangelical” and not just merely satisfied with your service, every company will have to focus on the customer enthusiasm to actually be a referral for them.

   I guess that is what Reichheld was trying to highlight in a roundabout way using NPS etc etc….where clearly Enterprise might have let him down -:)

   Raghu

        
6.    
         Larry Freed      
          
         February 23rd, 2007      |       4:39 am      
      

   Raghu, Excellent comments.
   Word of mouth is an important metric. A decent proxy for positive word of mouth is likelihood to recommend. NPS takes a decent question and pollutes it by the scheme of promoters and detractors. It introduces a very high margin of error.

   The other important point is that recommend only monitors positive word of mouth. It is a dangerous assumption to think that those that don’t recommend are “bad profits”. They are not a positive word of mouth, but that doesn’t mean they are a negative word of mouth. That requires a different set of questions.

   Our solution to measuring satisfaction is a model of satisfaction that includes drivers of satisfaction (product, price, etc.), overall satisfaction and future behaviors (recommend, return, purchase intent, etc.). You need this model to really have a handle on your customers.

   Satisfaction will drive recommendations, loyalty, purchase intent. So, recommendations are an outcome of satisfaction. The complete model of satisfaction (drivers, satisfaction and future intent) provides the depth to make the results actionable.

   Word of mouth is a great marketing tool, but not a proxy for satisfaction or loyalty.

        

 

 

Smoke and Mirrors: NPS and the Pet Rock

The Pet Rock. That is the first thing that comes to mind when I hear people talk about Net Promoter (NPS). OK, if you aren’t old enough to know about the pet rock, check it out at wikipedia, http://en.wikipedia.org/wiki/Pet_rock , better yet, ask your Mom or Dad.

So, what do the Pet Rock and NPS have in common? They are both fads that have no value.

Pretty strong statement considering all the press it gets, considering all the fantastic claims its creator makes. While I could go on for days about all the problems, let’s just focus on a couple of highlilghts (or maybe lowlights is a better way to describe them).

First myth of NPS: NPS predicts revenue growth. Time for stats 101. At best Reichheld tries to show correlation between NPS and revenue growth. Correlation and causation are very different. Causation predicts the future, correlation tries to explain the past, but often confuses coincidence with relationship. That is the first problem. How about an example. Did you know that the amount of damage caused by a fire is highly correlated to the number of firemen fighting the fire? But, the amount of firemen fighting the fire does not cause more damage. Hmmm. So what is the cause. It is missing from the equation. If we introduce the size of the fire, we see that the size of the fire causes more firemen to fight the fire and the size of the fire also causes more damage. That is causation. So, let’s bring that back to NPS. Customer Satisfaction causes revenue growth and customer satisfaction causes the number of recommendations to increase. That is the causal and predictive relationship we should care about.
The second problem is even bigger. Take a close look at the data that is presented to demonstrate this correlation. It compares revenue growth over a period of time, and then looks at NPS at the tail end of the period (pages 192-194 of the book).  For example, comparing ISP providers three-year growth from 1999-2002 and NPS scores from 2002.

Now, if we want to take the leap that correlation could be interpreted as causal, we would add a time dimension. Something that happens today being correlated to something happening in the future. It is a leap of faith, but at best, that is what the data presented could show us. The data presented in the book and papers would actually suggest that revenue growth drives NPS. Is that possible? Well, let’s look at the psychology of recommendations and human nature. When we make a recommendation it reflects back on us. So, we are inclined to recommend those things that are popular (experiencing revenue growth), even if we haven’t experienced them. That explains the possibility that revenue growth drives NPS. The exact opposite of the NPS claims. The real issue here is, when you have a theory, use data to prove it. Bottom line, I have not seen that data for NPS.

Now for my pet peeve (I guess that goes with the theme of the Pet Rock). All those people that should know better, but are jumping on the NPS bandwagon. You know who you are! Many don’t know better (ignorance can be bliss), but many are jumping on the bandwagon because they can capitalize on it, and monetize it. You should be ashamed of yourself. It is time you gave back your pet rock…and got back to reality. Remember those key principles we learned in school. Things like the scientific method. Reichheld himself states in his blog “Frankly, we see little value in continued debate about cause versus correlation, timeframes, or statistical methods”. I guess we should throw out the hundreds of years of great research. Who knows, maybe the world is really flat.

Check back for the second myth.

  Comments

   

      FREEDyourMind.com » More Bad News for Net Promoter      

       

      October 1st, 2007      |       7:11 pm      

   

[…] I don’t really expect that this new round of criticism and research will change the minds of true devotees, and perhaps Bob Thompson’s prediction that this will pass in a year is a bit optimistic. Besides, as Bob points out, NPS will just be replaced by the next fad (like the pet rock?) As Bob said, kudos to Reichheld for bringing the idea of loyalty to the forefront and starting the discussion about it. Now it’s time for business leaders to stop looking for easy answers and do the hard work of figuring out how to measure AND influence loyalty. October 01st 2007 Posted to NPS […]