More on Methodology

I wrote a post recently on credibility, and it occurs to me that I should talk a little bit about the other tests of a solid methodology. Is it:

· Credible: How widely accepted is the measure? Does it have a good track record of results? Is it based on a scientifically and academically rigorous methodology? Will management trust it?

· Reliable: Is it a consistent standard that can be applied across the customer lifecycle and multiple channels? When all remains the same do we get the same results with every measurement?

· Precise: Is it specific enough to provide insight? Does it use multiple related questions to deliver greater accuracy and insight?

· Accurate: Is the measurement right? Is it representative of the entire customer base, or just an outspoken minority? Do the questions capture self-reported importance or can they derive importance based on what customers say? Does it have an acceptable margin of error and realistic sample sizes? Most customers will report that a lower price is important to them, but lowering the price may not induce them to buy.

· Actionable: Does it provide any insight into what can be done to encourage customers to return to the site, buy again, or recommend it? Does it prioritize improvements according to biggest impacts? A score without actionable insight helps us keep score but not improve.

· Predictive: Can it project the future behaviors of the customer based on their satisfaction with the site visit? The goal is to invest our efforts in those things that will yield value. Without predictive capability we are left to shoot at our targets in the dark.

A bit about reliability vs. accuracy vs. precision. An analogy I like to use is that of a watch. A second-hand gives you precision. Without a second hand, your watch can still be accurate and reliable, but it won’t be precise. Your watch can be precise but not accurate if it tells you that’s it’s 10:22:06 when it’s really 12:55:45. Your watch can be accurate and precise one morning, but if it doesn’t give you the same reading 24 hours later then it isn’t reliable.

Metrics that don’t have the above listed qualities can do more harm than good. They will provide with a false sense of security that will lead you to make bad decisions based on bad data – “garbage in … garbage out”. If you think it is 4pm but it is really 5pm you will be late for dinner. If you think your customers are happy with your product selection…and they are not… they will not be your customers anymore. The stakes are pretty high.