FREEDyourMind

More on Methodology

I wrote a post recently on credibility, and it occurs to me that I should talk a little bit about the other tests of a solid methodology. Is it:

· Credible: How widely accepted is the measure? Does it have a good track record of results? Is it based on a scientifically and academically rigorous methodology? Will management trust it?

· Reliable: Is it a consistent standard that can be applied across the customer lifecycle and multiple channels? When all remains the same do we get the same results with every measurement?

· Precise: Is it specific enough to provide insight? Does it use multiple related questions to deliver greater accuracy and insight?

· Accurate: Is the measurement right? Is it representative of the entire customer base, or just an outspoken minority? Do the questions capture self-reported importance or can they derive importance based on what customers say? Does it have an acceptable margin of error and realistic sample sizes? Most customers will report that a lower price is important to them, but lowering the price may not induce them to buy.

· Actionable: Does it provide any insight into what can be done to encourage customers to return to the site, buy again, or recommend it? Does it prioritize improvements according to biggest impacts? A score without actionable insight helps us keep score but not improve.

· Predictive: Can it project the future behaviors of the customer based on their satisfaction with the site visit? The goal is to invest our efforts in those things that will yield value. Without predictive capability we are left to shoot at our targets in the dark.

A bit about reliability vs. accuracy vs. precision. An analogy I like to use is that of a watch. A second-hand gives you precision. Without a second hand, your watch can still be accurate and reliable, but it won’t be precise. Your watch can be precise but not accurate if it tells you that’s it’s 10:22:06 when it’s really 12:55:45. Your watch can be accurate and precise one morning, but if it doesn’t give you the same reading 24 hours later then it isn’t reliable.

Metrics that don’t have the above listed qualities can do more harm than good. They will provide with a false sense of security that will lead you to make bad decisions based on bad data – “garbage in … garbage out”. If you think it is 4pm but it is really 5pm you will be late for dinner. If you think your customers are happy with your product selection…and they are not… they will not be your customers anymore. The stakes are pretty high.

What Do We Mean By Credible?

We in the metrics industry talk a lot about methodologies, what they are, what ours are, and why it’s important to have them. When we’re talking about online measurement, is it important to have a methodology? Absolutely. Of course. Without question. Dictionary.com defines methodology as “a set or system of methods, principles, and rules for regulating a given discipline.”

 

I used to think this went without saying that having a methodology was important, but I’ve seen too many CEO’s lately rallying around metrics that have none, so maybe it’s worth restating.

 

But my point is that yes, having a methodology is a good step, but what really provides you with value is to look at the quality and credibility of that methodology. Anyone can invent or establish a “set or system of methods, principles, and rules” any day of the week. And they can give it an impressive name with official-sounding initials. It’s the oldest play in the marketing playbook.

 

But credibility is key. How widely accepted is the measure? Does it have a good track record of results? Is it based on a scientifically and academically rigorous methodology? Will management trust it? 

 

Most important of all, is there financial evidence that it works? Is there any actual, scientific proof that this methodology will produce the results you want and need it to? You should be looking for academic, peer-reviewed research to back up any claims about what this methodology can do for your company.