Avinash Kaushik wrote a great post this week on how you should choose an online survey provider.
Avinash worked extensively with ForeSee Results, so he’s very familiar with the ACSI methodology and how we apply it online. Since he’s been both a customer of web metrics and a partner of web metrics companies, he has a lot of knowledge and has given a lot of thought to what kinds of things people should consider when choosing a tool.
I just want to expand on a few of Avinash’s points:
#1] Mathematical Rigor: No matter what you choose look for a partner that can apply mathematical rigor to your results. I find so many results mis-interpreted because poor math applied in the analysis. Measuring survey results is not simply taking the average of the answers (averages lie!), it is measuring distributions and doing regressions. You don’t want to be bothered to apply the statistics and statistical significance, stress test to ensure your vendor does (then you can focus on analysis and not reporting).
This is a great point that too many people undervalue (just look at the Net Promoter fad: margins of error in the double digits, poorly-designed distributions, and people love the “simplicity” of it!). But I would take this point about mathematic rigor even further. It’s not enough JUST to have mathematical rigor: you can ask bad questions and have a survey pool that is not representative of your audience and you can still use mathematical rigor to interpret the results and get a low margin of error and high confidence intervals. Mathematical rigor ALONE is not what leads you to the holy grail; what leads you there is having a methodology behind the survey that is accurate, precise, and predictive and THEN making sure you analyze those results with mathematical rigor.
This brings me to a comment I had on Avinash’s point #3 (skipping past #2, which I’ll come back to because #3 is related to what I’m already talking about):
#3] Benchmarks & Indexes: Few people in senior management will take action when you tell them “our score on satisfaction (or content or navigation) is 6.0″ (or 45, depending on your vendor). But most of them will get off their butt and give your money to take action when you go to them and say “our score on satisfaction is 6.0 and amazon.com is 9.4 out of 10″.
Your actual score does not drive action, the difference between that and a industry benchmark drives action (no one wants to look bad by comparison!). And here is another subtle human factor: no one wants your opinion about anything, providing a comparison to a external benchmark depersonalizes the number and it is more likely that it will be “heard”.
iPerceptions uses the iPSI (developed by iPerceptions), ForeSee user the ACSI (developed by University of Michigan), both wonderful benchmarks you can compare your scores to and help motivate your management to take action! There are other vendors, look to see if your vendor will provide you with a benchmark or a index to compare.
Another great point, especially about the executive tendency to listen up when you can create an aura of rivalry with a competitor on industry leader. But again, just having a benchmark isn’t enough—anybody can create a methodology and an index and a benchmark. I could create the Larry Satisfaction Index—LSI ™ , write a white paper on it, and call it a methodology, but that doesn’t make it MEAN anything unless it is accurate and precise. An analogy: you’re on the golf course, you look down the fairway and decide it’s 120 yards to the green. You take out your nine iron and hit a perfect shot: exactly 120 yards. But you’re still 30 yards short of the green. You weren’t 120 yards from the green, you were 150 yards! You had a measurement, you had a methodology, but it was worthless because it wasn’t accurate, reliable, or precise.
To to this point about benchmarking, I would only add that people should make sure that whatever benchmark they chose has a proven history of being credible, accurate, precise, predictive, and reliable. What is a proven history? Well, the proof is in the pudding—there should be scientific, academic, published proof that the metrics are linked to financial performance and desirable future behaviors like word of mouth and loyalty. Just ask your vendor to show you the proof that their benchmark actually means something. The American Customer Satisfaction Index methodology has that proof.
Back to Avinash’s #2, about on-the-fly segmentation capabilities:
#2] On The Fly Segmentation Capabilities: Look for the capabilities that are provided to do on the fly segmentation of your data (aggregates lie!) . . .
There is gold in your ability to pick a particular segment of traffic
that absolutely hates you and slice it off and drill down to why they
were on your site, what products they own, what did they not find, and…
then go fix it fast. Segmentation rocks! You want to be able to do it
efficiently, yourself if possible, to reduce chances of vendor delays.
My one addition to this is
that some companies may want to choose a company who will let you do
the segmentation yourself, but it may be that what you really want and
need is a seasoned team of experts to do this for you or help you do.
Data overload is a big problem with metrics, and sometimes companies
plan to do it themselves and end up doing nothing. So look for a vendor
that can enable you to do that segmentation OR that can do it for you.
Last, but not least, Avinash mentions page-level vs. site-level surveys in his discussion of pre-survey considerations:
Page level surveys are user initiated (”tell me what you think of this page” “rate this page” etc) and serve the purpose of measuring the experience of the page. You can ask “site” level experience questions but the way they are initiated by users and local level at which initiation happens makes them a sub optimal choice for measuring site experience. They can good for measuring effectiveness of a page.
Site level surveys are usually presented to the users (”please give us a few minutes to answer these questions”), typically on exit, and are a great measure of the site experience (”what got you here today” “what were you able to do” “how much did we suck” “did you find what you were looking for”). They are good at measuring effectiveness of a site.
I would disagree slightly: page level surveys are more feedback and less measurement per se, for two reasons:
1) Since they are opt-in, they aren’t representative of your larger audience, which means they aren’t a true measurement.
2) The page that people give you feedback on tends to be several pages after the page that caused them the problem, because people don’t get immediately frustrated and opt-in to a survey to tell you about it. Frustration builds a little more slowly. So feedback that shows up on page surveys is not necessarily representative of the experience or effectiveness of THAT page in particular.
That’s it. Those would be my additions or clarifications to a very thoughtful post on how to choose a survey vendor. Any of your own to add?
