Avinash Kaushik wrote a great post this week on how you should choose an online survey provider.
Avinash worked extensively with ForeSee Results,
so he’s very familiar with the ACSI methodology and how we apply it
online. Since he’s been both a customer of web metrics and a partner of
web metrics companies, he has a lot of knowledge and has given a lot of
thought to what kinds of things people should consider when choosing a
tool.
I just want to expand on a few of Avinash’s points:
#1] Mathematical Rigor:
No matter what you choose look for a partner that can apply
mathematical rigor to your results. I find so many results
mis-interpreted because poor math applied in the analysis. Measuring
survey results is not simply taking the average of the answers (averages lie!),
it is measuring distributions and doing regressions. You don’t want to
be bothered to apply the statistics and statistical significance,
stress test to ensure your vendor does (then you can focus on analysis
and not reporting).
This is a great point that too many people
undervalue (just look at the Net Promoter fad: margins of error in the
double digits, poorly-designed distributions, and people love the
“simplicity” of it!). But I would take this point about mathematic
rigor even further. It’s not enough JUST to have mathematical rigor:
you can ask bad questions and have a survey pool that is not
representative of your audience and you can still use mathematical
rigor to interpret the results and get a low margin of error and high
confidence intervals. Mathematical rigor ALONE is not what leads you to
the holy grail; what leads you there is having a methodology behind the
survey that is accurate, precise, and predictive and THEN making sure
you analyze those results with mathematical rigor.
This brings me to a comment I had on Avinash’s
point #3 (skipping past #2, which I’ll come back to because #3 is
related to what I’m already talking about):
#3] Benchmarks & Indexes:
Few people in senior management will take action when you tell them
“our score on satisfaction (or content or navigation) is 6.0″ (or 45,
depending on your vendor). But most of them will get off their butt and
give your money to take action when you go to them and say “our score
on satisfaction is 6.0 and amazon.com is 9.4 out of 10″.
Your actual score does
not drive action, the difference between that and a industry benchmark
drives action (no one wants to look bad by comparison!). And here is
another subtle human factor: no one wants your opinion about anything,
providing a comparison to a external benchmark depersonalizes the
number and it is more likely that it will be “heard”.
iPerceptions uses the
iPSI (developed by iPerceptions), ForeSee user the ACSI (developed by
University of Michigan), both wonderful benchmarks you can compare your
scores to and help motivate your management to take action! There are
other vendors, look to see if your vendor will provide you with a
benchmark or a index to compare.
Another great point,
especially about the executive tendency to listen up when you can
create an aura of rivalry with a competitor on industry leader. But
again, just having a benchmark isn’t enough—anybody can create a
methodology and an index and a benchmark. I could create the Larry
Satisfaction Index—LSI ™ , write a white paper on it, and call it a
methodology, but that doesn’t make it MEAN anything unless it is
accurate and precise. An analogy: you’re on the golf course, you look
down the fairway and decide it’s 120 yards to the green. You take out
your nine iron and hit a perfect shot: exactly 120 yards. But you’re
still 30 yards short of the green. You weren’t 120 yards from the
green, you were 150 yards! You had a measurement, you had a
methodology, but it was worthless because it wasn’t accurate, reliable,
or precise.
To to this point about
benchmarking, I would only add that people should make sure that
whatever benchmark they chose has a proven history of being credible,
accurate, precise, predictive, and reliable. What is a proven history?
Well, the proof is in the pudding—there should be scientific, academic,
published proof that the metrics are linked to financial performance
and desirable future behaviors like word of mouth and loyalty. Just ask
your vendor to show you the proof that their benchmark actually means
something. The American Customer Satisfaction Index methodology has
that proof.
Back to Avinash’s #2, about on-the-fly segmentation capabilities:
#2] On The Fly Segmentation Capabilities: Look for the capabilities that are provided to do on the fly segmentation of your data (aggregates lie!) . . .
There is gold in your ability to pick a particular segment of traffic
that absolutely hates you and slice it off and drill down to why they
were on your site, what products they own, what did they not find, and…
then go fix it fast. Segmentation rocks! You want to be able to do it
efficiently, yourself if possible, to reduce chances of vendor delays.
My one addition to this is
that some companies may want to choose a company who will let you do
the segmentation yourself, but it may be that what you really want and
need is a seasoned team of experts to do this for you or help you do.
Data overload is a big problem with metrics, and sometimes companies
plan to do it themselves and end up doing nothing. So look for a vendor
that can enable you to do that segmentation OR that can do it for you.
Last, but not least, Avinash mentions page-level vs. site-level surveys in his discussion of pre-survey considerations:
Page level surveys are
user initiated (”tell me what you think of this page” “rate this page”
etc) and serve the purpose of measuring the experience of the page. You
can ask “site” level experience questions but the way they are
initiated by users and local level at which initiation happens makes
them a sub optimal choice for measuring site experience. They can good
for measuring effectiveness of a page.
Site level surveys are
usually presented to the users (”please give us a few minutes to answer
these questions”), typically on exit, and are a great measure of the
site experience (”what got you here today” “what were you able to do”
“how much did we suck” “did you find what you were looking for”). They
are good at measuring effectiveness of a site.
I would disagree slightly: page level surveys are more feedback and less measurement per se, for two reasons:
1) Since they are opt-in, they aren’t representative of your larger audience, which means they aren’t a true measurement.
2) The page that people give
you feedback on tends to be several pages after the page that caused
them the problem, because people don’t get immediately frustrated and
opt-in to a survey to tell you about it. Frustration builds a little
more slowly. So feedback that shows up on page surveys is not
necessarily representative of the experience or effectiveness of THAT
page in particular.
That’s it. Those would be my
additions or clarifications to a very thoughtful post on how to choose
a survey vendor. Any of your own to add?
.
This seems to be an obvious point. Customer satisfaction has always been a primary driver. What you didn’t mention was the effect that the social networking dynamic has now that it did not have as early as 2 years ago. Now that frustrated customer may actually have influence online because of their own blog/site or the ear if someone who had influence. Now instead of the water cooler, you have a browser.
Another aspect of social networking is customer reviews (when they’re honest and not “reviews for purchase” like BzzAgent), where some critical comments can color opinions of purchasing a specific product or from a specific retailer. My daughter appreciated both the positive and negative comments about some speakers she was planning to purchase — she felt she could live with the negative (comment had more to do with the reviewer’s audiophile preferences) and the positive reviews reinforced her purchase decision.