FREEDyourMind

The Limitations of Traditional Online Surveys

I could probably write a book with this title (hey...there's an idea!), but the reason I'm thinking of it now is that I was just popped a survey on a magazine website.

It asked all the basic demographic information that might be useful for research. But so much more was left on the table.  It is great to gather information, but if you are going to put your site visitors through a survey, make sure you are making the most of the opportunity. 

Some quick thoughts (or you could call them "must haves"):

• Make sure you are getting a representative audience to complete the survey.  Without it you have individuals opinions (there is some value there) but with a representative audience you have taken the first step towards having a useful metric.
• Demographics can be important,  but lets also understand what is driving their satisfaction and what is likely to result from the online experience (loyalty, retention, purchase online, purchase offline, etc.).
• Understand the relationship between those drivers of satisfaction, overall satisfaction, future behaviors and financial success. It is great to know where we are today, but the holy grail is to know what we need to do to improve -- and what impact it will have on our success.  This is not an easy task, but it is the key to turning voice of customer into actionable information.
• Rely on a proven methodology.

I like to think of it as "applying science to voice of customer."

Marrying Online Customer Satisfaction Surveys and Interactive Promotions

Jeff Blackman is the CFO at ForeSee Results, and will be a guest blogger at FreedYourMind. This is his first post.

Hi everyone.  My name is Jeff Blackman, and I am honored to be a guest blogger on freedyourmind.com. I’m the new CFO at ForeSee, and this is my first blog entry, so you’ll have to excuse me if I don’t use proper “blog” etiquette – if there is such a thing!  Having been associated with both ForeSee Results and with ePrize (the market leaders in online customer satisfaction and interactive promotions respectively), it strikes me that a natural synergistic relationship exists between these services.  While measuring online customer sat and engaging in online promotions are both powerful in and of themselves, the combination is truly 1+1=3.  After all – one of the primary intents of an interactive promotion is to engage your customers to increase awareness and perception of your brand.  How best to determine if the promotion is achieving these goals than through measuring a visitor’s experience via a customer sat survey that is scientific enough to be able to predict customer behavior?

For example, you may know that 200,000 consumers registered for your promotion and your goal may have been 300,000.  On the surface this appears to be disappointing – right?  Not necessarily – the real question is: Did the consumer‘s perception of your brand increase and thus are they more likely to engage with your brand in the future?  A pre and post customer sat measure would give you the answer.  The fact of the matter is that if your brand’s perception / satisfaction increased significantly with a high percentage of the 200,000 registrants the promotion was probably a winner. And you should go beyond satisfaction… the ACSI methodology that ForeSee Results uses can actually tell you if people who registered for the promotion are more or less likely to buy, be loyal, recommend you, etc.

As we all know, the goal is not just the number of registrants but the intersection between the number of qualified registrants and the (hopefully) positive impact to customer sat that you generate.  As has been proven time and time again, there is a strong link between consumer satisfaction and financial results.

Furthermore, a customer satisfaction survey coupled with an interactive promotion will allow you to gain deeper insight into your customer’s mindset.  In the attempt to keep registration pages as lean as possible to increase participation, many critical data points are understandably not collected.   Besides the core questions that are part of the model, customized survey questions can help answer questions such as where the consumer was driven from (a key stat as we all know that it is imperative to “promote” your promotion properly).  In addition you may want to know if the customers were driven by an attraction to the brand (higher quality leads) or by the incentive itself which may have attracted gamers to your site, who will puff up your number of participants but may not be likely to buy or recommend.  This information will allow you to customize your next interactive promotion – in terms of look, feel and prize, in order to drive even more participation and higher quality leads.  It is these higher quality leads which will drive your ROI (and likely allow you to lobby for an increase in your marketing budget).  While we have seen clients take advantage of the synergistic effects of these two services – all indications show that this trend will continue to increase significantly over the next year as brand managers are being asked to prove that their interactive promotions are achieving the goal of increasing awareness and perception of their brand. 

Voice of Customer Analysis

At a recent conference, I heard a CMO say that what she liked best about her customer satisfaction vendor was that she got a list of open-ended comments every day. It made her feel like she had her finger on the pulse of what her clients were thinking and saying and feeling. Her vendor would lump comments in to positive or negative buckets for her, so she felt like she’d have an early warning sign if the tide started to shift against her company.

Like most things in life, getting value out of open-ended comments is unfortunately a bit more complicated than that. You need to be able to take customer comments in the right context—not giving too much oil to the squeaky wheel, not ignoring the people who are mostly happy except for a few small issues that can and should be addressed just because they don’t complain loudly enough.

A comment by itself is just a loud voice. We want to hear that voice, but we want to hear it in context.

1. How satisfied and how likely to buy are the people leaving those comments? You might get a cluster of comments complaining about prices, but when you crosscheck the comments with the satisfaction scores, you see that all the people who are complaining about prices are likely to buy anyway. Should you be scrambling to change prices in that case?

You might get another cluster of comments with mild complaints about fulfillment, only to find out when you cross-reference their comment with their likely future behaviors that this particular complaint is actually keeping people from coming back.

Which of those two issues should be top on your agenda?

2. Let comments shed light on things you already know are a problem, but don’t let them decide what the priorities should be. Just looking at comments is not a scientific way to determine what the real issues are. But comments can help you hone in on things that your scientific surveys have already identified as problems.

The key here is to be able to weight voice of customer. Every customer is important, but not every customer’s voice should be equal when you’re making business decisions. You have to have a way to make what you learn in open-ended comments actionable, and using customer satisfaction to analyze and group open-ended comments can help you do that.

The Trouble With Page-Level Surveys

It’s becoming more and more common for companies to deploy opt-in page-level surveys in order to hone-in on problems with specific pages.

The problem is that if something isn’t working for you on a website, you don’t complain on that page, you complain when you are good and frustrated 3 pages later. The first time something doesn’t work, you aren’t necessarily going to pipe up and tell someone with an opt-in survey, you’re going to poke around for a few seconds and try to figure it out on your own.

The results can be misleading.

Response to Avinash

Avinash Kaushik wrote a great post this week on how you should choose an online survey provider.

Avinash worked extensively with ForeSee Results, so he’s very familiar with the ACSI methodology and how we apply it online. Since he’s been both a customer of web metrics and a partner of web metrics companies, he has a lot of knowledge and has given a lot of thought to what kinds of things people should consider when choosing a tool.

I just want to expand on a few of Avinash’s points:

#1] Mathematical Rigor: No matter what you choose look for a partner that can apply mathematical rigor to your results. I find so many results mis-interpreted because poor math applied in the analysis. Measuring survey results is not simply taking the average of the answers (averages lie!), it is measuring distributions and doing regressions. You don’t want to be bothered to apply the statistics and statistical significance, stress test to ensure your vendor does (then you can focus on analysis and not reporting).

This is a great point that too many people undervalue (just look at the Net Promoter fad: margins of error in the double digits, poorly-designed distributions, and people love the “simplicity” of it!). But I would take this point about mathematic rigor even further. It’s not enough JUST to have mathematical rigor: you can ask bad questions and have a survey pool that is not representative of your audience and you can still use mathematical rigor to interpret the results and get a low margin of error and high confidence intervals. Mathematical rigor ALONE is not what leads you to the holy grail; what leads you there is having a methodology behind the survey that is accurate, precise, and predictive and THEN making sure you analyze those results with mathematical rigor.

This brings me to a comment I had on Avinash’s point #3 (skipping past #2, which I’ll come back to because #3 is related to what I’m already talking about):

#3] Benchmarks & Indexes: Few people in senior management will take action when you tell them “our score on satisfaction (or content or navigation) is 6.0″ (or 45, depending on your vendor). But most of them will get off their butt and give your money to take action when you go to them and say “our score on satisfaction is 6.0 and amazon.com is 9.4 out of 10″.

 

Your actual score does not drive action, the difference between that and a industry benchmark drives action (no one wants to look bad by comparison!). And here is another subtle human factor: no one wants your opinion about anything, providing a comparison to a external benchmark depersonalizes the number and it is more likely that it will be “heard”.

 

 

iPerceptions uses the iPSI (developed by iPerceptions), ForeSee user the ACSI (developed by University of Michigan), both wonderful benchmarks you can compare your scores to and help motivate your management to take action! There are other vendors, look to see if your vendor will provide you with a benchmark or a index to compare.

 

Another great point, especially about the executive tendency to listen up when you can create an aura of rivalry with a competitor on industry leader. But again, just having a benchmark isn’t enough—anybody can create a methodology and an index and a benchmark. I could create the Larry Satisfaction Index—LSI ™ , write a white paper on it, and call it a methodology, but that doesn’t make it MEAN anything unless it is accurate and precise. An analogy: you’re on the golf course, you look down the fairway and decide it’s 120 yards to the green. You take out your nine iron and hit a perfect shot: exactly 120 yards. But you’re still 30 yards short of the green. You weren’t 120 yards from the green, you were 150 yards! You had a measurement, you had a methodology, but it was worthless because it wasn’t accurate, reliable, or precise.

 

To to this point about benchmarking, I would only add that people should make sure that whatever benchmark they chose has a proven history of being credible, accurate, precise, predictive, and reliable. What is a proven history? Well, the proof is in the pudding—there should be scientific, academic, published proof that the metrics are linked to financial performance and desirable future behaviors like word of mouth and loyalty. Just ask your vendor to show you the proof that their benchmark actually means something. The American Customer Satisfaction Index methodology has that proof.

 

Back to Avinash’s #2, about on-the-fly segmentation capabilities:

 

#2] On The Fly Segmentation Capabilities: Look for the capabilities that are provided to do on the fly segmentation of your data (aggregates lie!) . . .

There is gold in your ability to pick a particular segment of traffic that absolutely hates you and slice it off and drill down to why they were on your site, what products they own, what did they not find, and… then go fix it fast. Segmentation rocks! You want to be able to do it efficiently, yourself if possible, to reduce chances of vendor delays.


My one addition to this is that some companies may want to choose a company who will let you do the segmentation yourself, but it may be that what you really want and need is a seasoned team of experts to do this for you or help you do. Data overload is a big problem with metrics, and sometimes companies plan to do it themselves and end up doing nothing. So look for a vendor that can enable you to do that segmentation OR that can do it for you.

Last, but not least, Avinash mentions page-level vs. site-level surveys in his discussion of pre-survey considerations:

 

Page level surveys are user initiated (”tell me what you think of this page” “rate this page” etc) and serve the purpose of measuring the experience of the page. You can ask “site” level experience questions but the way they are initiated by users and local level at which initiation happens makes them a sub optimal choice for measuring site experience. They can good for measuring effectiveness of a page.

 

Site level surveys are usually presented to the users (”please give us a few minutes to answer these questions”), typically on exit, and are a great measure of the site experience (”what got you here today” “what were you able to do” “how much did we suck” “did you find what you were looking for”). They are good at measuring effectiveness of a site.

I would disagree slightly: page level surveys are more feedback and less measurement per se, for two reasons:

 

1) Since they are opt-in, they aren’t representative of your larger audience, which means they aren’t a true measurement.

 

2) The page that people give you feedback on tends to be several pages after the page that caused them the problem, because people don’t get immediately frustrated and opt-in to a survey to tell you about it. Frustration builds a little more slowly. So feedback that shows up on page surveys is not necessarily representative of the experience or effectiveness of THAT page in particular.

That’s it. Those would be my additions or clarifications to a very thoughtful post on how to choose a survey vendor. Any of your own to add?