FREEDyourMind

Netflix Retains Top E-Retailer Title

We've been measuring satisfaction with the top e-retailers for four years now (as defined by Internet Retailer's top 500 Guide). We did the top 40 for two years, and due to popular demand, we increased that to the Top 100 last year.

So, just in case we weren't busy enough hosting our annual user conference this week, we thought it would be fun to also release the Top 100 today (actually, it was timed to be in conjunction with the release of Internet Retailer's Top 500).

Satisfaction with e-retail overall is up 1.4% to 75 (we used the ACSI methodology and its 100-point scale to do the research, as always). 45 of the 100 sites have higher scores this year (many with huge year-over-yea increases) and 17 had lower scores.

Other interesting findings:

• Netflix (86), QVC.com (84), and Amazon (83) hold the top three spots for the fourth year in a row. Netflix has been #1 for four years in a row, and QVC and Amazon have traded back and forth between the #2 and #3 spots. But the consistency of these top sites is amazing-they are setting the standards and expectations for sites across industries.

• Online satisfaction (as measured by ACSI) drives loyalty and sales: highly satisfied online shoppers are 69% more likely to purchase from the retailer next time they're in the market for similar merchandise, 75% more likely to purchase online, 42% more likely purchase offline, and 75% more likely to recommend the retailer. How's that for proof to management that customer satisfaction is a key indicator of online success?

Download the report (for free!) to see who comes out on top in individual head-to-head matchups: Apple vs. HP vs. Gateway; Best Buy vs. Circuit City; Home Depot vs, Lowe's; Nordstrom vs. Neiman Marcus vs. Saks; LLBean vs. Coldwater Creek vs. JCrew; Drugstore.com vs. CVS vs. Walgreens; Netflix vs. Blockbuster; etc. There are 100 of them rated, after all. Almost all of the major e-retailer categories are represented. So check it out if your company is on the list or if you want to benchmark against these top-selling e-retailers. You can also read about it in Computerworld, ClickZ, and Network World, among other places. We have free reports available on the Top 100 overall, along with special reports on apparel/accessories e-retailers and computers/electronics e-retailers.

Response to Avinash

Avinash Kaushik wrote a great post this week on how you should choose an online survey provider.

Avinash worked extensively with ForeSee Results, so he’s very familiar with the ACSI methodology and how we apply it online. Since he’s been both a customer of web metrics and a partner of web metrics companies, he has a lot of knowledge and has given a lot of thought to what kinds of things people should consider when choosing a tool.

I just want to expand on a few of Avinash’s points:

#1] Mathematical Rigor: No matter what you choose look for a partner that can apply mathematical rigor to your results. I find so many results mis-interpreted because poor math applied in the analysis. Measuring survey results is not simply taking the average of the answers (averages lie!), it is measuring distributions and doing regressions. You don’t want to be bothered to apply the statistics and statistical significance, stress test to ensure your vendor does (then you can focus on analysis and not reporting).

This is a great point that too many people undervalue (just look at the Net Promoter fad: margins of error in the double digits, poorly-designed distributions, and people love the “simplicity” of it!). But I would take this point about mathematic rigor even further. It’s not enough JUST to have mathematical rigor: you can ask bad questions and have a survey pool that is not representative of your audience and you can still use mathematical rigor to interpret the results and get a low margin of error and high confidence intervals. Mathematical rigor ALONE is not what leads you to the holy grail; what leads you there is having a methodology behind the survey that is accurate, precise, and predictive and THEN making sure you analyze those results with mathematical rigor.

This brings me to a comment I had on Avinash’s point #3 (skipping past #2, which I’ll come back to because #3 is related to what I’m already talking about):

#3] Benchmarks & Indexes: Few people in senior management will take action when you tell them “our score on satisfaction (or content or navigation) is 6.0″ (or 45, depending on your vendor). But most of them will get off their butt and give your money to take action when you go to them and say “our score on satisfaction is 6.0 and amazon.com is 9.4 out of 10″.

 

Your actual score does not drive action, the difference between that and a industry benchmark drives action (no one wants to look bad by comparison!). And here is another subtle human factor: no one wants your opinion about anything, providing a comparison to a external benchmark depersonalizes the number and it is more likely that it will be “heard”.

 

 

iPerceptions uses the iPSI (developed by iPerceptions), ForeSee user the ACSI (developed by University of Michigan), both wonderful benchmarks you can compare your scores to and help motivate your management to take action! There are other vendors, look to see if your vendor will provide you with a benchmark or a index to compare.

 

Another great point, especially about the executive tendency to listen up when you can create an aura of rivalry with a competitor on industry leader. But again, just having a benchmark isn’t enough—anybody can create a methodology and an index and a benchmark. I could create the Larry Satisfaction Index—LSI ™ , write a white paper on it, and call it a methodology, but that doesn’t make it MEAN anything unless it is accurate and precise. An analogy: you’re on the golf course, you look down the fairway and decide it’s 120 yards to the green. You take out your nine iron and hit a perfect shot: exactly 120 yards. But you’re still 30 yards short of the green. You weren’t 120 yards from the green, you were 150 yards! You had a measurement, you had a methodology, but it was worthless because it wasn’t accurate, reliable, or precise.

 

To to this point about benchmarking, I would only add that people should make sure that whatever benchmark they chose has a proven history of being credible, accurate, precise, predictive, and reliable. What is a proven history? Well, the proof is in the pudding—there should be scientific, academic, published proof that the metrics are linked to financial performance and desirable future behaviors like word of mouth and loyalty. Just ask your vendor to show you the proof that their benchmark actually means something. The American Customer Satisfaction Index methodology has that proof.

 

Back to Avinash’s #2, about on-the-fly segmentation capabilities:

 

#2] On The Fly Segmentation Capabilities: Look for the capabilities that are provided to do on the fly segmentation of your data (aggregates lie!) . . .

There is gold in your ability to pick a particular segment of traffic that absolutely hates you and slice it off and drill down to why they were on your site, what products they own, what did they not find, and… then go fix it fast. Segmentation rocks! You want to be able to do it efficiently, yourself if possible, to reduce chances of vendor delays.


My one addition to this is that some companies may want to choose a company who will let you do the segmentation yourself, but it may be that what you really want and need is a seasoned team of experts to do this for you or help you do. Data overload is a big problem with metrics, and sometimes companies plan to do it themselves and end up doing nothing. So look for a vendor that can enable you to do that segmentation OR that can do it for you.

Last, but not least, Avinash mentions page-level vs. site-level surveys in his discussion of pre-survey considerations:

 

Page level surveys are user initiated (”tell me what you think of this page” “rate this page” etc) and serve the purpose of measuring the experience of the page. You can ask “site” level experience questions but the way they are initiated by users and local level at which initiation happens makes them a sub optimal choice for measuring site experience. They can good for measuring effectiveness of a page.

 

Site level surveys are usually presented to the users (”please give us a few minutes to answer these questions”), typically on exit, and are a great measure of the site experience (”what got you here today” “what were you able to do” “how much did we suck” “did you find what you were looking for”). They are good at measuring effectiveness of a site.

I would disagree slightly: page level surveys are more feedback and less measurement per se, for two reasons:

 

1) Since they are opt-in, they aren’t representative of your larger audience, which means they aren’t a true measurement.

 

2) The page that people give you feedback on tends to be several pages after the page that caused them the problem, because people don’t get immediately frustrated and opt-in to a survey to tell you about it. Frustration builds a little more slowly. So feedback that shows up on page surveys is not necessarily representative of the experience or effectiveness of THAT page in particular.

That’s it. Those would be my additions or clarifications to a very thoughtful post on how to choose a survey vendor. Any of your own to add?

Keeping Up With the Joneses

I met with a CEO last week who was only interested in one metric: how his company performed in terms of customer satisfaction when compared to their nearest competitor.

Comparing yourself to your can be an important reality check (and every company should monitor their performance relative to competitors as part of a dashboard of metrics), but it shouldn’t be a guiding principal or sole metric. Let’s not lose the forest for the trees.

Let’s say your competition has a score of 75 for navigation and yours is a 71. Does that mean yours needs to be better? Maybe. But more important is to see if improving that score of 71 in navigation will impact their behavior. Will they buy more, return more frequently or recommend to more people? The impact of changing is what should drive your decision, not just the comparison of the scores. It is about impacting the behavior of your customers and prospects, not keeping up with the Joneses? Is there another area you could improve (product descriptions, search, etc.) that would have a bigger impact on your customers’ likelihood to buy, recommend, and return?

Don’t spend too much time chasing your competitor; spend your time chasing your customers and prospects! Improving their satisfaction will yield significant improvements to your bottom line.. Satisfy your own customers in the ways that are most important to them, and use competitive benchmarks as one piece of the puzzle.

March Madness is Upon Us

tIt's that time of the year. Time for March Madness.

The first theme of March Madness to think about is strength of schedule. Who gets into the NCAA tourney and who doesn’t often depends on what teams you beat, and how good those teams were. Why is that so important? For the same reasons companies look to benchmark their performance against their peers. It helps you determine whether your performance is good, bad or average.

So what does this have to do with web metrics?

First, all things can relate to the game of basketball. But more importantly, let’s examine the role of benchmarking in our web strategy.

Chase your prospects, not your competitors!

It is good to be able to compare to your peers and other leading websites, but that should not be the driving force in identifying where to focus your improvement efforts. It is not about matching your competitors, but rather about making improvements in those areas that will improve your prospects’ satisfaction and their likelihood to become a customer.

With benchmarks, quality is the key!

Benchmarks are great. But if you have a poor measurement device, your benchmarks are worthless. Without accuracy, precision and reliability you have nothing. Actually you have less then nothing. If you have a poor measurement device it may give you a false sense of security that know what to do. You are better off ignoring poor data and going on instinct. The saying “Garbage In, Garbage Out” definitely applies to poor measurement.