It was great to be able to participate in and help sponsor the Web Analytics Wednesday gathering in San Francisco during the eMetrics Marketing Optimization Summit. Eric Peterson along with others pulled off the largest Web Analytics Wednesday in history...and it was held on a Tuesday! Just goes to show you what an open bar can do! There was great conversation and some fun!
Just back last night from the eMetrics Marketing Optimization Summit in San Francisco. I had the honor of speaking after Tom Davenport; celebrated author of Competing on Analytics: the New Science of Winning. Tom's presentation was excellent and gave the conference a great foundation and focus. Tom did a good job of explaining the potential competitive advantage that can be gained by using analytics in both your strategic and tactical decisions, with great examples from companies like Netflix; Harrahs and even the Boston Red Sox.
This year's summit centered on bringing multiple metrics together, including customer satisfaction/voice of customer. It reminded me of the early days at the eMetrics conferences where the focus was almost exclusively on clickstream analytics. It is great to see the industry moving forward and as a result providing better insight. If you haven't checked it out yet, there is a great white paper on the web metric ecosystem by Eric Peterson. You can get a copy here.
Congratulations to Jim Sterne for once again putting on a great event that not only helps educate the newcomers to the web analytics/metrics world, but also provides a great forum for those of us that have been around this industry for a while.
I’m off to foggy San Francisco next week for the eMetrics Marketing
Optimization Summit. I’m speaking Monday at 10 a.m. about how to quantify
the contribution of the online experience to a company’s overall strategic
goals. You can also stop by our booth (#21 and 22) to say hello. We’re also
going to be sponsoring the Web Metrics Wednesday gathering at a nightclub called
Fluid (662 Mission Street). Oddly enough, Web Metrics Wednesday will be
meeting on, wait for it, TUESDAY, from 6:30-8:30. Anyone can go, not just
eMetrics attendees, and you can register here.
Hope to see some of you there! This
is usually a great show.
Watching discussion of all the thorny political issues raised in this campaign has me thinking about the state of our union. Since ForeSee Results was founded in late 2001, we have:
It’s great to see that the ACSI has become such an industry standard online, and it’s an exciting time to be in this business!
These
are the complaints I hear most often from people in charge of online marketing:
2. The measures show what
the customer is doing, but not why he’s doing it – Many measurement
systems reveal nothing about the driving factor behind behaviors. Following site visitor paths throughout a web
session will show where a customer has been and what they bought, but it
provides no information about the customer's website experience, satisfaction,
or loyalty. Did they visit 15 pages because they were incredibly engaged or
because they couldn’t find what they were looking for? Tracking sales data is
absolutely essential, but it doesn't grant insight into what they didn't buy,
why they didn’t buy it, or what it will take to get them to buy it next time.
4. The value of a website
is measured only by its sales or its ability to generate leads– The website should be
an integrated channel with the rest of your business. Even if a consumer doesn't execute a purchase
on your site or submit a request for more information, their experience will
shape their future buying decisions and their overall impression of your
company.
5. Most metrics provide
only an instant snapshot of customer behavior without long-term implications – Just as it's
important to measure website traffic on a continuous basis, companies should be
measuring customer satisfaction continuously to observe the impact of
seasonality, competitive activity and other changes in the market that effect
the demand for their products and services (e.g. high gas prices, legislative
changes, weather, etc.).
6. Most metrics look at the
past, but reveal nothing about the future – Most web metrics have no predictive
quality. Sound investments are based not
only on what the customer has done in the past, but on what she will do in the
future. Driving a car forward while looking only in the rear-view mirror
invites disaster; similarly, businesses can not only rely on making decisions
based on the past.
What other challenges are you
facing when you think about the metrics you have in place? We’ve structured our
product to address all of these issues, but are there others that keep you up
at night?
Here’s the last segment of my first series of audiocasts with Evan Schuman of StoreFront Back Talk. We talk about the future of the analytics industry and how it’s essential for any analytic vendor to be completely honest and transparent about what their metric is truly capable of (without overpromising or overreaching). The future will be about how metrics work together to provide online companies with a complete picture of actionable data and insights.
Eric Peterson, in his quest for meaningful metrics, has laid
out his proposal for a comprehensive engagement metric.
I love the effort in the pursuit of the “best” metrics,
hopefully helping make our jobs of web analytics not so hard. There are some very interesting components
included in Eric’s visitor engagement metric, such as the brand component. But I still have some skepticism:
First, I’m not sure I really see what this tells me as a
site owner or how to make this actionable. Because engagement, as calculated according
to Eric’s visitor engagement formula can be good or bad.
Fundamentally, what does engagement really tell us? If I’m
in charge of Target.com, I care about what people bought, how the website
influenced in-store sales, etc. Engagement doesn’t really get at what I need to
know because it doesn’t tell me what people will do next week or next month or
how to influence that behavior.
My website objective might be to sell products or services,
to inform, to use as a primary resource, to build the brand, etc. How many
sites have engagement as a website objective? What does engagement tell me
about my success in meeting those objectives?
In Eric’s case, I assume his primary objective is to sell
consulting services, secondary is to sell books. And that the features of his website, like
the blog and the job board, are there to increase likelihood of the first two
things and increase the brand value of Eric. So he should care a lot more about
how engaged potential clients are than how engaged job seekers are. And what about a
long visit that results in a negative comment (although – I am sure that never
happens on Eric’s site :) ).
And again, while I appreciate the attempt to make this
comprehensive, I think by measuring this many things and giving them equal
weight, what you end up with is a watered down metric that will not be able to
predict how well a site will meet the kinds of key objectives I mentioned above.
Bottom line: While a step forward, the visitor engagement
calculation will not measure success. Success is: "did our users accomplish what they wanted?" and "were they
satisfied?"
Avinash Kaushik wrote a great post this week on how you should choose an online survey provider.
Avinash worked extensively with ForeSee Results, so he’s very familiar with the ACSI methodology and how we apply it online. Since he’s been both a customer of web metrics and a partner of web metrics companies, he has a lot of knowledge and has given a lot of thought to what kinds of things people should consider when choosing a tool.
I just want to expand on a few of Avinash’s points:
#1] Mathematical Rigor: No matter what you choose look for a partner that can apply mathematical rigor to your results. I find so many results mis-interpreted because poor math applied in the analysis. Measuring survey results is not simply taking the average of the answers (averages lie!), it is measuring distributions and doing regressions. You don’t want to be bothered to apply the statistics and statistical significance, stress test to ensure your vendor does (then you can focus on analysis and not reporting).
This is a great point that too many people undervalue (just look at the Net Promoter fad: margins of error in the double digits, poorly-designed distributions, and people love the “simplicity” of it!). But I would take this point about mathematic rigor even further. It’s not enough JUST to have mathematical rigor: you can ask bad questions and have a survey pool that is not representative of your audience and you can still use mathematical rigor to interpret the results and get a low margin of error and high confidence intervals. Mathematical rigor ALONE is not what leads you to the holy grail; what leads you there is having a methodology behind the survey that is accurate, precise, and predictive and THEN making sure you analyze those results with mathematical rigor.
This brings me to a comment I had on Avinash’s point #3 (skipping past #2, which I’ll come back to because #3 is related to what I’m already talking about):
#3] Benchmarks & Indexes: Few people in senior management will take action when you tell them “our score on satisfaction (or content or navigation) is 6.0″ (or 45, depending on your vendor). But most of them will get off their butt and give your money to take action when you go to them and say “our score on satisfaction is 6.0 and amazon.com is 9.4 out of 10″.
Your actual score does not drive action, the difference between that and a industry benchmark drives action (no one wants to look bad by comparison!). And here is another subtle human factor: no one wants your opinion about anything, providing a comparison to a external benchmark depersonalizes the number and it is more likely that it will be “heard”.
iPerceptions uses the iPSI (developed by iPerceptions), ForeSee user the ACSI (developed by University of Michigan), both wonderful benchmarks you can compare your scores to and help motivate your management to take action! There are other vendors, look to see if your vendor will provide you with a benchmark or a index to compare.
Another great point, especially about the executive tendency to listen up when you can create an aura of rivalry with a competitor on industry leader. But again, just having a benchmark isn’t enough—anybody can create a methodology and an index and a benchmark. I could create the Larry Satisfaction Index—LSI ™ , write a white paper on it, and call it a methodology, but that doesn’t make it MEAN anything unless it is accurate and precise. An analogy: you’re on the golf course, you look down the fairway and decide it’s 120 yards to the green. You take out your nine iron and hit a perfect shot: exactly 120 yards. But you’re still 30 yards short of the green. You weren’t 120 yards from the green, you were 150 yards! You had a measurement, you had a methodology, but it was worthless because it wasn’t accurate, reliable, or precise.
To to this point about benchmarking, I would only add that people should make sure that whatever benchmark they chose has a proven history of being credible, accurate, precise, predictive, and reliable. What is a proven history? Well, the proof is in the pudding—there should be scientific, academic, published proof that the metrics are linked to financial performance and desirable future behaviors like word of mouth and loyalty. Just ask your vendor to show you the proof that their benchmark actually means something. The American Customer Satisfaction Index methodology has that proof.
Back to Avinash’s #2, about on-the-fly segmentation capabilities:
#2] On The Fly Segmentation Capabilities: Look for the capabilities that are provided to do on the fly segmentation of your data (aggregates lie!) . . .
There is gold in your ability to pick a particular segment of traffic
that absolutely hates you and slice it off and drill down to why they
were on your site, what products they own, what did they not find, and…
then go fix it fast. Segmentation rocks! You want to be able to do it
efficiently, yourself if possible, to reduce chances of vendor delays.
My one addition to this is
that some companies may want to choose a company who will let you do
the segmentation yourself, but it may be that what you really want and
need is a seasoned team of experts to do this for you or help you do.
Data overload is a big problem with metrics, and sometimes companies
plan to do it themselves and end up doing nothing. So look for a vendor
that can enable you to do that segmentation OR that can do it for you.
Last, but not least, Avinash mentions page-level vs. site-level surveys in his discussion of pre-survey considerations:
Page level surveys are user initiated (”tell me what you think of this page” “rate this page” etc) and serve the purpose of measuring the experience of the page. You can ask “site” level experience questions but the way they are initiated by users and local level at which initiation happens makes them a sub optimal choice for measuring site experience. They can good for measuring effectiveness of a page.
Site level surveys are usually presented to the users (”please give us a few minutes to answer these questions”), typically on exit, and are a great measure of the site experience (”what got you here today” “what were you able to do” “how much did we suck” “did you find what you were looking for”). They are good at measuring effectiveness of a site.
I would disagree slightly: page level surveys are more feedback and less measurement per se, for two reasons:
1) Since they are opt-in, they aren’t representative of your larger audience, which means they aren’t a true measurement.
2) The page that people give you feedback on tends to be several pages after the page that caused them the problem, because people don’t get immediately frustrated and opt-in to a survey to tell you about it. Frustration builds a little more slowly. So feedback that shows up on page surveys is not necessarily representative of the experience or effectiveness of THAT page in particular.
That’s it. Those would be my additions or clarifications to a very thoughtful post on how to choose a survey vendor. Any of your own to add?
From Catalog Success
“Customer 2.0” has revolutionized the way that companies are doing business. Customers are more in control of their transactions than ever before and can be in multiple places at multiple times, all thanks to the Internet. At a session I led during last week’s ACCM conference in Boston, the panel and I discussed ways multichannel marketers can get to know such customers better.
For instance, when companies use online surveying methods to find out what their Web customers want, they should be aware that misinformation is often more dangerous than no information at all. It’s critical that companies gather “voice of customer” information that provides strategic insight. Organizations need data that is accurate, precise, reliable and actionable. Otherwise, predictive capability is compromised and the results won’t point you to making improvements that will contribute most to the bottom line .
During the session, we brought out several key factors that companies must remember when conducting online surveys.
1. You can’t manage what you can’t measure. How do you define success unless you have a way to measure it? You can’t manage a Web site without measuring whether you’re meeting your customers’ needs and expectations. Also, survey questions need to be consistent so that over time, you can track trends and gauge progress.
2. What you measure will determine what you do. The results of your measurement will determine where you decide to allocate resources. If you’re asking survey questions about the wrong area, then you’re going to invest in improvements to the wrong area without improving what you set out to improve.
3. Your customer survey measurements must be accurate, precise and reliable. In otherwords, garbage in, garbage out! If your surveys are biased or aren’t representative of your customers, they’re useless. It’s more damaging to have poor surveys and measurement than to have none at all. How you ask questions will influence your responses. So carefully consider whether your questions will lead you to the response you’re looking for, or toward a customer-driven insight.
4. Measure what matters most; your customers.
Don’t forget to support Relay For Life - Help us help others - fighting cancer.
http://www.acsevents.org/relay/mi/westbloomfield/thefreeds?faf=1&e=1081995739
[…] As Larry Freed points out: […]
The draft. The one day that every team and every fan is the eternal optimist. In fact, it is the only day that there is any optimism as a Detroit Lion fan. But we have hope. If they win their next 48 games in a row, Matt Millen, President of the Detroit Lions, will have a .500 winning percentage.
Well, let’s say we all had a draft; a draft of the top attributes you need in a Voice of Customer measurement system. The difference between this draft and the NFL draft is that these picks are “can’t miss.”
With the number one pick in the 2007 Voice of Customer draft
… A high quality measurement: accurate, precise and reliable. A
measurement is only as good as the quality of the measurement. In fact,
a measurement that doesn’t have accuracy, precision and reliability can
do more harm then good. It will give you a false sense of security and
can lead to bad decisions.
With the number two pick in the 2007 Voice of Customer draft …
Predictability. It is great to know where you are, but we also need to
know where we are going. A high quality measurement of satisfaction
should predict the future. If customers are satisfied they will remain
customers, buy more and recommend us to others. If customers are not
satisfied they won’t come back and may even pass their negative views
on to others. Behavior tells us what has happened and satisfaction
tells us what will happen.
With the number three pick in the 2007 Voice of Customer…
Actionability. A voice of customer measurement should be actionable. It
should provide us with the insight to make changes to improve
satisfaction and future behaviors. All too often poor methodologies are
used that do not bring actionability. First, we need a complex set of
measurements, not a single question. A single question provides no
actionability. Secondly we need to have a methodology to drive our
actionability. It is not about what people complain the most about, it
is about what changes can I make that will improve their future
behavior.
With the number four pick in the 2007 Voice of Customer Draft
… Don’t chase your competitors…chase your customers and prospects. It
is important to have benchmarks, and they need to be accurate, precise
and reliable to have any value. But, your objective is not to match
your competitor. Your objective is to meet your customers and prospects
needs. Focus on what will satisfy your customers and prospects, and
change their behavior in a positive manner.
With the number five pick in the 2007 Voice of Customer Draft
… Measure, Measure, Measure. And do it on a continuous basis. The most
important asset any company has is its customers. Let’s measure them on
a continuous basis, because we cannot manage what we cannot measure.
