Web Analytics Tools Comparison: A Recommendation

A frequent query I get via email is to recommend a web analytics tool. Something like: “is Omniture better than WebTrends” or “would you recommend CoreMetrics or WebSideStory” or “is Visual Sciences worth its cost” or ….. you get the idea.

I rarely recommend a specific tool in reply to emails from readers (see Disclaimers & Disclosures), I usually reply back with things to consider when selecting a tool. I recommend these posts:

As I am fairly new to reading posts and I am spending a lot of time off work to find solutions I was wondering if you have ever done any comparison on the different tools out there? Webtrends vs Omniture vs HBX and so on? If so can you point me to some good comparisons?

I am able find a lot of information on all the products but no true comparisons. Not sure if this information is in your book but I will be picking that up this week so I will see then

The nuance on looking for comparisons was interesting and also that the reader had already looked and not found anything really helpful.

I am afraid that in my reply I could not point to any comparisons that might be really helpful. I have of course read loads of ‘em from individuals and agencies and consulting companies and industry Analysts. But either the quality is uncertain or the author’s experience with real world needs / implementation / uses is sorely lacking.

What is available is well meaning but rarely will it give you, as the reader put it, true comparison. Lots of check boxes where X tool meets Y criteria or not compared to its competitors, but enough to give you confidence to really choose? Not really.

Buying a web analytics tool is like buying a car, you can do all the research on the web you want but you have to get your butt into the seat of a real car and take it for a test drive and see how you feel about it. Press the buttons, get a sense for how your hands feel on the steering wheel, what happens when you take quick turns or go over pot holes on the road. Then you start to get a real idea of what the difference is between a Saturn Ion and Toyota Matrix.

Each web organization is unique when it comes to its culture, existing systems, IT skills and infrastructure, the need for reporting or analysis etc (even if different organizations are in exactly the same line of business). What works in your favor is that in our world you don’t have to take the tools you want to compare for a 30 min test drive. Any vendor, especially one that wants your business, will let you implement their tool on your site for free for a few weeks and take it for a spin.

Bottom-line: If you want to pick the right tool for your company you’ll have to actually try a couple in real life on your real site before you can decide which one is optimal for you.

If you do decide that you are going to choose a tool not based on a read-only (or worse Vendor Marketing VP PowerPoint presentation) then here is something to make your process less painful: Pick one tool from each of the three buckets below to ensure you get the broadest exposure to what is possible.

Bucket One : Omniture, HBX (WebSideStory), WebTrends, CoreMetrics.

Bucket Two : ClickTracks, IndexTools, Unica NetInsight.

Bucket Three : Google Analytics, Microsoft Gatineau.

Rather than choosing from tools that will give you kinda – sorta the same kinds of features or functionality the bucketing recommended above will ensure that you make the most optimal decision from a diverse set of choices.

I find that very often people choose from tools that are very similar and ultimately their final choice is not the best. Every once in a while you need a Ralph Nader in the mix!

Important: There is no implication in terms of features or greatness or price etc in the bucketing system. 75% of what you’ll need is actually present in any tool in any bucket. The reason each tool ended up in its bucket is because that bucket offers something compelling and different.

Bucket One “What’s Unique About Me” Sound bites:

Omniture will literally do anything you want it to do with a ever expanding set of features, HBX is your friend if your company’s best friend is Excel, WebTrends is morphing into messaging itself as a customer insights company, CoreMetrics has a few unique features for retailers. These are tools that are often top of mind when people think of web analytics. Each does something better than the other, but there is also a large overlap.

Bucket Two “What’s Unique About Me” Sound bites:

If you want to do real post facto analysis then ClickTracks will shine, I wish a lot more people realized how awesome IndexTools’s custom reporting interface is (Ajax!!) and it comes at a great price, NetInsight integrates efficiently with your online and offline campaigns. Each of these tools provides a truly compelling alternative to bucket one and three.

Bucket Three “What’s Unique About Me” Sound bites:

Google Analytics brings 100% of the standard web analytics reports with integrated end to end adwords reporting, for free. Microsoft Gatineauwill join the group hug soon (Ian: we are all waiting with a bated breath!). Both of these tools will prove to you that you only have to pay if you can prove to yourself that your needs are complex enough to need something special.

[I am not including Visual Sciences in the mix above because it does stand by itself. It is a true analysis tool, that some might say is expensive (based on pricing shared by a competitor at Emetrics 2007 in London). But if you have true advanced Analyst skills to apply in your web organization, and support from your IT team for implementation, then then you’ll be able to glean far superior actionable insights.]

Summary: Compare by implementing. Learn by using. Identify the best cost – benefit choice by having atleast one tool from each of bucket above in your comparison.

I had written a post in September 2006 that lays out how you can choose the right tool for you. It recommends a linear process (start with a free tool, implement and learn, identify gaps, then go for comparing paid tools – ones in bucket one and two above).

See what I mean? Picking the right tool for you is harder than finding a wife / husband!!

Have web based or Analyst comparisons helped you choose the right tool? Do you agree with how each tool has been bucketed? No? Have a different soundbite for a tool? Please share your feedback, critique, expertise via comments.

Comments

I not only think you are spot on in your recommendations, but it seems like – and this is a subjective comment based on my direct client experiences over the last year – most of the clients we deal with (from a RFP, RFI or any other sales pitch point of view) do exactly that!

We end up competing with one of the four American (bucket one) companies every time. Not because ClickTracks or NetInsight does not exist – but clients tend to add in only one or two Contenders (that is what I called bucket two – sounds better …. and you can add “Eye of the Tiger” as background music while reading this.. ha ha)

“It is a true analysis tool, that some might say is expensive (based on pricing shared by a competitor at Emetrics 2007 in London)”

That competitor was ME! :-)

And I stand by it. ANY client looking into a price difference of some $400,000 per year on a average+ sized web property – should make sure that they not think twice and hard on how to justify a additional tool cost of that size.

I truly believe that the Analyst is the one creating the questions and the one trying to figure out the answers and the one finally recommending potential opportunities for improvement based on his findings. Do not get me wrong; I love Eric Peterson and his Visual Site as much as anybody else, but if I had the opportunity between a Contender (like us) and the opportunity to hire three “Avinash’s” (or would $400K only get me 2? :-) – I would go for the contender and the 2 to 4 additional Analysts any time.

In your comparison of tools, you muight want to look at the other methods of collecting data that do not need logs or tags. Our passive capture engine is used by a number of the largest e-commerce vendors because tagging is just too much work and standard server logs are too massive to contenplate processing. We can emulate the tags that packages like Webtrrends expect at a fraction of the implementation cost, and more importantly, teh maintenance.

Passive capture has the other huge advantage in that web data is available to experiement with with out altering web servers, etc. which is the organization's bread and butter. This means taht you can experiment with fraud-detection, compliance, feeding other data warehouses, or whatever without the fear of harming your web site.

True. This discipline is call web analytics. Any major should do a great job providing with report. You need to act with those data afterward, this is what this discipline so powerful. The rest is ease of use I believe.

It turns out that, thanks to Phil Kemelor and the fine folks at CMSWatch we can all stop worrying about paying expensive and potentially biased consultants to help decide which analytics tool is most appropriate. I've just finished browsing a 281 page review copy of Phil's "Web Analytics Report" and I have to say he did an ** excellent job ** and has produced the most comprehensive and technically complete document covering specific web measurement technologies ever written.

While Visual Sciences was not able to participate at the level Phil wanted, pretty much everything you'd want to know about the other major vendors is contained in this document. And while some will consider $1,175 pretty steep, given the cost of the alternatives and the amount of work that went into the document, this is a thousand bucks that could save you tens of thousands of dollars worth of headaches down the road.

Anyway, if you're in the market for some unbiased research on the vendors, check out Phil's work here:

cmswatch.com/Analytics/Report/

If you're not sure about dropping a grand on the report, you can get a free sample (not sure exactly what you get) here:

In my experience, companies spend FAR too much time looking at product features instead of figuring out how they are going to use the information.

When considering an analytics solution, companies should first determine the questions they need to have answered, and then look for a tool that helps them find those answers.

We prefer Indextools over even the most expensive analytics packages out there because of it can be extensively customized, it has a very useful API, and it's more competitively priced than other enterprise level tools. Heck, you can get a basic account for $50 a month!

1. Your latest post you mention that Visual Sciences is out there by itself. I'm wondering whether you'd class SAS eBIS tool in the same light. Given the fact that both are enterprise solutions, integrate seamlessly with campaigns and additional data-sources and customisable/shared dashboards for users/content groups. I would say that Visual Sciences does have a massive appeal with functionality, the unusual, but effective database structure and the sheer quickness of the tool – and SAS whilst benefited from a world class back end does have a lot of ground to make up for their reporting interface and dashboards.

2. Our mutual friend (and your fellow director of Web Analytics Association (congratulations by the way)), Neil Mason, did a fantastic slide at eMetrics showing what he perceived as the place of each tool (believe the axis were cost and functionality). Thought this may agree with your blog and perhaps would be a good addition.

3. With regards to the statement about "taking it for a spin". I'm very aware of the exorbitant costs of running a proof-of-concept (POC) and the pressures that sales people put firms. I feel that the high cost of POCs may force managers to go with the tool despite a rocky/unsuccessful implementation and trial – merely because the tool is already installed and the accountants (which run most companies) say "Justify this expenditure of implementation, removal and the tool itself". In financial services (for the last three years) getting any dedicated IT resource for web analytics is pretty much on par with trying to get to the moon with an elastic band and a lollypop stick.

So that there is a helluva lot more cost to consider that managers perhaps don't realise on the onset.

I feel that a level of standardisation (which I hope the WAA can help on a global basis) will enable non-technical users and non-experienced companies to understand web analytics, measurement and its importance in the 21st century. Looking at my most recent blog £72bn ($150bn) in the UK will be spent online by 2010.

I like your segmentation and I think that you are right in advising to test one of each. Testing seems to be the 'silver bullet' lately ;-)

Eric, if you allow me, as manager of a few of those 'expensive and potentially biased consultants' ;-):

I would like to say that a good consultant will also take into consideration the needs of the client (and if he's a good one, not only from a technical point of view). At OX2 when helping a client (and I say help, not taking the decision on their behalf) we always start by discussing the needs and this sometimes leads to a workshop or a bunch of interviews. IMHO with the complexity of Web Analytics, before engaging hundreds of thousands of dollars/euros, it is wise to be helped by someone who has seen lots of different projects and clients in order to avoid the most common pitfalls and try to engage the Journey the right path. And in the end it's a question of testing before buying. Nevertheless I think that the report will be very valuable for users already familiar with WA, but I don't think that someone that doesn't know a thing about WA should take the decision only based on a written report. Disclaimer: Eric, I love you too ;-)

Avinash, if I may, I would like to add a couple of variables in the selection process you describe:

First is to see if there's not yet an existing product somewhere in the company. For example, IT guys sometimes have a WA product somewhere in the basement. This can sometimes avoid expensive licence fees (I've seen a few examples). And you might even will find out that your IT guys can provide you with technical support because one of them attended a training! This should be taken into consideration when deciding.

The second variable is about your favourite topic: Brains ;-) If you have in your team, let's say, someone with 3 years experience in the product X, you might consider to acquire product X as it will allow to speed up your Web Analytics Journey. Taking into consideration that WA takes time, this is not something to overlook.

1. Your latest post you mention that Visual Sciences is out there by itself. I’m wondering whether you’d class SAS eBIS tool in the same light.

I have not had enough experience with SAS eBIS, the last time I saw it was a couple of years ago and I am sure it is not the same anymore. But based on the presentation from SAS at the Emetrics summit panel it seems that SAS eBIS might be a compelling alternative (and I was surprised that SAS was honest about the true cost and kinds of clients who would use the tool, that honesty is refreshing).

Most analytics tools that need heavy $$$ investment are used by big companies (say more than 7 – 10k employees). These same companies are full of "business or functional" units and loads of teams, layers, process, bureaucracy. Expensive tools (be they Visual Sciences, SAS or SPSS or…) needs superheros to pull of any insights because of the sheer intelligence and complexity (or the data, tool, business). Usually there aren't a lot of superheros around.

So I have this simple "Expensive Analytics Tool Cultural Test":

In your big company is the cultural environment such that a few (one, two or three) people can do all the analysis and find the insights and that is sufficient to drive major changes to your webiste / business / organization?

Or your big company cultural environment is such that loads of people / analysts need to have access to data and rather than massive global decisions made by a few, change happens by numerous local decisions made by the many?

The choice above dictates what tool you need (sort of in the vein of what Rene's comment above).

I am not making a judgment call on which is better. I am encouraging a strong and self critical review of the organization and then finding a fit.

Analytics tools rarely fail. We let them down because we don't make the right choice for our companies.

I could not locate the slide you mention in Neil's deck but I'll ask him when I see him next. Neil is a leading authority in our space so it is unsurprising that he is full of fantastic slides! :)

3. With regards to the statement about “taking it for a spin”. I’m very aware of the exorbitant costs of running a proof-of-concept (POC) and the pressures that sales people put firms

This I completely agree with, especially for traditionally huge companies. More than the work to be done it is the process and people convincing hurdles that have to be overcome.

But at the same time I don\'t know what the options are. Most people select the wrong tool because they do the selection process wrong. My thought is that if you want to know what you need then you have to put in the effort. But having been in a few huge companies I know how really really hard this is.

Thanks again for the comment Miles, it enriches the discussion and is very welcome.

You probably want to check out WebAbacus too. Our company had reviewed Webtrends, ClickTracks, IndexTools, among many others. We've even had a custom built tool created for POC using SAS technology. Suffice to say, at the end of our evalutation exercise, we chose a WebAbacus for its transparency, robustness, flexibility, and very powerful and fully customizable reporting features, at a fraction of the price of its competitors (at time of evaluation).

If you want a in-house hosted solution then that is your option, you can use it with javascript tags.

If you want a in-house solutions for your web log files, it is your solution.

Urchin is astonishingly cheap and provides world class reporting. I encourage you, and others, to try Urchin if you are looking for a in-house solution (others in that category include Unica, WebTrends).

A good list, thanks. We’ve had a great experience with Overstat (overstat.com), which allows you to pull down a window over a web page and see the stats for that page in real time. See also my list of tools:
digitalvinci.com/dv/post/Top-10-Web-Analytics-Tools-%28That-Are-Not-Google%29.aspx.

[…] I've got more details coming on the other products I've been comparing, but if you are itching for more, now, I sugget you check out Avinash Kaushik's Web Analytics Tools Comparison: A Recommendation. Bookmark on del.icio.us […]

[…]
Can I read more about web analytics tools?
Yes, check out Sitepoint, they provides a list of tracking tools. Avinash Kaushik also provides a good blog about what to ask web analytics providers.
[…]