Subscribe to the CardioLog Analytics Blog

Tracking Real Funnels: Mixpanel vs. KISSmetrics vs. Google Analytics

By Shay Sharon

Over the past few weeks I have been trying to figure out how to track funnels where the order of the steps is relevant and the conversion can happen over more than one visit.

During my research I came across a great post by Arthur Klepchukov. In his post, Arthur Klepchukov compares three tools: Mixpanel Analytics, Google Analytics and KISSmetrics. For me, the only part missing was the support of real funnel tracking.

If you have not yet read my past posts, a real funnel, in my opinion, is a set of steps that are in a specific order, and the user must go through all of them, in the correct order, before completing a goal. I often call them “chronological funnels” and in actual fact I think that ALL funnels should be chronological funnels. Otherwise they’re not really funnels.

In this post I will try to fill in the missing part and add my comments to the above mentioned comparison.

Mixpanel supports real/chronological funnels. Arthur’s post was published before Mixpanel 2.0 was launched with its improved (and beautiful) funnel report, but version 1.0, Mixpanel does also support chronological funnels. You can try it yourself or read about it in their FAQ page.

Anyway, I created a funnel with 3 steps: A, B and C and ran a quick test on three users:

The first user went from A to B in one visit, and then on to C in a different visit.

The second user went from A to C to B in one visit.

The third user went from A to B to C in one visit.

With Mixpanel I get the following results:

Mixpanel presented the correct results and the setup phase was pretty simple – I managed to implement it on my site in next to no time. Another advantage is the fact that you can use it for free, providing you have up to 100,000 events per month, which is pretty generous.

With KISSmetrics I received the following results, for the same 3 users:

Clearly, KISSmetrics interprets funnels differently. As I mentioned in one of my previous posts, I received the following answer from their Customer Support: When you set up the following funnel: “A -> B -> C”, with KISSmetrics, this means “Of all the people who performed A within the date range, how many also performed B and C – in any order”.

I have a feeling that their huge advantage of allowing their users to create funnels on the fly is actually the reason why they cannot provide a real funnel analytic solution.

I am not quite sure why they used this flow/funnel chart to present the data. It’s a bit confusing. From the screenshot above, I am pretty sure that 10 out 10 people will tell you that you had a 100% conversion rate from step B to C, while this is clearly not the case.

I did, however, manage to get what I was looking for using the same trick that I used with Google Analytics. To use this trick you have to define your funnels in advanced, but I think that is a reasonable enough requirement. If you want to be able to define funnels on the fly, you will have to define all possible combinations. For example, if you have steps/pages A, B and C, you will have to define 15 possible funnels in the tracking code: A, B, C, AB, ABC, AC, ACB, BA, BAC, BC, BCA, CA, CAB, CB and CBA. This is, of course, not realistic if you have a few dozen pages on your site.

After defining the funnel ABC in my JavaScript tracking code, this is the result that I got from KISSmetrics:

I wish KISSmetrics would add a feature for tracking real funnels. I think they have a great product, with an amazing UI, but without this option there are many cases it will not be able to track your funnels correctly.

I have already written about my experiences with Google Analytics and chronological funnels. As long as users go through the whole funnel in one visit, Google Analytics provides a great solution. If however there is a possibility for your users to go through the funnel in more than one visit, then without my workaround or a similar solution you will not be able to use Google Analytics for tracking your funnels accurately.

While I was researching this issue, I tested PadiTrack, a free conversion funnel tracking tool that allows you to create funnels on the fly. This tool imports its data from Google Analytics so I guess the same problem occurs here as well. The screen shot below is the result for the exact same test that I performed with KISSmetrics and mixpanel analytics:

I am not sure how to explain the number in step C. The number in orange (1) indicates the amount people who visited both this step and the previous one, and the number in green (2) shows the amount of people who entered the funnel through this step.

It is currently very hard to find a web analytic tool that provides an accurate chronological funnel data on the fly (if you know of any, please let me know). If the order of the steps is not that important, you can go with Google Analytics or KISSmetrics. But if you can live without the option of creating funnels on the fly, you should definitely go with Mixpanel Analytics, or implement a similar solution to your current analytics system.

Thanks for the detailed analysis on this topic. As of right now KISSmetrics processes data and timestamps events on the day they happened, so when looking at a funnel for one day, you will encounter he issue you described. When looking at multiple days and events that happen across days, the data is accurate. All this was to support funnels on the fly, as you mentioned

We are actually going to be making some changes next week so that the timestamps will be down to the second, thus this discrepancy will no longer exist.

With mixpanel you can dynamically create funnels using any data point you track.

shay.sharon

I didnâ€™t know that. Is this a new feature? As far as I know, In version 1.0 you could not create a funnel and look at historical data – at least that what I was told by Mixpanelâ€™s Customer Support.

I’ve found that often conversion errors are just a simple mis-step by a human. Someone forgot to attached the code to the correct page or in the correct spot, and bam, faulty data.

Heck, just getting website owners to track conversions at all is a big hurdle, right?

shay.sharon

True. Besides reading the data correctly, you have to set it up correctly. It could be a problem with putting the code on the correct page/spot, as you said, but it could also can be an issue with the way you define the funnel on your web analytics tool. One of the best things about MixPanel and KISSmetrics is that you can easily test it as soon as the funnel is created (MixPanel with its real time feature and KISSmetrics with the debugger), while with Google Analytics you have to wait sometimes even a whole day before you can check that you set everything correctly.

I remember when I decided to put time into developing PadiTrack that I had the same problem as you define here which is one of accuracy. Google Analytics doesn’t offer timestamps, so without hacks, getting pinpoint accurate funnels ain’t really possible.

We decided to put development time into the product, believing it can prove itself useful, for one main consideration: for larger data sets there is no much difference between different funnels out there. A big abandonment rate shown in one tool will show just as big in another.

shay.sharon

Hey Claudiu,

I agree that in many cases, the order of the events does not effect the funnel, or at least does not drastically change the Abandonment Rate of the funnel.

Even if most users don’t see a big difference when KISSmetrics launches the fix, this change can definitely change the complete picture for some websites.