The engagement metric, defined (part IV in a series)

SEARCH BLOG

For those of you keeping track at home, this is the fourth in what will likely be a five-part series on calculating an “engagement metric”. The first three posts are here:

Part I

Part II

Part III

Originally I had postulated that an engaged visitor, at least on my web site, can be characterized as follows:

The visitor views “critical” content on the web site

The visitor has returned to the web site recently

The visitor returns directly to the web site some of the time

Some high percentage of the visitor’s sessions are “long” sessions

If available, the visitor is subscribed to at least one available site feed

Basically, the final calculation, one revised thanks to the valuable feedback of dozens of folks, is essentially the same with a few slight modifications. The final goals for my site, goals easily tweaked for any site, are as follows.

Well-engaged visitors will:

View a relatively large number of page views in a given session

Have visited the site in the last four weeks

Have relatively long sessions

Come directly to my site or come from a “Eric Peterson” branded search

Be reading my weblog in addition to non-blog content

Buy one or more of my books through my web site

As you can hopefully see, the first item in my original list (view “critical” content) has been softened somewhat. While the act of purchasing is necessitated by viewing critical content (my “thank you” page) ultimately I agreed with several reader comments that the a priori definition of visitor goals would skew the metric and reduce the metric’s ability to tell me about all of the content on my web site. Thanks to Victor and others for hammering this home.

Given all this, the visitor engagement metric is composed of six sub-metrics, each of which can be examined individually to provide context to the larger calculation. The six sub-metrics are:

Click-Depth Index: Percent of visitor sessions of “n” or more pages

Recency Index: Percent of visitor sessions occurring in the last “small n” weeks

Keep in mind, engagement is a visitor-based calculation, one designed to look at the lifetime of visitor sessions to the web site. So that the engagement of any visitor is a function of their lifetime of visits. Yeah, this assumes some stability in cookies so always use first-party cookies.

The final calculation is simply a summation of the component indices divided by the total number of components which yields a simple percentage:

If you’re looking across multiple visitors, you would read this as “the average visitor is just under 27 percent engaged, as defined by X, Y, and Z.” If you’re looking at a single visitor you can break engagement down on a session-by-session basis, watching for increases and decreases in the visitor’s engagement over time. In aggregate, visitor engagement becomes a very powerful but elegant key performance indicator that tells you a great deal about the make-up of your audience.

Once you decide that you need more information about the basis for an increase or decrease in visitor engagement, and assuming you have the right technology powering your analysis, you would simply visualize each of the core components over time:

As Clint commented in my last post, there is a surprising stability in each of the components, which is in my mind what you’re looking for. I want to see the variation show up when I examine engagement against my business-critical dimensions (referrer, campaign, page, search term, etc.)

When you analyze the visitor engagement calculation against all of your site visitors, you’re looking for a more-or-less normal distribution. This distribution is spiky because of the calculation, but if you’re able to drill-down, you should see something like this:

(The bars that exceed the visualization’s scale represent peaks that occur as visitors achieve 100% of sessions for 1, 2, 3, 4, and 5 of the engagement calculation’s core components. If you want to see this image at 100% scale let me know …)

Another way I can think about this is to use a scatter-plot, basically showing the same thing but easier to visualize differences as you drill-down into specific dimensions:

All of these calculations actually become relevant when you actually apply them to a dimension of data. Here, for example, is visitor engagement mapped to blog posts from my and Avinash Kaushik’s weblog:

Pretty cool, huh? I mean, it’s no great surprise that Avinash’s 2007 Web Analytics Predictions post has the highest visitor engagement score in this image when you think about all of the follow-up predictions his original post spawned. But boy-howdy, isn’t it nice to see that in a metric that you can understand and actually use?!

Here is the visitor engagement metric applied to some of the referrers to my web site:

No great surprise again that Feedburner, Technorati, and WordPress are driving visitor engagement given that they are likely to be driving visitors maxing out their blog index score. But what about the folks at ROI Revolution, sending me visitors who are on average over 30% engaged, or Blackbeak and the folks at Conversion Chronicles, sending me visitors who are as engaged as my 27 percent site-wide average?

Arrrrrrr, indeed!

Finally, and I know that I showed this already but I just think it’s damn sexy, I can map visitor engagement against any geographic dimension in my system (continent, country, city, state, zip code, DMA, etc.) to see where I might want to focus my local marketing efforts in the future:

You can add non-page view events (RIAs, AJAX, Flash, etc.) into the calculation easily. I don’t have much of that on my site but I have an “Event Index” calculation that can be added for sites heavily leveraging these types of applications.

You can add the “social media index” that I discussed in my last post just as easily as you can add content-based indices for retail, customer support, business-to-business, or content.

You can take or leave my idea of scoring the brand index against specific search terms. Visual Site gives me a really easy way to do that and I believe that engagement is very much a function of brand awareness, something notoriously difficult to measure in any practical way.

Visual Sciences customers interested in deploying the visitor engagement metric should contact me directly via normal company channels. I have pretty much everything you need to get up and running with this in a ZIP file and I’d be happy to talk you through the process.

As always, I welcome your comments and feedback on the engagement calculation and anything else that comes to mind. In the next (and perhaps final) installment I will cover Clint’s inevitable complaint of “What the heck happened to all the great ‘social media’ stuff?!?” as well as talk about some specific applications of the engagement metric.

14 Comments

ClintJanuary 15th, 2007

What Happened to the Social Media Stuff? – Well you’ve got part of it covered with your blog index so I guess I would ask What about the Feed stuff and the conversation Index stuff?

I always worry about using ‘average’ anything in web analytics – even though it is a common practice – because ‘average’ assumes a normal distribution and your charts show a non-normal distribution. So in cases like that I immediately want to know what’s the distribution look like (which you’ve shown) and what is the median engagement value?

In other words, what’s the potential that ‘average’ is obscuring what’s really going on with Engagement?

P.S. What’s the likelihood that all these blog posts are leading brides-to-be astray?

EricJanuary 16th, 2007

Clint: I’ll (re)cover the social media overlay in the final post on the subject. In the end I think that you can measure engagement in a non-Web 2.0 world and that this metric does have value. The Web 2.0 stuff just becomes icing on the proverbial cake.

I agree with you about “averages” which is why I have a visualization for the population’s distribution. Still, given the nascence of this calculation and the general tendency for web analytics to rely on averages I would argue that an engagement calculation in which you can examine the distribution is better than no engagement calculation at all. Your thoughts?

You lost me on the brides thing. Sorry!

ClintJanuary 16th, 2007

It was a bad joke, you do remember being engaged don’t you – a groom-to-be?

EricJanuary 16th, 2007

Clint: Phew. I was worried that your comment had something to do with our trip to the Yardhouse last week, an event I am still trying to piece back together. 😉

[…] site engagement using six normalized metrics that combine all aspects of website interaction, which can be found here. He defines engagement as: an estimate of the degree and depth of visitor interaction on the […]

Eric T. Peterson is author of Web Analytics Demystified, Web Site Measurement Hacks and The Big Book of Key Performance Indicators and a long-time member of the web analytics community. He frequently presents on web analytics and is often cited in articles about digital measurement. In the past Mr. Peterson has worked with well-known brands like Microsoft, HP, Cisco, Best Buy, Disney, LEGO, CBS and CBS News, and ESPN. More recently, Mr. Peterson has founded The Analysis Exchange, a completely new way to gain experience with digital measurement.
As an employee of Analytics Demystified Eric is a member of the Digital Analytics Association (DAA), an Adobe Business Partner, and a Google Analytics Certified Partner.