Oracle Blog

Focus on eMarketing and Electronic Software Distribution

Friday Aug 28, 2009

I wrote an overview earlier this month about Predictive Analytics
(PA), what it is and what sorts of benefits it offers. I've also
written about the Sun Machine Learning Engine (SMILE) Project, putting
PA to work on Sun's web sites. Today marks the end of SMILE Phase 1, as
we are in the midst of releasing SMILE v2.0. You'll see the results on
www.sun.com over the course of the next month, as we gradually enable
more and more of the site with our new "Recommendations for you"
carousel. Here's what you'll be seeing soon!

So I thought today would be a good time to touch on the results from
SMILE 1.0. The initial release was simply about serving small text-only
ads in the right hand column on many sun.com sites. Here's an example
of a SMILE-served ad (outlined in red):

To evaluate results, we calculated standard metrics, such as impressions (number of times
the ad was displayed), clicks (number of times the "call to action"
link in the ad was clicked), and CTR (click-through rate -- number of
clicks divided by number of impressions, as a percentage). I'm not able
to share the detailed CTRs at this time, but suffice it to say they're low.
That's not unexpected -- these are small ads, easily overlooked or ignored, and you shouldn't expect a lot
of clicks on this type of banner ad.

However, we also calculated Uplift, and that's where we looked to
measure the power of our analytics. We did not have recommended ads for
all customers, just return visitors with anonymous cookies that we
recognized from earlier visits. Thus, we served many default ads, as
well as many recommended ads. By comparing the CTRs of both, we can
explicitly measure the influence of our predictive system, measured as
"uplift." Here's a simple example:

On a web web page, we show
100 SMILE-recommended ads that get 15 clicks, for a 15% CTR.

On the same page, we show 100 default ads (same size, same
location, just not personally targeted), and they get 10 clicks, for a
10% CTR.

The SMILE Uplift in this case is 50% ((15-10)/10 \* 100).

We carefully tracked SMILE Uplift for the last five months, and we saw an average uplift of 58.3%.
As we serve millions of ad impressions, that translates into 1000's of
additional clicks generated by our PA system. The ads often point to
downloads or white paper offers that customers sign in to get, and thus
we collect 1000's more contacts and what they're interested in, which we can then (hopefully) turn into qualified
leads and ultimately new customers. So we can see a definite ROI for this
effort. And keep in mind this was "version 1" of the analytics, which
we're continuously refining, enhancing, and developing -- we expect
ongoing improvements in future results.

Actual weekly Uplift gyrated pretty wildly -- here's a summary chart:

You can see general improvement over time as we improved the
algorithms, steadying for the most part in the 40-80% range. In the
last week, we released SMILE ads on the Sun Download Center,
which had
(as you can see) an interesting impact on Uplift! SDLC gets a huge
volume of visitors, and most users are there to download and nothing
else. We also found a large proportion of users there for whom we did
not have recommendations (either because they were new or
they'd deleted their Sun cookies).
The result was a pretty big dip in CTR for the default ads, while we
held steady on the recommended ads, thus the skyrocketing Uplift score
the last week.

With the release of SMILE 2.0, we're completely changing how we do our measurements (it's a long story),
so we'll be tweaking our weekly measurement system and reporting. We'll
have new functionality and new measuring capabilities, and I'm looking
forward to seeing the results from our newest release.

As I hope these numbers
portray, we've demonstrated solid benefit to our emerging PA
technology. It's a great start, but there's still a lot of upside
potential remaining -- we're optimistic of delivering even more
dramatic results in the future.

About

I helped design, build, and manage download systems at Sun for many years. Recently I've focused on web eMarketing systems. Occasionally, I write about other interests, such as holography and jazz guitar. Follow me on Twitter: http://twitter.com/garyzel