Notenboom also wanted to reduce their reliance on Google for traffic to his website for techies looking for answers. Increasing their newsletter subscribers seemed to be the only clear-cut way of doing it without breaking the bank.

Their audience was fickle by nature, however. Viewers typically came to the site to get a specific how-to nugget of information and then left abruptly. Plus, techies are among the touchiest demographic to market to online.

Many of Notenboom’s peers understand. MarketingSherpa’s Email Marketing Benchmark Guide 2008 says that 11% of emailers state the willingness of people to join new lists is a significant concern. So, how could they get visitors to spend extra time to sign up for their newsletter?

CAMPAIGNNotenboom started first by investigating alternatives and looking at analytics data. They discovered that the average time visitors spent on their site was 66 seconds.

They decided to test an opt-in process that used a dhtml (dynamic html) time-delay *hover box,* which mimics the actions of a pop-up. Unlike a pop-up, a hover box – sometimes called a ‘slide-in’ – doesn’t get produced by another window being opened. Rather, the hover box code is part of the actual Web page being viewed; it remains hidden for an amount of time to avert pop-up blockers.

“Personally, I had always been turned off by anything resembling a pop-up,” Notenboom says. “But I thought the combination of a [hover box] and a delay was a potential big win.”

A key part of his plan was the delay: not serving viewers with the offer right away but after they took some time to explore the site. But how long should they delay it? They followed three steps to get the answer:

-> Step #1. Set up A/B test

Notenboom and his team knew readers normally stuck around for about a minute before leaving the site. So, they homed in on a time-delay at or near the 60-second mark with A/B testing. They tested three combinations three days apiece -- “long enough for the results to become statistically significant.”

They wanted the time delay to be the only test variable. But they still had to design a pleasing hover box for the test.

Here’s what Notenboom and his team came up with: o Box size of 35,781 byteso Two-tone blue background color o Two paragraphs of copy in white type

Underneath the copy, viewers were encouraged to sign up by inputting the following information into entry fields:o Nameo Email addresso “Where you heard about Ask Leo!” (optional)

-> Step #3. Set restriction

They set a restriction on the hover box. Regular readers who didn’t subscribe would see it only every six months at that IP address. The exceptions were users who changed computers or cleared their cookies.

“I didn’t want to annoy my readers. But I wanted to give them a reasonable chance to opt in,” Notenboom says.

RESULTSThe test demonstrated that the time-delay hover box works, and it works best at 60 seconds. It has been capturing a 1000% average daily increase in subscriptions.

“In all of these cases, the difference was small but detectable,” Notenboom says. “For me, what I have learned is that there is a sweet spot for my site at 60 seconds. It surprised me that 60 seconds was better than 45. I expected more people would leave if I hit them later.”

The 30-second delay performed as well as the 60-second one, but the latter was a better fit for his audience. “What I think is happening is that the people who are hanging around and reading more are increasing their perceived value of my site. And then, at the point when the hover box appears, they’re deciding that my newsletter is something worth signing up for.”

Did any of the data surprise him? The pop-up-like annoyance factor that Notenboom was a bit wary of at the beginning has not popped up. “We’ve had no real negative feedback. The hover box has given me the best of both worlds.”

He says the 60-second delay for the hover box shouldn’t be counted on as a magic number, however. “It’s specific to my site, but what the results do show is that the time-delays are definitely worth testing.”

Comments about this Case Study

Jul 04, 2008 -
Jon of
Ninja Blog Setup says:
I think if he removed the question "Where did you hear about us" he'd see a further increase in %. Even though it's optional, it is clutter.
Also, it would have been nice to have a link to a reference on how to create the delay popunders, or can you do that directly from Aweber?

Post a Comment

Note: Comments are lightly moderated. We post all comments without editing as
long as they (a) relate to the topic at hand, (b)
do not contain offensive content, and (c) are not overt sales
pitches for your company's own products/services.

The views and opinions expressed in the articles of this website are strictly those of the author and do not necessarily reflect in any way the views of MarketingSherpa, its affiliates, or its employees.