If you work on a site that is funded by advertising, then you can guarantee that every once in a while someone will have the great idea to put a pop-under advertisement on it – there are sales targets to meet don’t you know! Us folk who actually use the web know this is a bad idea as it only serves to irritate the user and detract them from what they’re doing.

I had always suspected that pop-unders would have a negative effect on the metrics of a website, but I struggled to find any hard evidence that backed this up. The vast majority of negative words about popups and pop-unders tended to be Nielson esque usability pieces or very subjective “I think” rants, and thus not great for building a convincing case against them.

Some months ago at one of my previous employers, I decided to run a test in agreement with the business, that aimed to truly understand the effect of pop-unders on site metrics, and in turn, the bottom line. With that agreement I teamed up with the resident web analyst and performed this little experiment.

The first thing we did was segment our user base into four groups:

A – No popunder served

B – 1 popunder every day

C – 1 popunder every 3 days

D – 1 popunder every 7 days

We then got the ad agency to generate the javascript code for each of the 3 pop-under serving frequencies. Equipped with this code we then had to segment our users into 4 groups. We did this by targeting the last digit of an incremental six digit number that is our logged in users member ID, and then put them in a group that would be served the targeted advert code:

Last Digit

Group

0, 1 or 2

B – 1 pop-under a day

3 or 4

C – 1 pop-under every 3 days

5, 6 or 7

D – 1 pop-under every 7 days

8 or 9

A – No pop-under

We then put the code live, and waited. We had planned to leave the code live for one month, but after about 8 days we thought we’d take a quick look at the results to see how they were progressing. At just over a week into the experiment it became apparent that we already had enough information to build a convincing argument against pop-unders.

The below table looks at customer behaviour and the associated revenue (I’ve twiddled the real figures, but kept the ratios) we got from our pop-under trial. It shows how much revenue was generated from logged in members based upon both pop-unders and ad space.

segment

uniques

visits

impressions

repeat visits

visits per impression

pop under revenue

ad revenue

total

C – 1 every 3 days

7188

23165

486323

3.22

20.99

£0.05

£1.62

1.67

no pop-under

6102

20365

434502

3.34

21.34

–

£1.71

1.71

1 a day

7833

21228

407515

2.71

19.20

£0.11

1.25

1.36

1 every 7 days

3770

12959

288144

3.44

22.24

£0.02

£1.83

1.85

We can see that members who are served one pop-under per day repeat visit 0.6 times fewer than the control group and see 2 pages fewer per visit. Aggregating this out to a revenue figure (assumptions below), means that so far in the trial, those members who have seen one pop-under a day drive 40p per unique less in revenue than the control group. Therefore in order to make pop-unders viable for the business, they would need to pay circa £50CPM for a 1 a day serving frequency.

The group that was served 1 pop-under every 3 days also delivered slightly reduced revenue (-4p), but the 1 a week segment outperformed the control group. Had the trial not been canned after little over a week, it would have been interesting to see how these two groups would have performed over a longer period. I did try and encourage the business to see the experiment through, but suddenly they weren’t interested in pop-unders anymore…

Related Posts

Who is daipratt?

daipratt.co.uk is the online home of David Pratt.

What do I do?

Broadly speaking, I specialise in creating websites. Not the design, but the stuff that makes it work, web application development some might call it. Expect to read about JavaScript & jQuery, new web technology, innovative front-end solutions, SEO, Drupal & PHP, and the occasional post about where I feign knowledge in marketing and analytics.

If you'd like to talk to me about a project you can either email me or say hello over at my companies website.