A/B testing for headlines: Now available for WordPress

Audience data is the new currency in journalism. I don’t just mean the traditional Costco buy-in-bulk kind — “our readers are 52 percent male, 46 percent over $75,000 household income, 14 percent under age 35,” and so on. I mean data that looks at how individual readers interact with individual pieces of content. And beyond that shift there’s also the move from observational data — watching what your audience does — to experimental data, testing various ways of presenting or structuring content to see what works and what doesn’t.

Not sure if you want a straight, newsy headline or something with a little more pizzazz? Something keyword-dense and SEO friendly or something more feature-y? This plugin lets you write two headlines for each post and have them presented at random to readers. The plugin records how often each version of the headline has been clicked and, once it has enough data, swaps full-time to the most effective one.

If you’re in the kind of operation that has regular debates over headline strategy, here’s a great way to test it. (Although note that this is measuring clicks on articles within your site — it doesn’t tell you anything about the SEO effectiveness of a headline. You’d have to wait for Google data for that.)

We have lots of debates over the appropriate role of audience metrics in journalism. But personally, I’d rather have those debates armed with as much data as possible. If you want your site to be filled with puns and plays on words instead of SEO-friendly nouns, fine — but it’s worth knowing how much of a traffic impact that decision has when you make it.

I’m happy to say we apparently played a small role in its creation: Halliburton writes that he was inspired by an old Lab post that described how The Huffington Post uses A/B split testing on some of its headlines:

Readers are randomly shown one of two headlines for the same story. After five minutes, which is enough time for such a high-traffic site, the version with the most clicks becomes the wood that everyone sees.

Give it a try — and if you’re a PHP coder, try to make it better, as patches are welcome. (Another, more ambitious A/B testing project for WordPress, ShrimpTest, is also in development and in preview release.)

Halliburton (who runs Cogmap and Deconstruct Media) and Bessman (who’s an engineer at marketing firm R2integrated) built the plugin in as 2010 a way as possible: at last weekend’s Baltimore Hackathon, where the plugin won a prize for best prototype. Have a good idea, bang out code in a weekend, share it with a potential audience of millions using the same platform — that’s the promise of open source and collaboration in a nutshell.

Josh, I think this is a start and thank you for this update. I’d like to note that we are still looking at individual readers interacting with content. One step away from the aggregate mass… I think for audience metrics to be meaningful, we need to look at how content percolates through a news ecology – how it is spread socially, among friends, around social media, on shared google readers, etc. As web-metric Dana Chinn mentioned yesterday in the twitterverse, the question is how do you measure that. Not sure. Is Facebook the next google news? Maybe, in a certain way.

We have been experimenting with Post Rank. They attempt to track the interaction with a specific url across social networks so for instance, you see Tweets that included the url, instead of just keywords or mentions, like you would get with a tool like Tweetdeck or Co-Tweet.

How does this work with RSS feeds of the article? Will the RSS also randomly pull each of the headlines or will it key only off of one of them?

http://www.niemanlab.org/ Joshua Benton

Stephen, I’m not 100% sure, but I believe the plugin asks you declare a headline and an alternate headline. I’d suspect that the first one is the one that goes out via RSS, initially. If the alternate one gets declared the better one, I imagine it’d swap the two and make that one the RSS headline too.

Every story could have two lede paragraphs, and you could test those. Or you could generate feedback to editorial by starting with only publishing the lede, and then incrementally publishing additional paragraphs based on the number of people who clicked through to the “more” button. Every story is breaking news!

Love this idea but I can only imagine the winner every time would be the headline that got pushed out over Twitter. And you can’t push more than one without annoying people. It feels like the world online is more complicated than this allows for. Would love to see it work, but am skeptical.

Marshall, I think it’s smarter than you think. Two different headlines don’t equal two different URLs — one tweet would lead to a URL that would display each of the headlines to roughly half the visitors.

Also, as I understand it, a link from Twitter wouldn’t impact the data-gathering process at all, since it’s actually measure the number of internal clicks. In other words, it’s testing how many times saw one of the headlines (say, on a site’s home page) and then clicked on it. An external link wouldn’t count toward one headline’s success, since what was be being clicked on was probably a tweet or a blog summary or whatever.

Thank you for you comprehensive list of plugins for A/B testing for wordpress.
As much as Shrimp is powerful is also not easy to use.
The headline test is amazing, but I’ve only wished I could test more stuff in the text, so performing a multivariate test.

Oliver Paton

How does statistical significance factor into this? Surely you would have to wait for a long time to achieve stat signigficance??

If you’re lucky enough to have the right deep-pocketed owner buy your paper and steady it, you’ve won the lottery. If you’re in a town whose paper is owned by the better chains, or committed local ownership, your loss will probably be mitigated. Otherwise, you’re out of luck.