29Apr 12

So what was that poll all about then? (This poll, the one I linked on Twitter and Tumblr – a basic tick-the-box job on the best-selling music acts of last year)

Well, the truth is it was trying something out for my day job. I wanted to try out DIY split-testing tool Optimizely and see how easy it was to run basic experiments.

In this case there were actually two different polls – the one you saw should have been random.* One of them asked the question as follows:

“This is a list of the best-selling acts of 2011. Please tick any acts that you enjoy.”

And the other asked it this way:

“This is a list of the best-selling musicians of 2011. Please tick any artists that you enjoy.”

So the idea** was to see if phrasing the question using more loaded words – “musicians” and “artists” would have any aggregate impact on the results, perhaps making people more careful in their judgement*** or favouring particular acts.

And did it? Well, maybe. As a researcher, I’d only report a result as significant at the 95% confidence level, and none of the differences hit that.**** But two acts showed different voting patterns at the 90% confidence level (which roughly means, if we ran this again, there’d be a 9 in 10 chance of getting the same result, and to be ‘significant’ you need a 19 in 20 chance.)*****

These were Adele and Amy Winehouse. Amy beat Adele in both polls, but in the ‘acts’ poll Adele got 46% and Amy got 54%. Change the terms of reference to ‘artists’ and the gap between them widened enormously: Adele ended up with 35% (11 points lower) and Amy got 64% (12 points higher).

This is interesting enough that I wouldn’t mind running the test again on a wider population with a better designed survey.

Of course the trouble with A/B tests is that it can give you a result but it doesn’t tell you why. I was expecting Amy to do better once you started talking about ‘artists’ – she’s a recently canonised dead musician – but I’m really surprised Adele’s vote dropped (whereas Gaga and Rihanna held up fine – so it’s not a ‘triggering yr latent rockism’ thing). Maybe enjoying ‘acts’ implies less of a commitment, so Adele picked up more ‘she’s OK’ ticks on that poll. I don’t know!

Even 90% confidence intervals were stronger than I expected, though, and the “is this tool usable” element of the test worked fine, so if you voted, thankyou very much for doing so!

*It may be that there were browser issues in some cases, or that some of you have plug-ins which avoid javascript nonsense like the Optimizely code. For whatever reason, a lot more people ended up filling in the “acts” poll than the “artists” one, but I don’t think this was a ‘result’.

**Beyond just trying out Optimizely.

***This is why I REALLY should have put a “none of the above” in!

****Except poor old Ed Sheeran, who got 10 votes in the “acts” poll and NONE AT ALL in the “artists” one – apparently this is significant, but since even his 10 votes only got him a 7% share I don’t think it is really.

*****The whole poll ought to be very bad research anyway, since the sample is opt-in and very skewed (people who follow me on Twitter don’t have the same music tastes as the general population, it’s fair to say.) BUT the great thing about split tests is that this doesn’t matter in terms of examining the split, since the sample is identically dodgy on both tests!

Comments

My actual very first reaction was that I didn’t really like any of these enough to tick them except Gaga, but then I relaxed and ticked based on whether I liked at least three songs by them which lets Rihanna, Amy and even Adele in.

It did make me think more about the current status of solo male artists in general right now, but particularly as successful-yet-credible (according to ‘us’) acts. Even if you polled the top 50 male solo acts based on sales I’d probably find it tough to tick as many as five (and they might all be rappers). So more interested in why this is and what can be done about it.

Not being on Twitter nor able to follow anyone’s tumblr I didn’t see this poll. But I would have thought the words “best selling” would neutralise the rather subtle (IMO) acts/artists distinction you were testing. In any case, just now I zeroed in on the word “enjoy” and thought solely about that in deciding which names I would have ticked (before reading below the cut).

Aren’t you supposed to stick some people’s hands in cold water while they’re filling out the poll, and other people’s in warm water?

Are you thinking of doing this not as a test? If so, are you going to have people access the poll elsewhere (i.e., not anywhere near Tom Ewing’s Tumblr or Twitter)? My hypothesis is that you’d get a stronger result in the population as a whole than among people near to you intellectually and sociologically etc. That is, I’d expect that people like us have long since accepted that someone like Rihanna can be an artist (no matter whether we think she’s good or not), whereas fewer people in the general population would. On the other hand, indie fans who might not accept that someone like Rihanna is an artist are probably closer to “us” than the average Brit or American. Not that all indie fans feel the same way about this. And still on the other hand, people like “us” are more likely to give a different answer on Timbaland c. 2007 as artist versus Timbaland 2007 as act (’cause he was a much better artist than act) – though probably once you put “best-selling” in your description, I’d behave as if what you meant was “act’ even if I’d gotten the “artist” poll.

#4 the problem w/split tests is that you need a stream of people to be coming to the site you’re doing it on. So I’d need somewhere with access to a large “population as a whole” audience, which I don’t have. (Well, I do have, I have access to survey sample, but if I’m using that I might as well do a survey).

I think this is probably the end of split-testing on FT – since we’re not ad-supported I don’t care about “conversion rates” etc. IF ANYONE ELSE LIKES IT IT’S A BONUS, I think is the phrase for it.

26 Apr 1999
The mode of the music changes, the city quakes, or at least those blocks of primer-than-thou office space quake that house the HQs of worldwide record companies. The reason, apparently, is MP3 technology, which you all know about and most of you use, and which has been the subject of acres of ruminative, pessimistic music […]

23 Apr 2000
My first encounter with the exotic was on the Magic Mill at Thorpe Park, a South-East England theme park which from appearances had originally been based around a cramped zoo or city-farm set-up. At some point in the late 70s it had seemingly panicked, though, and parked itself up in order to survive. With its […]

24 Dec 1999
Low’s Christmas tree is a simple one, with eight plain wooden baubles. The band write four tracks themselves, offer interpretations of two carols and crooners’ favourite “Blue Christmas”, and nobody’s bothered to take credit for “Taking Down The Tree” (but it sounds like one of Low’s own). The record is packaged – beautifully – like […]

12 Nov 2014
The Secret History Of Band Aid Everybody remembers Band Aid. And – despite everything – most people remember Band Aid 2. And now we have Band Aid 20 30. Which rather begs the question – why does nobody ever talk about Band Aids 3 to 29? Take a trip down memory lane as we remind […]

8 Feb 2001
Charlemagne Palestine – Four Manifestations on Six Elements 1. Charlemagne Palestine approaches a piano like a climber approaches a mountain. He does not play the instrument so much as he lets it test him: he starts each performance like an ascent, knowing that somewhere ahead there are the limits of the piano, and also the […]

3 Sep 2007
Mars Planets: the atomisation of the Mars Bar. An entropic dis-integration, the tendency of all things to become more chaotic, in confectionery form. I’m trying to resist the impulse to tie this stuff up to no-such-thing-as-Society atomisation because that’s not how we do things, right? And Mars Planets are better to share than a proper big […]

1 Mar 1981
My name is Tanya Headon. I hate music. All of it. And I’m bloody good at what I do. Which is why, when picketing a gig of The Feeling I managed to get over 90% of the fans to agree with me that they are just soft rock bobbins and their new album is just […]