August 11, 2008

Why Copy Testing Is Bogus

"My friend shot a 158," Joe said.“That’s the worst round of golf I’ve ever heard of,” said Mike.“No it’s not. It’s a great round of golf,” said Joe.“You’re crazy,” Mike said. “How can you say that?”“He only has one arm.”

I made up that little story to make a point. The point is about copy testing, and we'll get back to it in a minute.

Copy testing is despised in the ad industry for a number of good reasons. The MBA's hate it because it's unreliable and doesn't often correlate with real-world effectiveness. The creatives hate it because it rarely rewards non-linear thinking.

I hate it for a whole other reason. Having once been a science teacher, I have a healthy regard for the difference between science and bullshit. And copy testing ain't science.

Copy tests almost always fail to include controls. Controls are the one essential of all experimentation. Research without controls is like baseball without an umpire.

A control serves two purposes:

To make sure you’re studying what you think you’re studying.

To give you a factual basis for comparison

A perfect example of providing controls in research is the placebo in medical research. When studying a new drug, scientists always give some of the subjects sugar pills. Sugar pills have no effect. If a drug they are testing seems to provide benefits, but those benefits are no greater than the benefits of taking the sugar pills, they know that the new drug is worthless. The apparent benefits are not the result of the drug, but simply of giving the subject a pill—any pill.

By testing a known quantity (the control/placebo) along with the unknown quantity (the test drug), you have a factual basis for comparison. A scientist who did a study without controls would be laughed out of any lab in the world.

Most copy tests do not use controls. They use norms. It’s important to understand that norms are not controls. Advertising norms do not tell you about ads in your category, at this time, with these people, under these circumstances.

In order to evaluate a new toothpaste campaign, for example, an advertiser needs to test it along with a campaign in the toothpaste category that is known to have done well in the real world. This known campaign should be tested with the same people, at the same time, under the same circumstances as the campaign in question. Only then can you know what success will look like.

Without that, if you get a lousy result, you don’t know why. Is the new campaign lousy? Or was the tester irritating? Or were the M & Ms stale? Or would any ad about toothpaste do poorly at this time, with these people, under these circumstances?

If the control campaign did well at the same time under the same conditions, then you know the new campaign must be lousy. But if the control campaign also tested poorly, then maybe ads about toothpaste only have one arm.

By The Way:One reason direct marketing people think of us as idiots is that when they test they always use controls.

"Caustic Yet Truthful"

"The Most provocative Man In Advertising"

"Savage Critiques Of Digital Hype"

"Fabulously Irreverent"

CONTACT BOB

Over 60,000 people have watched Bob's talk at Advertising week, Europe

You Are Caller Number...

Click Image

Ad Contrarian Says:

"Creative people make the ads. Everyone else makes the arrangements."

"Delusional thinking isn't just acceptable in marketing today -- it's mandatory.""Good ads appeal to us as consumers. Great ads appeal to us as humans."

"Social Media: Tens of millions of disagreeable people looking to make trouble."

"As an ad medium, the web is a much better yellow pages and a much worse television."

"Sometimes success in the advertising business is about sitting quietly and letting clients proceed with their hysterical delusions."

"Marketers prefer precise answers that are wrong to imprecise answers that are right."

"Brand studies last for months, cost hundreds of thousands of dollars, and generally have less impact on business than cleaning the drapes."

"The idea that the same consumer who was frantically clicking her TV remote to escape from advertising was going to merrily click her mouse to interact with it is going to go down as one of the great advertising delusions of all time."

"Nobody really knows what "creativity" is. Every year thousands of people take a pilgrimage to find out. This involves flying to Cannes, snorting cocaine, and having sex with smokers."

"Marketers habitually overestimate the attraction of new things and underestimate the power of traditional consumer behavior."

"We don’t get them to try our product by convincing them to love our brand. We get them to love our brand by convincing them to try our product."

"In American business, there is nothing stupider than the previous generation of management."

"If the message is right, who cares what screen people see it on? If the message is wrong, what difference does it make?"

"The only form of product information on the planet less trustworthy than advertising is the shrill ravings of web maniacs."

"There's no bigger sucker than a gullible marketer convinced he's missing a trend."

"All ad campaigns are branding campaigns. Whether you intend it to be a branding campaign is irrelevant. It will create an impression of your brand regardless of your intent."

"Nobody ever got famous predicting that things would stay pretty much the same."