ALWAYS SPLIT TEST

Update 5/17/11: Internet Marketer Ian Fernando launched a product called Split Pistons based on this case study! There is an interview with me and Ian as part of the marketing behind the product, asking further questions about this case study!

Split testing is extremely important. I’m going to explain to you why it’s so important, using a real life example with a homeless man.

I’m going to show you that there’s always a way to be testing something else, never settle for the industry norm, no matter what you are doing.

Realistically this is BARELY split testing, this is only scratching the surface as to how far you should go in your efforts. You could be testing many many more variations on top of the single change I’m showing you in the below examples. But, the article is to get the point across that you should be doing it in the first place, rather than going in depth about it with my volunteer, Kevin.

So, let’s go through an example of an experiment that I did with Keith, a homeless man. First, realize this… This experiment improved this mans earnings by over 100% over several days thanks to me and on top of that I paid him to let me take pictures (which he was very happy about being featured in a blog article online, especially after I explained exactly what I was doing). I truly believe it was not so much about the actual sign in this case and people would have been donating anyways if he REALLY got their attention in some other way (obviously the cardboard wasn’t working) because he was a really cool guy with a great story once I actually talked to him. Nonetheless, on to the story…

Boring. The rate of people giving him money is low. Time for a split test.

As you can see in the first picture, he was using a very standard “I’m homeless, help me” sign with no major call to action. Just bullets of what is wrong with him. The biggest call to action on this sign is “Hi, I’m Keith” which is not really a call to action, and isn’t anything that would really motivate many people to come over on. So, that’s the biggest flaws in his banner.

The next problem was that his cup (goal) of what he wanted people to do , was tucked between his sign between that and his leg and was not very inviting. This would be like putting an opt-in form or buy button on the footer of a lengthy website tucked in the corner blended with the website colors so you can barely see it, etc. You need to call action to what you want someone to do, not tuck it away in the corner.

So now we move on to a split test. The first thing we always do is make slight modifications to what we already have. In this case it wasn’t practical to change the copy on his banner, so we decided to at least make the goal of what he wanted people to do at least a little more appealing, so we asked him to pick up the cup and try that for a while. This didn’t help his conversions much at all.

Still boring, but this time he’s at least holding up the cup…

Next, we’re going to do an actual split test by comparing the results of another choice with the results and information that we already have about how well his current setup works (not very well).

Introduced a higher converting banner w/ bribe, working much better.

What we did here was quite different than what most homeless people would do. We focused on a different angle. We already have the “I’m homeless, help me” stigma attached to people that are sitting on the side of the street with a cup, so we don’t necessary need to make that a prominent part of our banner. The next big difference is that we changed colors and went from cardboard to white to spark the interest of people walking by instead of automatically having negative associations that they have with cardboard and homeless people. We want this to look like a new age homeless man who is really trying to make it work for himself. The biggest difference, is that we are now introducing a bribe. We are basically saying “hey I’m homeless, help me, donate to help feed my family and pay my medical bills… but not only that, if you donate right now I’ll give you a free squirt of hand sanitizer”. This can be applied to list building and all sorts of other aspects of Internet Marketing almost as is (don’t try to bribe people to complete offers though, just bribe them to get on your email list). One solid strategy of building a list is that you get people looking at your opt-in form then give them something for free aka a bribe to get them to pull the trigger and fill out the opt-in form for your email list.

Now he’s smiling, interacting and building relationships with potential customers, even more conversions.

By the way, this experiment wasn’t just for shits and giggles, the homeless man started to draw a crowd VERY quickly on this busy street and people were genuinely intrigued by the fact that a homeless man was giving away free hand sanitizer for donations to help his family/medical bills. I talked to a few people walking by on the street and asked them what was going on (playing stupid) and they were saying how there was this “genius” homeless man who they were happy to give money to because he was showing that he must be sober and really trying to make things happen based on his cool concept. In fact, this may have even have made the local news.

Moral of the story, always be split testing and trying different minor tweaks on your landing pages to try to increase conversions. Things as little as colors can play a major factor. I wrote in black as if a homeless man wrote the sign) then I added an “out of place” kind of irrational element to the equation by using a red cup. In affect, what it did was create a buzz of the people walking by and got them talking about it and debating where he got the cup and if it was a joke or if he was a genius.

The better call to action (interesting) banner with a bribe worked best!

P.S. This BARELY touches the surface of how far you can go with your split tests. The next avenue to explore would to be see how many different variations of the 2nd sign we can do (with the copy moved around and the product switched/moved around etc – there would be many more tests from that point).

P.S.S. This is a visual example from real life that’s meant to show you, hopefully you weren’t offended by it. If you were then you got the wrong message out of this. The homeless man, Keith, was proud and happy to do it. I may bring him in for an interview and do a video on him. He really has an amazing life story. I’d like to feature him in something that can potentially help others avoid his unfortunate situation and put some more money in his pocket to get back on track with his life.

If you enjoyed this, you’ll love our newsletter.

Secrets So Exclusive We Don’t Even Publish Them On Our Blog.They are totally free!

Thinking outside the box! Loving it! Not only does this example show split testing, but also shows how simple creativity can take you from fitting in to standing out. Whether that’s for a bum or a website, doing it differently is going to draw that crowd.

Overall revenue/ROI was over 100% better with the white than the cardboard.

Conversion rate itself is a bit hard to gauge because we would have to count every person that walked by vs how many of those people stopped and at this extremely busy location it would be next to impossible to count that many people. The point of the article is still there even without the actual conversion rate for the homeless example.

A homeless guy in Philadelphia had a sign that said:
“Lost? Need directions? I can help.” and in small print, below “donations gladly accepted”

I asked him where to find a pharmacy and a particular office building. Sure enough, he knew without hesitation, even drawing me a map. Saved me a LOT of wandering in near-blizard conditions around so I gave him $10.

This is great applied design – way more interesting than yet another endlessly ethereal article about theory.

However, this does show the trap of A vs B testing. The A design had some very basic problems (hard to read, too much text, etc.) – any design that fixed those would have done much better. And simply because B fixes these, some odd and likely negative things come along for the ride.

For example, I doubt most people would want hand sanitizer from a homeless person compared to less personal things (a blessing/fortune telling, a toy, a newspaper, etc). It’s an odd-match – but you don’t get data on how odd these things are as you’re comparing ALL of B vs. ALL of A, in which case you could offer almost anything in a bribe and it would score better than A.

Scott- I’m on the same page as you Scott. There is tons of additional split testing to do if we really wanted to keep on increasing his conversions, this is not a solution but more-so showing how to begin the process for Internet Marketers, I didn’t want to go too in depth with the homeless example and have thousands of people on the street with fancy signs trying to take advantage of it 😉

Michelle- I actually explained to the homeless man (Keith) exactly what I was doing and even told him what I was thinking of using for a title. He was all for it and said that he hoped people could learn from this. It was a totally amazing experience and although I used the subject to drum up some buzz if you read the entire post I’m very sincere about not mocking or making fun of Keith, the homeless man.

Brian, I did read the whole thing. You were very humble and gracious, and obviously cognizant of his position. Your kind attitude is what made it so gutsy — I think someone who didn’t care about Kevin’s position, but only about their own interests, wouldn’t need courage to approach him.

Great post idea. I often think about real world split testing when I’m out and about. But it never occurred to me when looking a the signs homeless guys hold up. But you made one glaring error: you didn’t give the guy a clipboard so he could build his list. Dude, the money is in the list!

I actually talked to him for a good hour after we were all done and he was up for doing some more testing. But, I told him that it wasn’t about getting too deep that I just wanted to get the main point across. Marketers would be able to take that and learn from there how to continue the process and branch it out within their niche. I’m thinking about doing a video with him to get his story out there and hopefully help him.

Always thought the most rewarding pieces of work are those that help others and you managed to do this using split testing – awesome! Great job Brian.

Completely agree with “never settle for the industry norm”. If anything, most successors within industries have shown us that by being creative and engaging in a helpful way, helps you to step ahead of the competition. In this case Keith with the creative white board stepped ahead of Keith with the cardboard.

Keith, if you ever read this, thanks for contributing to the community and good luck for the future!

I like this, but it really isn’t about split testing. What you did was so obviously better that really, there was nothing to test. It’s more about thinking through what you’re trying to achieve, giving it your best shot, and keeping on getting better.

A true split test would be: “is a blue or red cup better?” or “is it better to have free sanitizer or charge a fixed fee?” — Questions with non obvious answers that NEED to be tested.

Don’t know if it would help in this case, but a trick I learned from my musician cousin who played his guitar for donations, regarding the donation jar or equivalent, was to have money in there to start with.

Seems no one wants to be the first to place money in the container when it was empty. I think he used a small glass fishbowl so potential donators could see money in there first and he got more donations at the end of the day when he had a few dollars already in the fishbowl. For some psychological reason, potential donators were resistant when no one before them had donated, but were willing to follow the lead of a previous donator when they saw money and make a donation also.

I think that would be worth splt-testing for Keith and others to try.
Hope this helps.

Love it. And this is beside the point but driving me crazy: the hand sanitizer and cup should be at the top for easier access and less proximity to gross city sidewalk. I know, I know, you would have done further testing… I just can’t get past the idea of bending down to get at the hand sanitizer.

@Erin – Haha, yah. I agree with you. Actually, it was just on the ground for the pictures mostly, he was walking around with it and holding it up when people came over. I wanted to make the point that nothing should ever be considered perfect and you should always do tests on top of tests hence the randomness on the white one. If the banner was a website that you couldn’t hold up in the air (as he was able to with the banner), then the button/opt-in form would need to be above the fold/higher up in most cases.

When the student is ready the teacher will appear. You have given us
a powerful example of split testing, just what I needed for my next challenge. Whilst I was doing split testing at craft-markets with our step stool cows, it did not quite sink in that the same can be done on line. Thank-you for jogging my brain.

I would very much like to hear Keith’s story. Please do bring him in for an interview and post it. I have been self-motivated to help the homeless for years and nearly became homeless myself after a long-term illness.

I am also encouraged by your blog as I didn’t know that I have intuitively done split testing in order to run various events and organisations with better success than others doing the same thing. I’m encouraged to see somebody put into words the ‘scientific side’ of making something more appealling plus the organic side (in this case Keith’s interaction with the public). I have always found that the effort of genuine personal interaction has paid off.

i think the biggest change for this man is changing from a passive sitting and waiting for donations man to a man who is acting, looking forward and a person with takes a little business. i think the most people donating this point, that he start acting and willing to make the best of his situation.

best wishes for keith and much more ideas for him

greetings from germany

PS:
the conversion rate can also be calculated in donations per hour/day instead of donations per walk by persons

This is awesome.
You have done an outstanding job of teaching a man to fish…
and helping others by the example. What a concept!
If Keith put up his story online, his cup would be overflowing.
Awesome, just awesome.
I am putting a link on my blog…

Excellent post just love it and it shows how really important is split testing.
However… did you test the source traffic (:
did you tried the boring sign in near the wallstreart “traffic” v.s. the bus station “traffic” ?
conversion rate alone doesn’t tell the whole story.
you may get last people donating in wallstreat but more money…

@justaguy – Lol, ya there are many more possible ways and variables to test this out from here. Many, many, many. This is only the beginning in what you guys should be doing in your online testing. This is only step 1.

This is the most creative split test that I have seen.
Everyone wants something for a donation, even if it’s just a good feeling but hey..you’re in the city and your helping out the needy and keeping clean at the same time.
I’m sure people will go out of their way to toss some money in his cup.
Great job!

DUDe! You had me at homeless! I just read the post by following a vaynermedia link and am now a fan of whoever you are! I read perry marshals definitive guide to adwords a few years ago and really grasped the concept of split testing but the way you demonstrated your skills in real life…kudos bro!

There was a homeless guy who hung out near the Vanderbilt Campus in the late 90s. He would solicit drunk students for cash in exchange for a verse of rap about whatever you requested(He had even less lyrical talent then shame). He understood how to exploit race and socioeconomic anxiety better then any marketing consultant I’ve ever met. I gave him at least $200.00 over 2 years.

Great job Brian! i like the way you and Keith worked together to pull this off. The real life example of split testing is just Incredible! And I just could not help to notice how two people (Brian and Keith) came together to create such a buzz!

No one knows how far Keith will go from here. You have taught us all a very valuable lesson and how it could be applied to our business as well as our life!

Thanks Brian.

Keith, if you are reading this. Thanks! It’s people like you and Brian who makes a difference.

Brilliant. This is making me want to makeover all the homeless signs I see in the subway! Humor works!
I don’t know why but there are more “creative” homeless signs in San Francisco. Here are some I saw:
*Help! Family kidnapped by ninjas, need $$ for karate lessons.
*Can you help me with a deposit on a cheeseburger?

and my favorite:
*Bet you can’t hit me with a quarter! (seen at an intersection)

I’m curious to know why the original call to action wasn’t changed if the new board was going to brought in anyway.

Reminds me of the classic ‘What is Advertising’ story of the blind man who, much like your guy, had a banner around his neck saying, ‘BLIND’ and wasn’t getting any where until some kind advertising soul, changed the message to, “It’s spring and I’m blind!”. Guess what happened?

I am curious as to the ROI (while less important in a lot of A/B testing, it seems more relevant here):
When you say 100% improvement in ROI, is that actually “100% improvement in revenue”, or did you factor in the cost of the white paper (cardboard is easily found, but white paper of that size probably isn’t), cup (could be more readily found than then paper, admittedly), duct tape, and sanitizer?

The new sign may draw in more money, but it may be less effective from an ROI POV if the cost (amortized over all the uses) was too high, in which case it wouldn’t be a success.

This is all quibbling, I know, and mostly beside the well made point you were getting at, but we need to be careful about optimizing conversions or revenue and not net dollars (if that’s the goal).

Could you post the before and after dollars? I’d love to play with them 😉

I’m retarded. Okay…I am not retarded (no offense retarded folk), but I have always been SCARED of split testing. Not because I don’t want to succeed, but because I am afraid of screwing something up. In all actuality, by not split testing, I am screwing up a TON more than if I were to be split testing.

Thanks for sharing such a simple but obvious example. Sometimes, we all need something to relate better too and this is just that.

Oh, wow, Brian – This was just amazing! First of all, I’m a very visual learner, so this example explained split-testing in a way I could really understand. Thank you so much for that!

Also, I am truly moved at how you worked with Keith – I would love to see his story unfold. Undoubtedly, you were a blessing not only to him, and not only to those of us trying to learn marketing techniques, but also to the many people who passed by on the street that day. Who knows all the good you’ve done!

Fantastic, love it. Reminds me of the guy begging at the train station wearing different styles of clothes – made the most money dressed as a successful businessman claiming top have forgotten his wallet and asking for his trainfare.

Really a nice post, I love the vivid imagery of the stories . It’s important that they know who you really are beyond a name and address and job.” is what really captures the real meaning of a great personal blog

What a great post, I even had a little tear in my eye, the guy is so happy now you may have changed his life just by caring. Okay back to the reason the article was written this really shows an out of the box type message and evolkes the mind to think differently split testing is the only way to go. Thanks for posting

Hey! awesome blog! I occur to be a every day visitor to your website (somewhat much more like addict) of this web site. Just wanted to say I appreciate your blogs and am searching forward for much more!

i love this blogpost! Is that ok, to turn it into german and post it on my blog (http://superblogger.de)? I’am sure that my blogreaders love this article, too and willing to visit your blog (sorry my terribly english ;-)!

I loved this article! I’m just starting to learn about this technique and I was researching a bit more on the internet when I found your post. It’s truly inspiring and it makes me want to try these kinds of experiments, to think outside the box.
Thank you for writing this!

This kind of experimenting is something that can be taken to incredible extremes, as in the case of doing A-B testing to see if a homeless guy can generate more revenue by refining his sign design. I’m not sure if that’s even a true story,

Love this post (and blog), very creative and “outside the boxish” way of demostrating the value and importance of split testing, something which seems to be undervalued by a lot of marketers out there. At the end of the day nothing beats hard cold data.

nIcely, lately this article marketing is rolling out as being a huge business and it has been receiving sufficient attention at the same time ! consequently, what really is the cliche in writing an awesome …

Trackbacks

[…] Posted on March 28, 2010Filed Under Marketing & Branding | Leave a Comment A little creativity can take you a long way, whether your a bum begging for money, or a world class internet marketer. Our friends over at DailyConversions.com shows this ina GREAT, creative way. You have to check this one out! […]

[…] employed by corporations that are for-profit, using a cause as a method of communicating. This marketing blogger decided to illustrate the importance of split testing a campaign (A/B testing). To begin with, the […]

[…] Testing Marketing Principles Using the Homeless – This makes me feel sick, frankly. I am sure that the author would suggest that I was missing the point, but it’s just a bit crass as far as I can see. […]

[…] One area I believe can easily be overlooked online is the issue of getting the reader’s attention using copy, mostly in headlines. A lot of websites do this with imagery and flashy fonts which is fantastic at catching the eye but work mostly only on a superficial level. If you can combine these with some powerful words then you’re putting yourself into a great position to increase your conversion rate. To see a great real world example of how you can test headline copy which is very applicable online, check this out. […]

[…] Always Split Test – The benefits of split testing have been known for some time, but can the same concepts apply to real world situations? One man aimed to find out, and helped a local homeless man increase his profits by more than 100% through the use of split testing. Read the story to find out how. […]

[…] Well, actually, this tip is not from a homeless guy. It's from a marketing guy named Brian. But Brian demonstrated his technique with help from a homeless guy named Keith, and it's pretty instructive. […]

[…] Split Testing – This is testing 2 or more different things at one time. You could test 2 or more different Landing Pages with the same Traffic or you could test 2 or more different Offers with the same traffic. This will give you data regarding what is working and what isnt by comparing the results and differences in the split test variations. For example, take a look at this blog post: Always Split Test […]

[…] prove the idea that effectively targeting your message to your audience is critical, this marketing blogger decided to illustrate the importance of a split test campaign (A/B testing) using an experiment […]

[…] is the strategy I used that led to 50,000 unique visitors and 450 retweets of my Homeless Split Test within 5 days. The funny part about that one was that I didn't even think this process to myself […]

[…] prove the idea that effectively targeting your message to your audience is critical, this marketing blogger decided to illustrate the importance of a split test campaign (A/B testing) using an experiment […]

[…] This kind of experimenting is something that can be taken to incredible extremes, as in the case of doing A-B testing to see if a homeless guy can generate more revenue by refining his sign design. I’m not sure if that’s even a true story, but all of these stories reminded me that […]

[…] gonna finish this post off with another great example of how this beggar increased his revenue by over 100% by testing two types of signs (this is not the same guy from the video, it’s actually a very […]

[…] improvements).Know any tests out there that we’ve missed? Drop us a line in the comments.Real Life Split Test with a Homeless ManBaseCamp Homepage Redesign – 14% ImprovementHighrise Signup Page – 30% ImprovementSee […]