One of the things that make A/B testing terribly exciting is its ability to throw up results that go against popular perceptions. Data can sometimes prove wrong all your instincts, gut, intuition, experience or whatever you rely on to make strategic decisions. This case study is the perfect example of such an A/B test.

Company

BrookdaleLiving.com offers various community living solutions for senior citizens. It offers services to those planning on living independently to those needing caregivers or nursing help.

The Problem

BrookdaleLiving’s ‘Find a Community’ page was incredibly underutilized with just a bare-bone structure. It was very stark with no graphics, testimonials or content to encourage the visitor to convert. This is how the page looked:

A/B test

Brookdale hired digital marketing and analytics firm Fathom to optimize its ‘Find a Community’ page. Matt Fieldman, Senior Account Executive at Fathom, completely redesigned the page by adding content, testimonials, credibility logos, USPs, among other elements. He used Visual Website Optimizer to create two versions of the page.

Variation 1

The first variation had a photo of an elderly woman. This is how the photo version looked.

Variation 2

The second variation had a 1 minute 56 seconds heart-tugging video in which many elderly people talk about their positive experiences with Brookdale. In the second version, the video replaced the woman’s photo. Everything else on the page remained the same. Matt expected this version to perform better than Variation 1.

This is how the video version looked (And here’s the video. It will auto-play in a new tab)

Test results

The image version outperformed the video version. When pitted against the original, the image version recorded a 3.92% increase in ‘Find a Community’ searches while the video version recorded just a 0.85% increase in searches. The image version also achieved the statistical confidence of 99.99%. The test was run for 2 months on over 30,000 visitors.

According to Matt, the seemingly modest 3.92% increase in community searches will result in a more-than modest $106,000 additional monthly revenue.

Why the variations worked?

Matt used Fathom’s most-successful landing page template while redesigning the ‘Find a Community’ page. He made sure to speak to all the four major online personas by adding relevant elements for each of the types.

These are the four kinds of online personas Fathom targeted:

1) Methodical consumer

This user is detail oriented and tries to read up every bit of information present on the page. Matt used additional explanatory content to speak to this category.

2) Spontaneous consumer

This type is quick with the buying decision or already has the intent of purchase while going to a website. Matt used a clear and concise search box with an arrow pointing towards the ‘City’ field to speak to this customer type. He also cuts down on the number of fields — a practice that has increased conversions for many Visual Website Optimizer customers.

3) Humanitarian consumer

This type wants to know who you are and whether they can place their trust in you. Matt uses two tried and tested elements to ward off customers’ fears — testimonials and logos. WikiJob, a UK-based VWO client, tested a customer testimonial on its product page and saw 34% increase in conversions. Similarly, another company increased conversions by over 72% by adding a trust badge.

4) Competitive consumer

As the name suggests, this type wants to know why they should choose you over the competitors. Bullet points calling out Brookdale’s unique selling points (USPs) were put up on the page to get their attention. A VWO client increased CTA clickthroughs by almost 50% by changing its CTA text to a better value proposition.

Why the image version worked better than the video version?

It will be fair to say that Matt wasn’t the only one surprised by the results. Given an option to choose between a video or a photo, most people have no difficulty opting for the former. Videos are expected to miraculously make your product and website look super sexy. Here’s why videos are hailed:

So it’s no wonder Matt was left quite surprised by the results. “It’s the counter-intuitive nature of the result that was most fascinating,” he said.

The possible reasons why the image version worked better than the video version are:

1) Brookdale is already an established brand and the video acts as a distraction

The first hypothesis is that because Brookdale has been around for over three decades and is also listed on Nasdaq, it’s already a big brand and people don’t need much convincing to convert on the website. Visitors have usually done their research and arrive on the website knowing what they want, but the marketing video was a distraction.

The video has a lot of elderly people talking about specific points such as ‘food’ and the ‘caring nature of the staff’ at the communities. There’s a possibility that the visitors did not have these criteria in mind and only after listening to them thought of ‘food’ and the ‘qualities of staff’ as important factors in deciding on a community. They possibly then went back to read up more on what the competitors are offering.

2) Most users fall into spontaneous customer type

I am not so sure of this hypothesis since I feel community living solutions are seldom spontaneous decisions, but I would love to be corrected if you think I am wrong. Matt himself feels that the customers want to convert quickly and are not interested in the video.

“As you can see, we were wrong. We now understand our customers better: they aren’t interested in marketing videos so much as they want to convert quickly and easily on the page,” he said.

3) Target audience has slower internet connection so videos might not work for them

I used Alexa to find out that Brookdale’s target audience is over-represented by women who didn’t go to college and who browse internet from home.

Now, a report on average home internet speed in the US tells us that people who didn’t go to college are less likely to have high-speed broadband access than those who went to college.

Now by adding the two statements, we can come to a hypothesis that majority of Brookdale’s audience is represented by women who don’t have access to high-speed broadband access. The slower internet speed might be leading to a painful video watching experience, and hence the image version worked better.

Note: The third hypothesis was added to the post following a very valuable point made by Tommy of ConversionXL in the comment section.

Do you have a theory?

Can you think up of any other possible explanation why the non-video version worked better than the video version? If so, please share them in the comments below.

A lover of the written word, I plan to be the planet's first sit-down comedian. When I am not rethinking a misplaced comma, I write about conversion optimization and website usability. You can follow me @mohitanagpal

If interactions with my parents and grandparents are any sort of anecdotal indication,it’s that anything that’s not text or images is “scary” and too much internet. It’s intimidating.

I’d be interested in knowing what the internet connection speeds were for visitors to this page. Browsers? Each of these things could give some insight to who was visiting this page, and why video didn’t perform as well.

As a detail-oriented person, I can’t stand videos. If I can’t find what I’m after in text (which generally contain more detail) on a site, in a very short period of time, I’m outta there. A 2-minute is a massive time suck (and will usually gloss over important info).

Very interesting case study and I have always enjoyed reading the blogs from you.

My thoughts, without going into the video itself, could it be the look / feel of the elderly talent that has an effect?

The talent in the video looks somewhat ‘Granny’ vs the talent in the picture. I believe we are now in the age of ‘active-aging’ and the talent in the picture feels more contemporary and active at her age.

You made a very important point about browsers and internet speed. Though I didn’t have the exact numbers, I have taken some data sources to formulate a third hypothesis and added it in the post. Let me know what you think. I have also asked Matt (who ran the test) to join in the conversation.
P.S. The phrase ‘too much internet’ had me release a self-conscious chuckle 🙂

@Craig
Thanks for leaving the comment. I share your sentiments regarding videos but I guess neither of us are the target audience here.

@Foo Hup Wong
I myself found this case study quite fascinating and I am glad you like our blogs 🙂 Regarding the point you make about active aging, do you have any data for that? I ask this since I personally found the two women to be quite old.

I should have qualified my location when I mentioned about the point of us being in the ‘age of active-aging’.

I am based in Singapore and over at where I am, the topic is gaining traction in the media in the past year – with a recent report new development of housings being built for housing of seniors and having services to cater to their needs. These were modelled after examples in US and Australia.

I am sure I will see data popping soon and perhaps can link it to share if I see.

Competitive Consumer: “this type likes to read up on your competitors and do exhaustive research ”

That’s not the correct definition of the competitive persona type. The methodical persona will do this. The competitive persona type asks questions like, “how does this help me get ahead? How is this the best option for me?”

I love how you went through the different personas to identify how the elements on the page targeted them (though I agree with Chase – I don’t believe the Competitive persona would spend the time to do exhaustive research). Assuming the video was not auto-played (sorry if I missed where you said whether it was or wasn’t), I imagine the video served as a distraction. While all of the elements are targeting specific personas, I think its hard to make an assumption on customer types. I would be interested to see some follow up tests simplifying other elements on the page. While the page looks much more authoritative than it did before adding the confidence builders, there are elements that could possibly serve as distractions.

The conversion point sounds to have been the ‘find a community’ search results page. That makes sense, but it has its caveats.

1. If users land on the page looking for more information, the only place they can go on the ‘image’ page is that community search form. They may not specifically *want* to use it, but it’s the path they’re led down. Alternatively, if they land on the video page, the ‘logical first click’ is the video, which gives them a little more information. Are the users who then use the ‘community search’ form having watched the video therefore better qualified & more likely to convert? I couldn’t see the numbers in the post, but I may have missed them.
2. There’s a phone number on both of these pages. It may be that equal numbers of users called that having landed on either side of the A/B split. Or, it may be that users who are more likely to view videos are also more likely to call. It would be interesting to know!
3. We don’t know whether the video was good. A great video can be fantastic, a bad video can put people off. It’s not necessarily the medium in this test, but perhaps the execution?
4. We don’t know what the call to action was at the end of the video. I suspect a decent call to action & a ‘click to find your community’ button at the end of the video may have worked well.

I don’t know, but lately I tend to find videos
rather annoying. Have you ever landed on a page with almost nothing but a “short” video that goes on for
20 – 30 min. and your just waiting for the speaker to get to the point, and there’s no control at the video. I feel I’m vasting my time.
I prefere text I can scroll through.
Cheers
Gunnar

@Megan
Thanks for leaving such a well-thought comment. You are assuming right, the video was not on auto play. I find the part where you say ‘some elements could be distracting’ very interesting. I feel that ways too. Do you want to may be suggest some follow up tests?

I would definitely test whether the testimonials in addition to the logos are both needed, or if this particular audience had enough trust in one or the other. The contrast the form has with the rest of the page is great, and it stands out very well! I would probably test a primary CTA with less gradient to make it stand out even more.

You make a valid point here but we have to keep in mind that in the case of Brookdale, payments are not made online. VWO can track the data only till the process is online. The only way it can track the actual conversion rate of those who watched the video is if the VWO test and variation data is pushed in Brookdale’s CRM.

2) Those who watched video more likely to call?
I have asked Matt to join in the conversation. Hopefully, he will have some interesting data regarding this.

As a 70-year old, I guess I am part of your target market. No experience with split testing, but lots of experience with old people and retirement villages. Hope you don’t mind my adding my few cents’ worth.

@Craig @Gunnar. I agree entirely. As I have got older, I have lost patience and I very rarely click on a video. They tend to be badly made, boring and time-consuming. Sales videos are often hysterical and never seem to get to the point, if they ever had one. Much prefer text which can give thoughtful detail.
Having said that, the text here gives the impression that anyone over 60 (arbitrary chosen number) is only interested in being sheltered and molly-coddled. Actually, we are not. Maybe down the track a bit, we will be, but we certainly do not want to be reminded of it, whatever age we are.

@Foo Hup Wong. Hear! Hear!

@Tommy. A bit patronising, don’t you think? Scary? Bewildering, maybe? Nowadays, apart from those in their 90’s, perhaps, I would say that computer savvy or lack of it, is no longer necessarily a generational thing, but a consequence of life-style.
(my 89 year old cousin does not ‘do’ Facebook because she find it a bit too much. She sticks to emails, photography and driving to the theatre between trips overseas)

I watched the video and thought its message was suitably comforting. However, for me, it gave the impression of a confirmatory testimonial. Something you would wish to see that would validate a decision already made, perhaps.
Would something shorter and punchier, with fewer participants actually involved in some desirable activity be more relevant as an opening statement?
Make the message anticipatory. A new, active and exciting life, rather than a final solution.
Sorry for the rant!

I agree with a lot of the other comments. The average age of someone entering this website is going to be much higher than an average site visit. There were no CTA’s on the video to CLICK HERE FOR A VIDEO. With older generations that are not as familiar with technology, they need to be told what to do in order for them to actually watch the video. The video thumbnail compared to the picture was not as presentable. It may come off as not as genuine. One test that i did not see done is a site heat map. Did people click on the video or not? It could just be a CTA problem.

One hypothesis behind why image converted better than videos is because of your visitors are old people. They usually have very low attention span and want to move quickly, they are not so tech savvy (Youtube stats could verify that)

I would without question say that it is Age Demographic driven. More importantly I would say that the older generation is not willing to purchase “High Speed Internet” and look at such pages using a dial-up connection. Video and dial-up is not a good combination.

What might be a more interesting test in this scenario, is not so much the overall difference, but specifically if there was an advantage to each depending on the options that were being searched IE “Independent Living”, “Assisted Living” “Dementia Care” and “Skilled Nursing”.

I would venture to say that the “Dementia Care” option, may have benefitted from the Video presentation. My thinking here is that more than likely the visitor with all probability would be in a lower Age Demographic and have an increased data connection. IE the child of the person needing care.