Subscribe to Marketing Day

9 Questions To Ask When Reading A Digital Marketing Case Study

The digital marketing landscape is moving faster than ever before, with dozens of new tools, platforms, and advertising media coming into existence every year. When making a decision to adopt one tool over another or making an entrance into a new market, the case study oftentimes becomes an important factor in whether the trigger is pulled.

While a proven tool or medium is certainly less risky to adopt compared to one that is completely unproven, it’s worth remembering that case studies are generally crafted to make the object look as good as possible. The results presented may be distortions of reality, and any single study must be taken with a grain of salt.

In this article, I hope to shed some light on how to effectively review a case study — including what questions to ask yourself or others while reading them.

Case Study Result Metrics

Some case studies contain little more than the fact that a particular marketer was happy with the results of using the certain tool or market. Though such case studies help in a manner similar to product reviews on retail websites, marketers are increasingly demanding hard numbers to quantify the potential gain.

The most commonly cited quantitative results are the following:

Lower CPC or CPM (cost per click/mille)

Higher CTR (click-through rate)

Improved conversion rate

Reduction in CPA (cost per action)

Improvement in ROI/ROAS (return on investment/advertising spend)

Increase in amount of conversions, revenue, or other KPI

While all of these are positive results, some are more important than others, and some can be misleading. The questions below will help cut through smoke and mirrors to help determine how applicable the case study is to a marketer’s business goals.

Applicability

1. Does the subject of the case study have a similar business model and KPI as mine?

This is an obvious but important question. A tool or marketplace that works for one industry may not work for another, depending on the target audience and creative format. If the subject of the case study is in a similar vertical to yours with similar business goals, then the relevance of the study will improve.

2. When was the case study written, and is it still applicable now?

In today’s rapidly evolving digital landscape, case studies that are even two or three years old may become outdated, since the tool features, or market user base and competition, may have changed significantly since then. This is particularly applicable to relatively new markets such as social or mobile.

Experimental Design

3. Was this a simultaneous or sequential test?

Simultaneous tests are generally better than sequential, because seasonality or market changes may have affected results during the test period. However, simultaneous tests also have the challenge of selecting campaigns that are sufficiently similar to each other (an example of control factors, detailed in #6).

4. What was the sample size (number of impressions/clicks/conversions)?

If the sample size of the comparison is too small, the results do not mean much. For example, if 2 conversions occurred for one scenario and 3 for the other, this would be a 50% increase in conversions on paper, but it could have easily happened purely by chance alone.

5. How long did the test run for, and what periods were used in the comparison?

Similar to sample size, a test that is too short has greater potential for noise. However, a test that runs for too long introduces potential for various confounding factors that may have influenced test results (especially in a sequential test).

6. What kind of control factors (variables held constant) were used in the test?

For the comparison to be as fair as possible, it is vital for there to be controls put into place. Spend, for example, is one factor that absolutely needs to have been controlled for in the test, since it is easy to get higher efficiency with a lower budget.

7. Was the result statistically significant?

Related to #4, if the case study was based on too little data, a seemingly good result can be obtained just by chance. Statistical significance will put this doubt to rest; if the subject of the case study can respond to this question with a definitive yes, then there is greater reason to believe that the study was grounded on good data and science.

Results Validation

8. How about the results for [insert metric]?

I rarely see A/B tests where all metrics point towards a positive result. For instance, CPA may have improved, but conversion rate may have gone down (which means the CPA improvement was driven by CPC/CPM decrease). These negative or neutral components of the result are sometimes withheld from print. Particularly concerning is when results of higher-funnel metrics such as CPC or CPM are given, while lower-funnel metrics such as CPA and ROI are not.

9. How do I know that the CPC/CPM will not rise in the future?

Since price is influenced by competitive landscape, there is risk that an early-stage market will become increasingly more expensive in the future. Early adoption of new markets may pay off while the market is still fresh, but results may become stagnant with competition increase.

Conclusions And Tips For Marketers

With data-driven marketing becoming the new norm, case studies are also become increasingly quantified. While there are many metrics that can be presented to seemingly show a positive result, only a small subset may actually be applicable to any single marketer.

The above questions should help shed light on how much to value the results of a case study for a tool, platform, or marketplace.

Some opinions expressed in this article may be those of a guest author and not necessarily Marketing Land. Staff authors are listed here.

Attend MarTech and hear first-hand how brands like Coca-Cola, Aetna, Dell, EMC and Netflix are harnessing the power of technology to produce exceptional customer experiences that deliver business results. Visit with over 60 companies in expo hall. Don't miss the only US-based MarTech conference this year. Register today!

About The Author

Kohki Yamaguchi leads product marketing at Origami Logic, a cross-channel marketing intelligence solution for modern marketers. With a career of 8 years in marketing and analytics spanning various functions, Kohki's focus has always been on translating data into strategy, simplifying the complex, and bridging the gap between data and organizational silos.

Sponsored

I don’t see much about marketing. Just advertising and assuming all interactions are equal. Is this a media planner’s guide to the world?

Kohki Yamaguchi

Touche, the metrics I used was from the paid media side. But most of the general conclusions and pointers should be applicable for all marketing case studies.

Jane

Hi Kohki, I agree with your conclusion on Case studies. The pace at which things change in digital marketing is breakneck. The strategist is the number cruncher, the person who calls all the individual experts to account at the end of the month to see if they are increasing or reducing ROI. It is difficult to over-state the importance of this role.
Reference:- http://www.dpfoc.com/australia/blog/successful-digital-marketing-is-like-beautiful-music

http://www.acewebacademy.com/ Acewebacademy

It is extremely easy to get misled by so called
case studies as you work
on a marketing strategy- Mighty useful tips there.