12 Comments

The apparent rush to embrace ‘huge data’ needs to be tempered by a simple truth. Paraphrasing here: “Numbers don’t know where they come from.” Lord, F. (1953) On the statistical treatment of football numbers. American Psychologist, 8, 750-751 was required reading in graduate school (decades ago). It offered a contemplative alternative to the other required reading:”How To Lie With Statistics.” Implicitly, we were being schooled in judgment,the principles of scientific discovery, the identification and avoidance of bias, and the proper use of analytic tools. It seems nowadays the mere fact that numbers exist compels action. Best to ask: Who knows where these numbers come from?

For years, I worked with businesses in the printing industry – an industry plagued with overinvestment in rapidly aging technologies and characterized by continually declining production volumes. Year after year, industry experts reassured print-shop owners with surveys that showed “the slump was over” and print buyers would spend more in coming years. Invariably, these survey results were based on the owners’ collective opinion (read: “hope”) that the market was about to turn – that happy days would soon be here again. And many printers believed the forecasts… because they needed to. Despite obvious trends to the contrary. (Some of which were megatrends.)

What’s the root cause of our addiction to unsubstantiated good news? I’m no psychologist. But I can’t help but be reminded of a Monty Python skit (The Meaning of Life, 1983) in which a British army doctor offers a comforting prognosis after his patient’s leg has been “bitten sort of… off.”

“Yes, well, this is nothing to worry about… there’s a lot of it about… probably a virus. Uh, keep warm, plenty of rest, and if you’re playing football or anything, try and favor the other leg… Be right as rain in a couple of days… Any other problems I can REASSURE you about?”

We seem to have an undeniable need to believe in something. And it might as well be something pleasant. Maybe we find comfort in survey results that REASSURE us. Even when our legs have been bitten sort of… off.

Very valid point when it comes to the challanges of using data. In terms of the solution, it’s all about using the right methodology in the end? Big data still relies on all the basics we learned in market research and statistics. However, how many organizations are trully data-driven? I would imagine that the emotional and political factors still play a major role… so data is not the answer to everything!

For the record, I like data and the analytical process of working with it – if you can approach it with an open mind, it creates opportunities to think in new ways, see connections you might not have otherwise considered, etc.

But I did some work with one of the big guys in data analysis and two things stick with me: 1) the mathematical correlation between various data points is only meaningful if it can lead to *practical* action (noting that “practical” is something which depends on factors unique to each situation); and 2) there were many times when a group of people involved in a project were looking at the exact same data and coming up with very different “insights”, sometimes including ideas which were nearly opposite in terms of interpretation.

I chalked up those different interpretations to a sort of “bias” though not meant in any negative way but a factor regardless. Interpretations of information is of course influenced by each of our individual experiences, skills, etc. A senior-level marketer with top-10 CPG brand experience and a major budget will process, consider and interpret information differently than a mid-level marketer who has worked with smaller brands and all that implies. Marketers in the high-tech vertical will likely process information differently than those in low-cost consumables.

Regardless of my examples, the point is that there’s never only one “correct” way to interpret information. So I have come to believe that the trick is to pick an interpretation which “feels” right, which “fits”, and apply it *with passion* — stick to it, commit to it over a reasonable period of time.

In general, we don’t get to do (big) things two different ways simultaneously, measure them, and take credit for the better approach. That said, it’s my feeling that every practical approach will deliver results and the variation in outcomes, if they could all be done and measured, would likely *not* be substantial. Please assume I mean practical ideas and note that I’m not dismissing the benefits of better results (if you can predict them…) but rather making the point that over-thinking and over-analyzing data is less important to the outcome than making the commitment to an approach and sticking with it long enough to make a difference.

Why not just agree on the relevant metrics ahead of time? That’s part of my standard checklist prior to running every A/B test.

I think the problem you’re making fun of here is the difficulty in finding someone to run the data that understands business – that’s a *very* hard position to hire for. This is why I’m constantly advocating for marketers to understand a little bit of SQL – that way they can speak the data person’s language. One of the big challenges for marketers is that everyone thinks they know marketing. I also think that a much worse problem is marketers doing things on gut rather than trying things they think will work and then finding out if they do based on metrics agreed to ahead of time.

You need look no further than the swill delivered up by our elected so called “leaders” in the political sphere for examples of selective use of data that “supports” a pre-determined point of view.
I am sure it is the same in the US as it is here (Australia), these political ghouls break new ground on selective data manipulation on a regular basis.

Approaching data with a testable hypothesis is a way to find the needle in the haystack (the “signal in the noise”). Too many data mining projects are undertaken to see IF there is a needle in the haystack (which is a predictably open-ended, expensive, and unpredictable process), rather than proving or disproving a hypothesis.

It’s been my experience that organizations make decisions based on minimal or incomplete data because their platforms are not fully integrated. For example, the ability to track email clickthroughs but inability to relate them to web activity. These sets of unconnected data show 2 completely different results that provide no insight, although they are used as the basis for big monetary commitments.

“Big data” has been the big-buzzword in recent years. Don’t get me wrong, it still has a valid role for many marketers. What is becoming increasingly valid for more-and-more organizations is interpreting data gleaned from their own inbound marketing efforts. Their customers. Their prospects. Real beating hearts, raising their (real) hands–in real-time and telling us what they think, like and want. Analytical psycho-babble, I think not. Intuitive marketing, backed by solid data based insights…you bet. The beauty part of the software today is that you don’t need an army of IT people to run queries to interact and figure it out. Oh, and when the guy on the other side of the table is quoting his “unbiased data” to call your idea stupid…well your idea might just make more sense…and you can prove it with revenue and real-time customer interactions. Good stuff!

GM Credit card built a database with 250 variables on each person. They wanted to answer the question, “Who would have bought a car anyway, without the $5,000 rebate?”
“I love those kinds of questions.”I replied.
“Why?” They asked.
“Because I can answer any way you like.”
“But we thought you were a data person.”
“But I suspect that you have no data for that
question, so I can answer it any way that you like.” They hadn’t set up appropriate testing samples. With all that data, it wasn’t a data issue.

The problem with this critique is that data doesn’t tell a story. Only human beings can see an insight, a story coming together, and then turn it into action. Structuring criteria ahead of time (as Chris N suggests) is a useful discipline to help more objectively evaluate whether or not your story (hypothesis) is likely to be true. But data will never provide “objective” insight, because only human beings can connect pieces into a whole, and we are subjective by nature. The good news is, so are our consumers.