May 15, 2012

Your Data Is Only As Good As What You Do With It

The Paine of Measurement

We need a fundamental attitude shift in social media measurement. And, no, I’m not going to get on my “take more math and stats courses" soapbox. What I’m talking about here is a new attitude about data and research. Too many people seem to think data is the goal, but it's just the raw material.

1. Stop collecting data, start understanding it.

Our industry has been been collecting data for a long time. At least since Kellogg's was counting the buyers of their cereal by encouraging me to send in boxtops to get whatever useless plastic object they were giving away that month. Today we have no shortage of data, but there is an enormous shortage of sentient human beings that actually dive into it, understand it, and draw conclusions.

Personally, this is great for KDPaine & Partners; it’s what we like to do best. Short of a day on my sailboat or in the garden, there’s nothing that's as much fun as digging into a digital haystack of 40,000 records and finding that one needle that will actually change how my client communicates.

However, it is terrible for communications and marketing in general. It shouldn’t be just me digging in and finding insight. All of today's communicators need to be waist deep in that data; testing hypotheses, finding what’s not working, and most important, using the data to argue for better programs.

2. Stop making lists, start making informed decisions.

It’s probably a word vs. numbers thing, but why is it that so many people in our profession would rather make lists of tasks to be accomplished than study the data and learn what activities they should be doing? For every project you take on and every request that you get, your first question should be: “What’s the goal?” And the second: “What does the data tell us?”

3. Trusting a computer to get you the right answer is stupid.

Ever since text analytics came along in the late '90s, we’ve been expecting technology to come up with the Easy button for measurement. The reality is that research is complicated and you need to be incredibly detail oriented to do it correctly.

It also means that trusting a computer to get you the right answer is stupid. A computer can be programmed to tell you all kinds of things, but it can’t tell you what to do. It doesn’t understand the internal politics, the personalities and relationships involved, or anything else that you haven’t specifically told it.

And, of course, computers can make incredibly stupid decisions. Read Shel Israel's Forbes column about a computer that came to a conclusion that any human would have seen was silly, with serious reputational impact:

"More and more these bots are being trusted to know what’s best for us. In my view that is both Orwellian and overly parental. In themselves, these are useful tools, except these automated pieces of software lack common sense. They make goofy mistakes all of the time..."

I frequently tell a story about working for the Federal Reserve Bank of Cleveland and finding an alarming spike in negative comments in their coverage. It turned out that the computer was seeing headlines that said Cleveland was fed up with basketball star Lebron James after he left Cleveland and went to the Miami Heat. All it took to figure it out was an old-fashioned human reading a headline or two. We may be slower, but we have certain abilities to make judgements that machines can't match.

Do you really understand the data behind those pretty little green or red lines on the charts in your PowerPoint presentation? If you don’t you should. As smart as most computers can be, they still get fooled on a regular basis by bots, spam, content farms, and other fake stuff that then winds up in your database. So when you’re pointing to that enormous 10 trillion “hit” spike in coverage, you better know exactly what is behind it.

Happy Measuring,

###

Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements. Katie and Beth Kanter are authors of the book “Measuring the Networked Nonprofit,” to be published this year by Wiley.

Comments

You can follow this conversation by subscribing to the comment feed for this post.

Excellent post.

I don't trust any tool that doesn't show me it's raw data and/or how it calculated its results. If all it does is spit out a number or a pretty pie chart, I'm not touching it. I don't expect the data vendors to share their recipe, but at the very least I expect to see an ingredient list of what you're expecting me to swallow.

Great read about a crucial question that companies in India ask of Carma International - the company I represent. In most cases companies in India indicate their objectives of content analysis is to gain strategic insights about their PR program, draw learnings for reputation management, create competitive edge or respond to a crisis. In all such instances, value of human based content analysis is irreplaceable. We at CARMA have also deliberated on this matter http://bit.ly/wl8fuo and http://bit.ly/x8TuOV twtr:@khannaneelima