October 07, 2012

Report from The Conclave 2012: Social Media Measurement Standards In The Making

by Bill Paarlberg, Editor

Last Wednesday I was a part of the annual Conclave, at which a score of very smart social media measurement experts deliberated, debated, and decided on definitions of some of the fundamental terms of social media measurement. It was measurement geek heaven.

Mentions. Reach. Engagement. Influence. You might think that you know what these terms mean. And certainly the social media experts must know, right? Wrong.

Once you get in a room with people from all facets of the industry -- who do measurement, and buy and sell measurement, and do research on measurement, and who just think about measurement -- the current state of confusion becomes obvious. It is a very untidy situation. And The Conclave is one of the groups that is working to sort it out.

The problem facing the measurement industry is that a group of entirely competent social media measurement experts can start with the same raw data, yet their research may reach very different results, depending on who they are doing the research for, the type of industry they are studing, and how they define the fundamental terms involved. As a result the research can be flawed, and it probably can’t be compared to other research.

Without consistent and replicable research, the credibility of the whole industry suffers. And if everyone in the marketplace doesn’t agree on just what they are buying and selling, then commerce is slow and difficult. We often take for granted the many standards that allow us to engage in quick and convenient communication and commerce. Everything from the time of day and monetary units, to USB and HTML. It's time social media measurement had some standards of its own.

So, really, how exciting can it be to watch a room full of very smart people wrangle and wrestle over the meaning of research terms? Pretty darn exciting, as it turns out. Ideas and concepts bounce back and forth. Terms are introduced and the noise level increases as possible definitions, nuances, objections, and qualifications are debated. Eventually some conclusions are proposed and things calm down while everyone sits and ponders. Then someone says, "But what about ...?" New points and objections come up, the conclusions get thrown out, and the messy-yet-brilliant process starts again.

And when you consider that these definitions are the groundwork for progress in the field over the next few years, this undertaking is, well, kind of exhilarating.

Here's an example. "Influence" is one of the tricky terms The Conclave dealt with. It has already been defined in a useful way (thanks to Philip Sheldrake), and a not so useful way (thanks to Klout). Our discussion of it led to one of the most interesting moments of the day. Katie Paine, who was leading the session, said, “OK, suppose you had all the money and resources in the world and you wanted to design an experiment to measure influence, how would you do it?” And the room went absolutely quiet. Twenty very intelligent people were rendered speechless as they puzzled over this difficult problem. Now that’s some measurement geek excitement right there.

Here is how you can get involved. The results of The Conclave will be posted at #smmstandards.org, and your comments are important. The standards-setting process is that the first pass work is posted there and circulated for comment, and then the revised progress is published as an interim standard. What you can do right now is to go over to smmeasure.org and take a look at the proposed Transparency Table. It is on page 16 of the March to Standards report. Post your comments.
###