Evaluating the Impact of Social Media: Are We Wasting Our Time?

June 27, 2012

(Claire Gibbons, Ph.D., M.P.H., is a senior program officer in the Research & Evaluation Unit at the Robert Wood Johnson Foundation. She spends most of her time managing R&E projects for the foundation's Quality/Equality team. A version of this post appears on the Foundation Center's Transparency Talk blog.)

Last month Steve Downs and I discussed some of the Robert Wood Johnson Foundation's (RWJF) experiences using social media and our first steps toward evaluating the impact of RWJF's social media use in a webinar for Council on Foundation members (you can view the slides here). In response to our evaluation discussion, a webinar participant asked whether it makes any sense to evaluate something as spontaneous and fun as social media. This was also a question raised by Allison Fine in her blog.

Allison expresses concern that a logic model "misses the essence of what makes social media so unique, the serendipity and fun that are essential parts of 'being' social." This is an interesting and valid question -- by creating a stodgy old logic model do we defeat the purpose of social media?

Before I share some thoughts on this question, let me describe briefly what we've been doing at RWJF. The staff at the foundation is using social media, and many are enthusiastic about its potential to increase our impact, but until recently no one had sat down to elicit exactly what we expect social media to accomplish. In fact, we engaged in a fairly lengthy period of discussion and experimentation before we began to plan for evaluation. One of RWJF's initial steps was to form a working group to consider how the foundation could best take advantage of Web 2.0 tools, and what it would mean for the foundation if it did use these tools. This working group released a report internally to all foundation staff in December 2009, and we began a period in which all staff was strongly encouraged to experiment with social media. (Just signing up for Twitter and listening to the conversation by following others was encouraged, for example.) A second Web 2.0 group was formed after some time passed that was charged with getting some sense of whether RWJF was moving forward with its use of social media and sharing lessons across program areas. It was at this point that we began to focus our attention on evaluation.

We decided that the first step in evaluating our use of social media should be to develop a logic model. With the help of consultant Victoria Dougherty, we did this based on interviews with staff that were knowledgeable and involved in our social media efforts and on a review of documents about our social media philosophy.

Then we created a logic model to help evaluate the impact of our social media use.

The logic model has two pathways: the first describes how RWJF can approach its work over the next five years and the second describes some of the outcomes of the work. For the foundation to realize the potential of social media and eventually reach its long-term goals for being a more effective agent of change and a connector and facilitator that spurs broad participation in our work, it must first position itself as a Web 2.0 organization and work to become more open and nimble. Social media use may also lead to creating new connections outside the walls of the foundation and in turn lead to a greater ability to gather information from a broad network that can result in more effective programming. (See my earlier blog post for a more in-depth discussion of our logic model.)

So, back to our earlier question: Does creating a logic model to drive an evaluation of the impact of social media use defeat the whole purpose of social media?

I don't think so. But I'm pretty sure you guessed I was going to say that, given that I'm a research and evaluation officer! So let me share my thinking.

Yes, social media in many cases is driven by spontaneity. Videos that go viral on YouTube are completely driven by spontaneous interest. But not all social media use is purely spontaneous. We believe that social media can be used strategically to further our programmatic goals. That means we can plan ahead to use a social media tool or tools. For example, staff at the foundation used a virtual forum to create an open platform for discussion and idea-gathering about teen dating violence prevention. Through the platform, we received thoughtful input from people working in the field, teens who had experienced dating violence and parents who lost a child due to dating violence, and many others. Read more here.

A logic model does not squash innovation -- it describes it. Logic models are made to be broken and expanded and changed over time. The presence of a logic model is not meant to limit anyone's activities to something that happens to be featured in a little box in a diagram. The model doesn't dictate our programming -- it's simply a way to describe what we are doing and what we think the result will be.

RWJF's use of social media in the workplace is predicated on the idea that it will help us achieve our goals. We could be wrong. We won't know if social media is getting us anywhere good, or anywhere good faster, unless we measure some outcomes that we think are related to our programming activities. And one very useful tool for eliciting expected programmatic and policy outcomes is a logic model.

This isn't to say that we think all possible pathways between the use of social media and good outcomes are contained in our logic model. Absolutely not! This is just the best picture we could come up with at this point in time. New tools will become available, and we'll use them. Staff will continue to innovate in ways that we haven't imagined yet, and we welcome that innovation.

We still have a long ways to go in our journey to use social media in a way that helps us reach our strategic objectives -- and in measuring and evaluating our use of social media. We certainly don't have many answers, but I think we're on the right track. What do you think?

Comments

You can follow this conversation by subscribing to the comment feed for this post.

“We still have a long ways to go in our journey to use social media in a way that helps us reach our strategic objectives.” – I most certainly agree. But I must say that, although social media is considered as a baby in the marketing and communications’ world, it has already established itself as an influential medium to its audience. And I agree with your point, that to make this method spontaneous, critical planning is needed. Planning will help a media strategist correctly use the channels to relay the message wide and clear.