Evaluation

Amanda Cooper (@ACooperKMb) recently released her evaluation of 44 Canadian Research Brokering Organizations. She presents a quantitative method for evaluating the effort of a system of knowledge mobilization.

Knowledge mobilization struggles with evaluation. Evaluating an individual instance of knowledge mobilization is feasible with the right base line and pre/post intervention metrics. But rolling that up and evaluating a system of knowledge mobilization (like any one of the knowledge mobilization units in the ResearchImpact-RéseauImpactRecherche network) has so far proven challenging.

So thank you, Amanda Cooper (Assistant Professor in the Faculty of Education at Queen’s University, Kingston, Ontario). Amanda recently posted a report titled “Knowledge mobilization in education: A cross-case analysis of 44 research brokering organizations across Canada”. Amanda developed a quantitative methodology to evaluate the efforts of Canadian research brokering organizations (RBOs). The methodology is based on the evidence about research utilization. We know that people centred methods encourage greater research use than do those based solely on making package knowledge accessible to decision makers. In the words of Sandra Nutley and her colleagues in Using Evidence, “[p]ersonal contact is crucial … studies suggest that it is face-to-face interactions that are most likely to encourage policy and practice uses of research” (page 74). In Amanda’s methodology points are assigned depending on how the RBO employs products (12 points), events (20 points) and networks (20 points) as well as overall features (20 points). You can see that more points are assigned to people centred methods (events and networks) than are assigned to purely product based methods. How points are assigned is detailed in Appendix B of her report.

Amanda used RBO’s web sites as the data source and scored each of the 44 RBOs on a scale out of 100. Amanda cites ResearchImpact as one of the RBOs but the data she used pulled from York’s Knowledge Mobilization Unit. The Harris Centre, another RIR member, is also included separately as one of the 44 RBOs.

Key Point #1: This is a quantitative methodology that is reliable and reproducible citing satisfaction with the inter-rater reliability testing of the tool and the average intra-class correlation coefficient.

Key Point #2: This method evaluates a system of knowledge mobilization not the efficacy of an individual knowledge mobilization intervention.

Key Point #3: This method measures the efforts of Canadian RBOs. It does not measure impact of the RBOs efforts. That more effective RBO efforts will result in greater impact of those efforts is a testable hypothesis, but it makes sense that this would be the case.

Each with a score of 81%, RIR-York tied with the Fraser Institute and Canadian Education Association as the top performing RBOs. Fraser Institute achieved this score with a budget of $12.8M. CEA achieved this score with a budget of $2M. York’s budget for knowledge mobilization is approximately $250,000. RIR-York accomplished the same effort on a fraction of the budget. The data from the top nine ranked RBOs is presented below.

Rank

Organization

Type*

Size (FTE)

Operating

Expenditures

Score on KMb Matrix (%)

1

1.2.1 RI

NfP, university research centre

Small (3)

$250 000

81

1.2.4 Fraser

NfP, think tank

Large (60)

$12,808,690

81

1.4.2 CEA

Memb, network

Small (9)

$2,044,892

81

2

1.2.4 AIMS

NfP, think tank

Small (5)

$872 234

78

3

1.2.0 CCL

NfP, general

Large (77)

$20,583,490

76

1.2.3 The Centre

NfP, issue-based

Large (25)

$5,685,000

76

4

1.2.0 TLP

NfP, general

Large (74)

$5,293,039

75

1.2.1 HC

NfP university research centre

Med (11)

—

75

5

1.2.0 CCBR

NfP, general

Med (12)

—

74

6

1.1.2 E-BEST

Gov, district level

Small (6.5)

—

72

7

1.2.1 CEECD

NfP, university research centre

Small (9)

—

69

1.2.2 P4E

NfP, advocacy

Small (9)

$537,806

69

1.2.3 LEARN

NfP, issue-based

Large (33)

$3,000,000

69

8

1.2.1 HELP

NfP, university research centre

Large (50)

$7,200,200

67

9

1.1.3 CSC

Gov, standards

Large (20)

$3,849,254

65

We need more research like this into the processes of knowledge mobilization, engaged scholarship and community based research. Much of what we know comes from individual studies of individual instances of knowledge mobilization. As these activities become more embedded in institutions and systems we will increasingly need research on these systems and how they create infrastructure to support the individual instances. You can read more on other methods for evaluating the impact of research like Payback and Productive Interactions in a 2011 Special Edition (Volume 20, Number 3) of the journal, Research Evaluation.

Thank you to Amanda for your important contributions to this emerging field.

On Tuesday September 25 , I cleared my morning so that I could be the only Canadian participant in a workshop on Policy Influence and Monitoring. The workshop was in Cornwall in the UK. I was in Toronto in Canada. WebEx and Skype connected us.

Knowledge intermediary work is a global phenomenon. Look at the K* conference in April 2012 that was attended by participants from 5 continents. There are well established practices to enhance the impact of research on policy and practice in developing countries seeded by international organizations like the International Development Research Centre (Canada) and the RAPID program of the Oversees Development Initiative in the UK plus many more. These organizations work with local Southern partners to enhance the impact of research on the lives of citizens in the Global South.

Impact has become a significant component of the research cycle but how does one actually do this? Dr. Sarah Morton from the University of Edinburgh offers a one-day workshop with methodologies to assess the impact of your research.

David Phipps of York University (@researchimpact) has recently written about Knowledge Hypocrites. Well, I for one am taking action (sort of). I wouldn’t exactly call myself a bookworm, but I am taking opportunity to learn from other professionals to help inform my practice. Perhaps not a direct solution to David’s point, but I am happy about opportunities to learn from leaders in KMb from Canada and internationally.

I was one of 12 present and felt I was in a unique position, whereas I am not directly involved in or leading a research project, but play a brokering role in developing research projects. For the purposes of the workshop I used York University’s partnership with the United Way of York Region as an example (we are piloting a community knowledge broker role). For me, in my experience, impact is misunderstood with outcomes, or even outputs. So it was refreshing (and validating) for me to hear Sarah speak about a process of inputs, activities, uptake, use and impact.

The significant takeaway for me was a mapping exercise which will help me working with university researchers in developing knowledge mobilization plans. In Sarah’s research and experience, embarking on a process of examining potential assumptions and risks around the process listed above can actually help determine potential indicators around impact. Unlocking a procedure to support this process will help me in my brokering work. The fact that I can employ the tools and not have to read Sarah’s dissertation makes my life somewhat easier. Can I declare myself hypocrisy-free? No, not yet, but I do prefer this active process of knowledge exchange. My thanks to Sarah for sharing her research and methodologies to further unpack the notion of impact in research.

What is social innovation? If knowledge mobilization (the process) results in social innovation (the outcome), knowledge brokers need to capture and evaluate social innovation outcomes that arise from knowledge mobilization.

I have recently read four interesting and thought provoking pieces on social innovation.

The Power of Social Innovation (2010) by Stephen Goldsmith presents the lessons learned from a two term mayor of Indianapolis including case studies from education, poverty and housing. This book is also backed up by a website and the author may be found tweeting as @powerofsocinnov.

The Philanthropist (2010) released an Open Access edition (Volume 23, Number 3) of their journal featuring articles about Canadian social innovations. The entire edition and individual articles can be found here. This journal featured articles written by Stephen Huddart of the McConnell Foundation and Frances Westley of SIG @ Waterloo among others.

“Horizons” is published by Canada’s Policy Research Initiative. In February 2011, Horizons published a number of articles on social innovation and the community sector. This follows their earlier briefing on social innovation that did little more than ask the questions. You can read their analysis and their answers online here by clicking on the left hand menu bar. There are many articles including ones on social impact bonds, public sector innovation and an interview with Louise Pulford of the Young Foundation in the UK.

Social innovation is “the creation or application of research and knowledge to develop sustainable solutions to social, environmental and cultural challenges. Social innovation results in more efficient and effective human services, more responsive public policies and greater cultural understanding.”

It’s a good definition as it links to results but the most elegant definition I read was from Louise Pulford in which she defines social innovation as “devising new and better ways to tackle social problems”. This is simple and easily understood but what relationship does it have to knowledge mobilization (KMb) and the work of ResearchImpact-RéseauImpactRecherche and the many engaged scholarship and research projects underway in Canadian communities and universities?

In our 2009 paper we linked our definition of social innovation (above) to KMb by saying that “KMb (the how) enables social innovation (the what)”. We have recently refined this to say that social innovation is the outcome of the process of knowledge mobilization. KMb isn’t an end in itself. KMb is a process that allows researchers and their partners to develop social innovations which are new and better ways to tackle social problems. At RI-York we have spent 4 years focusing our attention on the process of KMb but without looking we have also supported some examples of social innovations:

The Parkdale Activity and Recreation Centre (PARC) Heat Registry. The Heat Registry is a community based innovation that tracks and provides services to poor and vulnerable populations at risk of heat exposure on hot summer days. The Heat Registry is based in part on the research and evidence collected by Tanya Gulliver, a York University Knowledge Mobilization Intern whose research helped the PARC Heat Registry raise operating funds that benefited the community and citizens of Parkdale. The increased risk of heat exposure due to poverty is an example of a social determinant of health.

In the summer of 2010, York U and United Way of York Region jointly supported three interns who undertook research to map social assets in York Region. The work of these three interns previously appeared in Mobilize This! on June 21, 2010 and their work provided data and evidence to inform a UWYR Board decision to launch Strength Investments, a new form of UWYR investment that invests in the strengths of coalitions of York region citizens and community groups.

Susan Lloyd Swail (former MES graduate working with Professor Gerda Wekerle) was a KMb intern jointly funded by Nottawasaga Futures and MITACS Accelerate program. Her research contributed to the development and launch of the South Simcoe Green Economy to Nottawasaga Futures (see Mobilize This! from April 13, 2010) . Susan is now Manager of their Green Economy Centre that provides knowledge and services to local business to improve and maximize the efficiency of Green Business Practices.

As we grow in our KMb efforts we need to evaluate on outcomes (social innovation) as much as on process (KMb). We first wrote about social innovation in our 2009 paper. Now that more people are writing about and acting on social innovation we will also update our thinking and re-energize our actions to capture and evaluate social innovation outcomes as well as KMb processes.

ResearchImpact-RéseauImpactRecherche is still Canada’s knowledge mobilization network but now we know that we need to articulate those social innovations that emerge from our KMb efforts.

SSHRC invited 34 knowledge mobilization projects from their Knowledge Impact in Society and SSHRC Clusters to a workshop in Ottawa October 22-23.

Day 1: Not being challenged by systemic introversion our mob of mobilizers (mostly academic leaders, some project coordinators and two lone staff leading institutional knowledge mobilization services at York and Memorial) had no problem mashing up in different combinations be it in their KIS or Cluster cohort or the sector of primary engagement. Most of the day was spent exploring “issues” around knowledge mobilization. The usual topics of incentives, barriers, metrics & evaluation were on the agenda. Refreshingly some new topics including an alleged research/KM dichotomy and social media were also discussed.

Research vs KM got a lot of play with opinions on both sides of and in between the hypotheses that research and KM are either on a spectrum of activities or they are two sides of a coin, related but separate. ResearchImpact works with researchers, their institutions and their non academic research collaborators to create space for basic research AND space for applied research linking to extra academic impact (thank you Clair Donovan) as well as a spectrum of activities and services in between. KM is a process intimately interwoven with research. It is not a discrete event that happens in isolation of the research. Measures of extra academic impact complement, they do not conflict, with measures of academic quality. A repeated theme was the desire for infrastructure (cash, expertise, systems) to support the spectrum between basic research and extra academic impact.

Also interesting was a breakout session on social media. ResearchImpact tweeted @researchimpact during this session resulting in a number of RTs and DMs – on the spot web 2.0 mobilization of knowledge about knowledge mobilization. SSHRC, our academic researchers and their non-academic research collaborators only need to look at their graduate students to see how social media will play an increasingly important role. You don’t have to lead the wave but if you don’t ride it, it will pass you by.

Reception done. Dinner done. Blog written. Beer being consumed thanks to Southern Cross Grill. Need to recharge before day 2 of this important workshop. Thank you SSHRC for creating this space where 34 mobilizers could walk into a bar and begin to network. Trouble was it was a cash bar and alcohol is not an eligible expense on a SSHRC grant even though it is a key success factor in networking and knowledge mobilization! Maybe we’ll make that a recommendation for future program development.

The KM Units which are leading the ResearchImpact network (York University and the University of Victoria) are undergoing an evaluation, and we would like your feedback! We are asking you to complete a survey regarding your experiences with the Knowledge Mobilization Unit. The purpose of the survey is to gather information on how a specific research partnership may have influences you, your organization, and/or your community. The survey should take approximately 15 minutes of your time and will be greatly beneficial in the improvement of the KM Unit. Please access the survey by clicking this link.