OUR NEWS

How can evaluation improve community and other systems change initiatives?

At last November's American Evaluation Association's (AEA) meeting, Community Science's David Chavis participated in an invited session on the evaluation of community change initiatives (CCIs) within their context. He reported on a Community Science study that reviewed 11 CCIs to identify factors related to scale, scope and sustainability. He also addressed the changing landscape of evaluation. The following post-session discussion sheds light on how the work of community change will evolve in 2010 and beyond.

Right now, it’s the readiness of foundations, evaluators, and technical assistance providers to undertake these types of initiatives; particularly funders, because the funding comes first. Funder readiness is the biggest challenge. Despite the repeated attempts at taking on long scale change efforts, very little time is spent on sharing and applying lessons learned from other initiatives. Particularly, the information and capacities of all participants needed to achieve the scope and scale required to do this type of work, how to build that capacity, and how to generate reliable information that will be used. This is extremely complex work. Rocket science and brain surgery are easier. The laws of physics are constant. However, that’s not true with communities, which are changing all the time.

How do science and research factor in?

When we {Community Science} get involved, we explore how to use knowledge, data and research to address changing systems. We help public and private funders strategically and operationally apply the knowledge and the science of how to improve communities on a larger scale. It’s a matter of balancing the visionary notions of philanthropy with the science and knowledge that are necessary to be as successful as possible. Funders are rarely organized to use information for decision making and improvement and do not reward that behavior, even if they have evaluation staff. Community Science helps funders develop learning systems that can enhance the performance of their grant making and other activities.

What’s a typical track of a foundation?

Foundations frequently jump in and learn how to implement their initiatives as they go along, even if they have done considerable strategic work in advance. Their implementation becomes a series of hits and misses. They make sincere promises and plans, but don’t do their homework and build their own capacity to implement their plans. If the foundation is unclear of its own initiative, and they don’t use data for learning, the participants and the communities don’t do it either.

What changes do you think would make a difference?

It will make a difference if foundation board and senior leadership recognize that community change is a process of organizational change and learning. Community and other systems change initiatives require funders to take on new roles. Staff are often not prepared for the challenges they are trying to change. They require training and the foundation needs to be structured to support this challenging and critical endeavor. Information systems and evaluation have to be integrated into program functioning and not be isolated or viewed as burdensome. In order to be worth the effort, the evaluation needs to be useful to foundation staff and all other participants, especially grantees. Information needs to be accessible and understandable to a broad and diverse audience.

Realistic expectations would also make a difference. Community and other systems change require organizations, including the funder, to change, as well as altering the relationship with other organizations before they are able to achieve the scale that is required to make their desired changes. Funders being prepared to accept and transform conflict would also make a huge difference. These initiatives, by their nature, stir up historical tensions and inequities that funders will need to prepare to address. The change process itself, if it really is making structural changes in inequitable conditions, will lead to conflicts. Where there’s change, there’s friction.

We {Community Science} can be most impactful when we are brought in early. One of our strengths is in helping foundations and other funders develop realistic plans based on previous knowledge of what’s happened in the past and an assessment of the current opportunities and challenges. We can help with staff and consultant training. We can help develop an overall capacity building or readiness plan which gets everyone involved, so the foundation or other funder is ready to take this on. Foundations can’t assume that the knowledge and capacity they already have are sufficient for this type of large-scale task. For example, foundations sometimes ignore historical tensions between groups. It is important to build an intentional strategy to transform conflicts up front to improve capacity-building.

How does Community Science factor in?

We use evaluation and capacity-building tools as part of the planning and learning process, in order to help funders. We help make the adjustments they need to become more effective and successful. Again, this is especially true when we are brought in early on in the planning process. A good example is in recent work we did with a large foundation funding a place-based initiative; there were many tensions and misunderstandings toward the foundation brewing early on. We met with community leaders and other participants, heard what their challenges were, and fed back that information to the foundation. After meeting with us to review, the foundation made significant adjustments early. Also, drawing from our past experience, we gave recommendations based on our experience as an evaluator and implementer of how they could handle these situations – we were able to help on a very practical level.

Did any particular issues surprise you at the session?

One issue that clearly riled the crowd was about how difficult it is to get data from government or other public agencies. And, how groups can’t gain access to data without paying for it, even if they were already paid for by public and foundation funds. For example, if there is a survey that a private company is paid to administer on substance abuse, the private company provides the government a report. Then, if another community groups wants access to that data, the group would have to pay for it. So a marketing or research firm that has the money might be able to pay for data, but a group with little money can’t. It’s really scandalous, and many people in our community are frustrated by the road blocks to data access that others are paid to collect from communities using public funds, but those same communities can not have access to that data.