CAT | Social Impact Measurement

I’m Andrea Nelson Trice, President of Trice & Associates, an evaluation and consulting firm. This case came from my research for a book project on the human dimensions of social enterprise success.

Tony, a successful entrepreneur, visited several emerging markets and determined that dependency on fire for light is unacceptable in the 21st century. The health and safety dangers alone make an alternative essential. He knew how to design low-cost solar lights, so he quit his job and began building a social enterprise to address this problem.

He received a grant to give away thousands of his lights with the goal of priming the pump in multiple markets. Now, two years later, we’re brought in to evaluate the enterprise’s impact. The problem is, the company is far from breaking even. “I don’t know how many more things I can try,” Tony says. “People just aren’t buying our lights.”

As evaluators, do we simply pull out a standard template to evaluate the work, or do we risk asking deeper, more difficult questions around assumptions that are driving the enterprise? In interviews with Tony and emerging market social entrepreneurs, I’ve heard very different perspectives about “the problem.”

Rad Resources:

Perhaps one of the most important things we can contribute as program evaluators is help identifying faulty assumptions that guide the work. Here is my website, which includes more on this.

Increase your understanding of cultural differences. One of my favorite resources is from Professor Geert Hofstede, whose research team highlights national cultural differences.

Hot Tips:

As a Do-It-Yourself culture, we often assume we can make sense of cultural differences on our own. That’s rarely the case. Expats, who have lived in a culture for years, can be great resources.

Consider the Amish. It may seem futile to market solar lights to people who have no problem with their current light sources. But how often do we unintentionally overlay our values onto another culture as we work to solve a “pressing need?”

The American Evaluation Association is celebratingSocial Impact Measurement Weekwith our colleagues in theSocial Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from ourSIM TIGmembers. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpageso that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org. aea365 is sponsored by theAmerican Evaluation Associationand provides a Tip-a-Day by and for evaluators.

This is Heather Esper, senior program manager, and Yaquta Fatehi, senior research associate, from the Performance Measurement Initiative at the William Davidson Institute at the University of Michigan. Our team specializes in performance measurement to improve organizations’ effectiveness, scalability, and sustainability and to create more value for their stakeholders in emerging economies.

Our contribution to social impact measurement (SIM) focuses on assessing poverty outcomes in a multi-dimensional manner. But what do we mean by multi-dimensional? For us, this refers to three things. It first means speaking to all local stakeholders when assessing change by a program or market-based approach in the community. This includes not only stakeholders that interact directly with the organization, such as customers or distributors from low-income households, but also those that do not engage with the venture ? like farmers who do not sell their product to the venture, or non-customers. Second, this requires moving beyond measuring only economic outcome indicators; it includes studying changes in capability and relationship well-being of local stakeholders. Capability refers to constructs such as the individual’s health, agency, self-efficacy, and self-esteem. Relationship well-being refers to changes in the individual’s role in the family and community and also in the quality of the local physical environment. Thirdly, multi-dimensional outcomes means assessing positive as well as negative changes on stakeholders and on the local physical and cultural environment.

We believe assessing multidimensional outcomes better informs internal decision-making. For example, we conducted an impact assessment with a last-mile distribution venture and focused on understanding the relationship between business and social outcomes. We found a relationship between self-efficacy and sales, and self-efficacy and turnover, meaning if the venture followed our recommendation to improve sellers’ self-efficacy through trainings, they would also likely see an increase in sales and retention.

Start with existing questions developed and tested by other researchers when possible and modify as necessary with a pretest.

Pretest using cognitive interviewing methodology to ensure a context-specific survey and informed consent. We tend to use a sample size of at least 12.

For all relevant questions, test reliability and variability using the data gathered from the pilot. We tend to use a sample size of at least 25 to conduct analysis, such as Cronbach’s alpha of multi-item scale questions).

The American Evaluation Association is celebratingSocial Impact Measurement Weekwith our colleagues in theSocial Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from ourSIM TIGmembers. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpageso that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org. aea365 is sponsored by theAmerican Evaluation Associationand provides a Tip-a-Day by and for evaluators.

Hello! We are Brian Beachkofski and Jeannie Friedman, Pay for Success (PFS) advisors at Third Sector Capital Partners. We spend most of our time assessing feasibility and designing social sector programs with rigorous evaluations and evidence-based interventions embedded into their contracting structure.

PFS is an innovative contracting model (shown in the figure below) that drives government resources toward high-performing social programs. The PFS model is designed to merge performance measurement using administrative data and rigorous evaluation of long-term outcomes into the contracting structure. This helps ensure that funding is directed toward programs that succeed in measurably improving the lives of people most in need.

Hot Tips:

Balance Factors in Evaluation Design: A randomized control trial (RCT) was once considered necessary for PFS evaluation, but now it is generally recognized that there is no one-size-fits-all answer. Factors such as operational complexities, sample size, observation windows, budget constraints, and limitations on service providers’ needs should be balanced against each other.

Focus on Outcomes: Aligning incentives around outcomes is a good first step. Providing interim insight on how the project progresses against those metrics allows the team to make improvements and act on those incentives. This feedback loop is fueled by interim outcome metrics and real-time program delivery modifications. Consistently keeping outcomes in mind maximizes final outcomes for those who are in need. Salt Lake County’s Homes Not Jail illustrates how different evaluation techniques apply to PFS.

Separate Payment from Policy: An evaluation intending to inform a payment decision is different than one evaluating a policy decision. A PFS project needs to clarify whether the evaluation is to inform payment, with quantifiable impact, or future policy, where causation is paramount. A good example of when we did that was in Santa Clara County’s Project Welcome Home.

Engage Stakeholders Early: Most PFS projects serve populations with complex, multi-faceted needs that cross multiple government agencies and community partners, which makes defining measurable and meaningful outcomes challenging. Collaboratively refining these goals into defined metrics can gain stakeholder buy-in from all partners.

Use a Pilot Period: Operationalizing data sharing, referral pathways, and randomization protocols are new skills for many projects. PFS projects are often a government’s first time releasing administrative data outside organizations. Protection requirements and prior practices can make data-sharing feel uncomfortable. A pilot period builds trust and experience in a collaborative shared-data project, easing the full project’s operations.

The American Evaluation Association is celebratingSocial Impact Measurement Weekwith our colleagues in theSocial Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from ourSIM TIGmembers. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpageso that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org. aea365 is sponsored by theAmerican Evaluation Associationand provides a Tip-a-Day by and for evaluators.

Social Impact Measurement (SIM) is important for the legitimacy, advancement, and management of impact investing. SIM can also help align incentives among stakeholders and improve communication. While innovative finance matured over the past decade, similar advancement in SIM is complicated by diverse approaches, methods, and tools responding to various stakeholders. Unfortunately, much of SIM is focuses on outputs, uses limited evaluative thinking, and doesn’t consider how change happens.

Lessons Learned:

To best capitalize on the currency of SIM, investors and development practitioners/evaluators need to bridge the gap between their practices. At the 8th African Evaluation Association Conference in Uganda last month, participants agreed that the evaluation profession has much to offer to overcome the challenges inherent in SIM. With support from The Rockefeller Foundation, Genesis Analytics curated the Innovations in Evaluation strand to start building this bridge by facilitating dialogue between investors and evaluators. A discussion output is below:

Lessons Learned:

For many years, the evaluation profession emphasized attribution of impact, but there is now a greater focus on contribution, which matters to investors looking to enhance the impact of their funds.

Investors use impact measurements for different objectives at different stages of the investment cycle. Evaluators must be flexible and responsive to meet these needs.

Some investors have been reluctant to embrace SIM, because they think a randomized control trial (RCT) is the only option, yet worry about the ethics of randomly assigning a treatment group. Evaluators and investors should share knowledge, particularly to explore the value of options beyond a randomized controlled trial, and jointly develop a contextualized definition of impact and a SIM technique based on this definition.

The American Evaluation Association is celebratingSocial Impact Measurement Weekwith our colleagues in theSocial Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from ourSIM TIGmembers. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpageso that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org. aea365 is sponsored by theAmerican Evaluation Associationand provides a Tip-a-Day by and for evaluators.

I’m Robert Picciotto, director general of the Independent Evaluation Group at the World Bank from 1992 to 2002. I oversaw evaluation there and in its sister institution, the International Finance Corporation, which uses equity finance to promote private sector development.

Evaluators charged with assessing the growing impact of the private sector in the social sphere face a significant challenge. The United Nations Conference on Trade and Development estimates that $4 trillion to $5 trillion in annual investments are needed to support the United Nations’ Sustainable Development Goals. Because development aid and charity represent only a fraction of this investment, private investment is a critical component of achieving these goals. Impact investing is a highly promising strategy, although currently only a fraction of all investment is channelled toward social impact.

Ethical investors expect a financial return and want corporate decisions to take full account of the public interest. Their concerns range widely and can include the environment, consumer protection, community relations, human rights, etc. Reliable information about the social and environmental effects of ethical investments is necessary to meeting their needs. Luckily, development assistance agencies have already laid the groundwork by collecting and reporting data to satisfy public and philanthropic funders who demanded results as a condition of continued funding.

As highlighted at the Impact Convergence Conference, impact investing has largely focused on articulating goals and tracking progress through a battery of indicators. By relying primarily on self-assessment, it is failing to do the independent analysis required in financial reporting.

The promise of impact investing could be fulfilled by bridging the impact investment and development evaluation worlds to share their vast operational experience, local country knowledge, and technical expertise in evaluation and investment. The multilateral development banks provide a starting place from their years of embedding independent evaluation into their corporate governance to identify, appraise, and fund major social programs.

Lessons Learned:

The Sustainable Development Goals and their adoption for goal-setting by the impact investing community may signal a relationship between social impact assessors and experienced development and evaluation practitioners. For example, ethical investors vitally concerned with results could seek the comfort of accurate social reporting by investing in derivatives that package social interventions funded by the loans and credits of multilateral development banks in Sustainable Development Goalbonds. These investment vehicles would be backed by existing systems to attest to the social value of each intervention. Finally, evaluation societies worldwide could establish communities of practice connecting impact assessment professionals and development evaluators.

The American Evaluation Association is celebratingSocial Impact Measurement Weekwith our colleagues in theSocial Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from ourSIM TIGmembers. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpageso that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org. aea365 is sponsored by theAmerican Evaluation Associationand provides a Tip-a-Day by and for evaluators.

Measuring social impact offers a new perspective into the bigger forces that affect all of us. As we interact with social media, consume goods and services, and work, it’s exciting to consider how our actions have the potential for positive or negative social impact.

In our role as evaluators, we have had several opportunities to learn about social impact. We consulted with companies and foundations interested in making a social impact to understand the business case for social investment; how to synthesize the investors’ interests with social outcomes; and the importance of knowing the field, the appetite of consumers, and where to push the boundaries in the name of social change. To further learn about social impact, we worked pro bono with a handful of newly formed public benefit corporations (PBCs) in Minnesota to craft logic models and articulate their intended social change.

As you’ll read all week, there is great potential and need for SIM in the private sector. We continue to be inspired and we are excited about the learning possibilities through the TIG – like the blog posts coming your way – because of the almost overwhelming complexity in which the private sector intersects with social impact. For example:

The financing/investment instrument itself can be intended to lead to a social impact – by clearly articulating outcomes, and then only paying for those results. ­­

Products and services can have a positive social impact, such as higher-efficiency/lower cost lighting, beauty products designed for underserved communities, etc. When The Improve Group became a Public Benefit Corporation in 2016, this was the social impact model we pursued.

Companies can attempt to have a social impact through their hiring, recruitment, and retention strategies – bringing more opportunities to specific targeted groups. As described in a recent Atlantic article, many companies want to benefit from broader perspectives – and avoid discriminating, whether intentionally or not.

Companies can choose to source products or services with a social impact in mind; for example, Minnesota-based Peace Coffee sources all its coffee from small-scale cooperatives.

Companies can attempt to have a social impact through their brand and marketing strategies, as Dove did with its Real Beauty campaign, launched in 2004 and followed with other initiatives This strategy can flop, too, as with some now infamous examples in recent months.

Rad Resource:

Our company joined our local Impact Hub to continue learning from other social impact organizations. The hub is part of an international network with resources in many cities around the world.

The American Evaluation Association is celebratingSocial Impact Measurement Weekwith our colleagues in theSocial Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from ourSIM TIGmembers. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpageso that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org. aea365 is sponsored by theAmerican Evaluation Associationand provides a Tip-a-Day by and for evaluators.

This is Karim Harji and Ted Jackson from ET Jackson & Associates, and our contribution on social impact measurement (SIM) focuses on training and curriculum development for evaluators and practitioners.

In collaboration with several African institutions – the CLEAR Centre at Wits University and Greater Capital in South Africa; and the Venture Capital Trust Fund, GIMPA Centre for Impact Investing, and the Institute for Policy Alternatives in Ghana – together with funding from the Rockefeller Foundation and IDRC, we developed and delivered executive education training workshops in Ghana and South Africa.

These workshops not only brought evaluators, practitioners and policymakers, but also stimulated cross-sector conversation on the relevance of measurement. Participants shared approaches and tools for their local contexts, and emphasized the need for greater cross-sector collaboration among investors, fund managers, social enterprises, evaluators and governments.

Developing Curriculum and Resources

Inspired by these sessions and the range of perspectives that were represented, we designed an open source curriculum on “Evaluating Impact Investing”. The course is built around three broad themes—building the field of impact investing, measuring the success of impact investments, and understanding the special issues emanating from the African context.

Impact investing in Africa is characterized by much dynamism, and as such, the course consists of 24 different modules – on topics such as Evaluation Approaches, Standards, Global Goals, Household Impacts, Negative Outcomes, and Evaluation Costs – with supporting notes, examples, case profiles, exercises and readings. We explore the use of well-known approaches such as Theory of Change, as well as contextual adaptations for engaging with the private sector.

This curriculum is a resource that we hope that academic organizations and evaluation networks can use and adapt for their own needs. That said, there is much more to be done, including more case studies authored by African researchers, and evaluation tools that are adapted for use within each region. We welcome contributions and suggestions from evaluators, as well as those who are designing and delivering training and education in evaluation.

(click for larger image)

Interested in learning more? For more information about the SIM TIG, see here. To join the SIM TIG, see here.

The American Evaluation Association is celebrating Social Impact Measurement Week with our colleagues in the Social Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from our SIM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Kelly McCarthy and I am the Director of Impact Measurement & Management at the Global Impact Investing Network – or GIIN – a nonprofit whose purpose is to increase the scale and effectiveness of impact investing. While we don’t make investments ourselves, we work alongside those who do, along with many other actors seeking to advance impact investing practice and channel more capital to generate positive social and environmental outcomes.

Though I am a new entrant to the AEA community, I share the same aim as many others in this forum: To collaborate with those who strive to advance sound IMM and evaluative practices to drive more effective decision-making. In our case, decision-making anchors on using social and environmental data to inform the allocation of investment capital and optimize the potential for it to deliver positive social and environmental results. One way we do this is through our oversight of the publicly available IRIS catalog of social and environmental performance metrics developed with input from evaluation professionals and investment practitioners. Other such ways include working with funds around the world to deepen the IMM capacities of their teams and supporting the growing profession of impact analystsin the financial sector.

To further this aim, as other contributors this week have highlighted, one important focus area for 2017 will be deepening meaningful connection (and action) points between the impact investment community and evaluators and other IMM stakeholders.

Action 1: Translating Existing IMM Practices for Investors

Impact measurement and management often engages diverse implementing stakeholders from a variety of disciplines, including impact analysts, evaluators, and impact investment strategists and decision-makers. As a result a large number and diversity of industry and proprietary IMM approaches (frameworks, ratings, and methodologies) has emerged. From the perspective of many impact investors, this creates confusion, is viewed as lack of consistency in the market, and presents a barrier for effective use of IMM in general.

As a very first step to address this challenge, we are developing a publicly available “map” of existing credible initiatives, methodologies, and assessment tools which are currently in use by – or can be translated to – the investment context (e.g., due diligence, metrics selection, etc.). We welcome you to join the GIIN in these initial efforts to deepen understanding of the appropriateness of different methods for impact investing.

(click for larger image)

If you or someone you know is interested in contributing to this effort, please reach out to kmccarthy@thegiin.org

For more information about AEA’s SIM TIG, see here. To join the SIM TIG, see here.

The American Evaluation Association is celebrating Social Impact Measurement Week with our colleagues in the Social Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from our SIM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, my name is Sara Olsen and I’m a founding board member of Social Value US and the CEO of SVT Group, an impact management services firm I founded to “make impact management ubiquitous worldwide.” And I’m Kate Ruff. I am an accounting prof at Carleton University in Ottawa, Canada. I study how organizations measure and report their social results (outputs, outcomes impact), focused on accounting problems, such as standard setting, materiality, relevance, presentation and valuation, as they apply to social impact.

Lessons Learned: There has been a lot of talk in impact investing and other circles about the need for “impact measurement standards,” but many if not most efforts to promote a fixed set of measures have met with overt or covert resistance, and failed to gain traction.

Recently we co-authored an article discussing a potential solution to this conundrum. Standardized indicators of impact can only get us so far because they are often insufficiently relevant to the decisions these different parties must make about how to understand and improve their impact. At the same time, having complete discretion over what to measure makes measures incomparable and hinders the development of an efficient social capital marketplace since it becomes virtually impossible to understand or improve impact at the portfolio, sector or systemic level. What’s an impact investor to do?

Drawing lessons from the evolution of financial accounting, we propose that the solution is skilled impact analysts:

“[I]t is possible to achieve comparability by focusing on the analytical skills needed to compare social impacts without mandating a rigid set of required metrics. The premise is that efficient capital markets demand analysts who are capable of interpreting and comparing apples and oranges. Why? Because they understand fruit. The market is best served when each organization can measure its social impact in the way that is most meaningful and insightful to its aim and operations, as long as it follows common principles for good measurement.”

We term this approach “bounded flexibility,” or choice within limits. We assert that it, “coupled with consistency (pick a method and stick to it) and disclosure (fine print gives details on methods used), is the solution to understanding impact that will enable the social capital markets to flourish.

Rad Resources:

For more information about the SIM TIG, see here. To join the SIM TIG, see here.

Social Value International is a professional home where folks from different domains (accounting, finance, philanthropy, evaluation, sociology, environmental economics, data science etc) have come together to work on changing the way society accounts for value. SVI has chapters in the US (newly), Canada and 18 other countries worldwide and members in many more.

The American Evaluation Association is celebrating Social Impact Measurement Week with our colleagues in the Social Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from our SIM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, my name is Donna Loveridge and I work with the Donor Committee for Enterprise Development (DCED), a forum for learning about the most effective ways to create economic opportunities for the poor, in line with the SDGs – based on practical experience in private sector development.

International development organisations, such as DCED members, have traditionally used grants to create economic opportunities for the poor. Now they are increasingly adding impact investing to their repertoire of strategies. But how do impact investors know social and environmental impacts are occurring? Recently, I compared current practices in social impact measurement (SIM) in the impact investing field and results measurement (or M&E) in international development to find the similarities and differences (the full paper can be found here).

Lessons Learned: Here are some insights:

Impact: In international development, the term ‘impact’ has traditionally meant long-term changes (positive, negative, intended, unintended) produced by an intervention directly and indirectly. In impact investing, impact can mean short, medium or long-term changes, including outputs and changes that a business may make to the way it operates, as a means to an end, e.g. more community or employee consultation.

Standardisation: Impact investing is more interested in standardising results measurement approaches and methods to compare investment opportunities and returns than international development. This can be challenging since appropriate approaches and methods are those that consider factors like the investment’s goals; implementation model; location; timescales; and stakeholders such as investor, investee and beneficiaries.

Intermediaries: Currently, impact investing and international development place few expectations on businesses to measure their social and environmental impact – greater onus is placed on intermediaries like fund or programme managers.

Monetary value: Impact investing places greater priority on assessing the monetary value of social impacts than most development programmes, although there has been a strong push towards assessing the value for money of development programmes in recent years by DFID, including using approaches such as social return on investment.

Lessons learned:

Communicate –Take the time to understand what people from the other field mean when they use certain terms or words.

Look to see what is already out there – exchanging views, sharing resources and cross-fertilisation will contribute to the development of SIM and results measurement.

Rad Resources:

More than 100 private sector development programmes use the DCED Standard for results measurement. Check out practical guidelines for results measurement on challenge funds, a type of development programme with some similar attributes to some impact investments, here.

In 2017, the DCED is looking at what information investees value enough to measure and evaluate themselves; and how results (financial, social and environmental) are attributed to different investors. Subscribe to the DCED newsletter here to keep up with developments.

Interested in learning more? For more information about the SIM TIG, see here. To join the SIM TIG, see here.

The American Evaluation Association is celebrating Social Impact Measurement Week with our colleagues in the Social Impact Measurement Topical Interest Group. The contributions all this week to aea365 come from our SIM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.