Page tags

Add a new page

Prioritizing Outcomes and Indicators to Measure

No project can evaluate all possible results, nor would it want to. Outcomes cover the spectrum of more and less important, more or less feasible, more or less desirable, and many other dimensions. For most projects the trick is to choose a modest number of outcomes that have meaning for stakeholders, provide representative information about the extent to which the project is achieving its aims, and can be measured reasonably robustly with the resources available, using strategies that are acceptable when considering participant time, quality of experience, and privacy.

Remember that for many programs, the best you may be able to achieve in a limited period of performance is to show that a reasonable number of participants established a skill, knowledge, attitude, or behavior that, if sustained over time and extended to enough people, will result in a desired impact in the long term.

Here are some of the considerations to hold outcomes against when trying to choose among the many possible outcomes and their priority for evaluation. You will want to consider all of them to some extent, but some may not apply to your project, or may cancel others out. Of course every program or project has special considerations that create exceptions, but these are useful dimensions to consider:

Priority Consideration

Example

Significance: Very important vs. not very important to important stakeholders. This can apply to participants, program/project staff, funder, or others. Subtle, small-scale changes may be a lower priority than clear, concretely observable changes.

A funder may care more about whether children increase their reading skill measurably than about whether they have fun doing it (even though we all know the two may be closely connected.) Parents may care more whether a child chooses healthy snacks than about whether he or she can put foods into the right category in the USDA food pyramid.

Importance to institutional mission or program purpose: Programs or projects usually produce a range of outcomes, some of which are key to achieving the organizations’ fundamental purposes and some of which are not. Outcomes that clearly and directly support purpose at the institutional or program level are usually higher priority than fringe benefits.

A community reads project is usually designed to encourage reading and discussion of books. It may be less valuable to know how many participants found an associated field trip a good venue to socialize with friends.

Stakeholder importance: Different stakeholders have greater claims on a project’s activities and services, including evaluation. Decision- and policy-makers have higher priority than groups with casual interest. Funders, governing bodies, participants, key partner institutions, and staff responsible for managing or delivering a program fall on the higher-priority side of the equation. In most cases competitors, media, staff of unrelated programs, and organizations peripherally connected to your project or institution fall on the lower-priority side.

It’s probably more valuable to address the information wants of your director and your project grant maker than it is to address the special interest of your evaluation consultant.

Timetable: When an outcome is expected to occur. It may be more desirable to measure immediate and intermediate changes to show short-term results, even when large-scale, long-term impact is what the program hopes to produce. Since outcomes measurement is in part about continuous improvement, it may be high priority to evaluate an interim result that can tell you the extent to which you’re making progress, and lower priority to evaluate an outcome that will only be apparent when the whole program (or product) is complete.

Strengthening democracy may be the end goal, but it’s very tough for most projects to show, say, a change in individual voting patterns. On the other hand, it may be entirely feasible to ask how many program participants over 21 voted in the last local election, and how many intend to vote in the next one (or to register, or to work on a political campaign).

Sequence: Closely related to timetable—when an outcome is expected in relation to other outcomes. Often one outcome depends on another. Here, too, immediate and intermediate outcomes may have high priority for showing if the project design or implementation needs change

A change in knowledge may change an attitude, and both may be needed before behavior changes. For instance, someone may need to know overweight is related to diabetes, and that exercise and diet contribute to weight control before he or she decides to make a lifestyle change; that change needs to precede weight loss. Your program may only be able to achieve the first in the time you have with participants.

Measurability: Some outcomes are very hard to measure directly; their inference may depend on proxies or formal research.

Increased creativity may be tough to show as a generic change. It may be more important and more feasible to show that participants are willing (or more willing) to try new media or to experiment, or that they plan to explore new ways of solving context-related problems.

Likely frequency of occurrence: If the choice is between a strong outcome that will probably happen for a very small percentage of participants or users, and a good outcome that will probably happen for many of them, the more frequent outcome is probably the best choice for measurement from a PR standpoint.

A program might have as its intended outcome significant improvements in the English-language level of a group of non- or minimal English speakers. Consider if it will be in the program’s best interest to show what proportion of participants feel two or three increments more confident in trying to speak English, or what percentage join an English-language conversation group, or how many participate once a week in some other activity that supports them in learning English.

Achievability: Some programs have genuinely visionary hopes, but are practically limited by the difficulty of the vision or the starting point of their target audience. Consider whether you are likely to be able to show the desired result in the time and with the resources available to your project or program.

Ending homelessness is a magnificent goal. Given all the factors that contribute to homelessness and the scale of the problem, consider if it would make your program and reporting stronger to choose a manageable outcome such as participants’ routine use of temporary shelter, or their transition from emergency housing to stable housing.

Feasibility: Do you have, or are you likely to be able to get, the various resources to measure an outcome? Those resources include time, skill, funds, access to participants, permissions, reliable instruments, and many other things that can impact your ability to get good evaluation data for a desired outcome.

It may be highly desirable to measure changes in reading skills using a standardized test, but an individual non-school program may not have the necessary working relationship with participants’ schools; may not have permission from parents to access individual student results or to administer standardized reading tests; or may feel reading tests are inappropriate in context. Then the priority for measurement becomes an outcome that can reasonably stand for reading skills—how often a child reads, or how many books in a fixed period, or how knowledgeable a child is about finding reading material that meets their needs.

Cost-effectiveness of available measures: Can you afford to measure the outcome? If not, choose something more immediate, more likely, more representative of the average participant’s experience, smaller scale, or easier to evaluate.

It might be most desirable to interview every participant in depth to understand the extent to which each individual’s attitude towards the importance of diversity. It might be wonderful to know what percentage of your community has a specific knowledge or attitude related to your program. Can you afford the independent interviewer, transcription, coding and analysis of interviews? Can you afford and generate a good response rate to a reliable community-wide sample for a survey? If not, consider measuring outcomes in a way that’s congruent with your project’s practical resources.

Increase vs. desired benchmark: If you don’t have and can’t establish a measurable condition that precedes your program, you can’t show increase. Consider whether it will be just as useful to know if participants reached a specific level of skill, knowledge, etc.

Your program or project goal may be to made a difference in the information literacy of, say, college freshman. You may not have a source for the level of information literacy of freshman coming into your learning process, and you may not want to spend precious time testing to establish a baseline. It may be sufficient to know that after your process, freshman can distinguish reliably between high- and low-quality Web sites, can use more than one search strategy efficiently, or can find a challenging item of information within a reasonable time limit.