HIV/AIDS Program Evaluation for Community Based Organizations

Why Should CBOs Evaluate?

Limited funding for social programs means Community Based Organizations (CBOs) are forced to compete for grants. Evaluation is a way to not only be accountable to funders, but also a way to verify program impact in order to obtain funding in the future. Funders, such as public or private foundations, want to ensure that their money is benefitting the community. Program evaluation is a way to prove that your program is making an impact.

An additional benefit of program evaluation is to ensure that vulnerable populations who bear the burden of HIV are receiving the support and services they need. By evaluating your program you are able to easily identify program strengths and weaknesses which will allow for adaptation and improvement of service delivery mechanisms when necessary.

If you would like a comprehensive guide to evaluation we recommend the following resources:

Need assistance in evaluation?

Contact the BSSV program to find out how to receive free capacity building assistance from an expert in the field. Email or call us at (202) 218-3993.

Disclaimer: This site contains HIV prevention messages that may not be appropriate for all audiences. If you are not seeking such information or may be offended by such materials, please exit this website.

Telling Your Story

Being able to tell funders why your program is both unique and effective is a good way to maintain your funding, as well as ask for more. Your organization’s “story” is a narrative telling how your organization is both unique and effective. In your story you might want to include topics such as:

Who you serve?
Eg: Minorities, high risk populations, hard to reach, etc.

Where you work?
Eg: Rural areas, migrant camps, street outreach, etc.

When you provide services?
Eg: After hours, weekends, 24 hour, etc.

Why your organization began?
Eg: No other services, community worked together, a passionate founder, etc.

Below is a fictional example of an HIV CBO’s “Story.” Notice how it brings out unique aspects of the CBO, as well as highlights the work they are doing in the community.

Example: Unique Story for the Women’s Circle Case Study

“At Women’s Circle we are devoted to preventing HIV and supporting those who are positive. It was started 1995 by Linda John, an HIV positive woman, who saw few opportunities for other positive women of color to find support.

Women’s Circle began as a weekly coffee club. We have grown a lot since then! Now we have classes on HIV, like SISTA, 5 support groups, HIV testing, and a group of women who go to the local beauty parlors to do outreach. Although Women’s Circle is in a mostly white, upper class city, most of the women who visit are of color. I think this shows how hard we work to offer things the community wants.

At the office we joke that once a woman comes in, she’s with us for life! Someone will come in for testing or a support group, and keep coming back for the other programs. Women say they feel like Women’s Circle is another home. They love the classes and often bring their friends. I love working for a place that really helps women in our community.”

Each CBO is unique, and each story is unique. Stories can take on many different formats other than the “Women’s Circle” narrative.

To learn more about how to develop your story using logic models and data to influence others, a detailed workbook from the Center for Disease Control and Prevention is available below. This workbook focuses specifically on creating and formatting your program’s unique story. Please note, it has been developed for oral health programs, but the information is equally applicable to HIV programs.

Need assistance in creating a unique story for your program?

Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.

SMART Objectives

Setting objectives is a way for your organization to determine what you would like to achieve and in what time period. Often objectives are broad, but SMART objectives provide a structure which ensures the objective is specific enough to know if it has been reached. SMART is an acronym that stands for:

Specific (What is the specific task?)
Measurable (What are the standards or parameters?)
Achievable (Is the task feasible and easy to put into action?)
Realistic (Are sufficient resources available?)
Time Period Specific (What are the start and end dates?)

SMART objectives will not only set the intentions for your organization, they will also appear in your program logic model.

Below is an example of questions you may ask when developing your program’s SMART Objectives along with an example that follow our “Women’s Circle” example.

Example: SMART Objectives for the Women’s Circle Case Study

Key Component

Questions to Ask Yourself

SMART Objective

Specific - What is the specific task?

What exactly are we going to do in this program?

What strategies will we use?

Is the objective clear?

Is the objective described with strong action verbs such as conduct, develop, build, plan, or execute?

Who will be involved?

Is the outcome specified?

Will this objective lead to the desired results?

Implementation of the evidence based curriculum, SISTA.

Measurable - What are the standards or parameters?

How will we know that change has occurred?

Are we able to gather these measurements?

A total of 400 women who have completed the curriculum. We hope to have 34 sessions with 12 participants in each session.

Achievable - Is the task feasible?

Can it be done in the proposed timeframe?

Are the limitations and constraints understood?

Can we meet this objective with the resources available to the organization?

There is a large demand for the course from our clients and we had 320 women complete the curriculum last year.

Realistic - Are sufficient resources available?

Do we have the resources available to achieve this objective?

Is it possible to achieve this objective?

Yes, we have been funded for the year and we have 3 facilitators who have already completed training for the program.

Need assistance in writing SMART objectives?

Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.

Logic Models

Logic Models illustrate how a program is supposed to work in terms of the logical relationship between resources invested, activities implemented, and changes that result. They help show how your programing activities connect to your SMART objectives and how you will reach your program’s greater goal—such as “Reduce the Incidence of HIV among Women of Color.” Additionally, they are able to effectively summarize your program to a reader/funder more efficiently than a full report or proposal.

Developing a program logic model before planning your evaluation is highly recommended. Logic models allow for all persons involved to see the expected achievements based upon the chain of events that link each element together. Further, they help focus the evaluation plan by identifying what needs to be evaluated, and where to extract the data from along the program implementation process. These elements include:

Inputs - includes all the human, financial, organizational, and community resources available to direct toward doing the work for the program.

Activities - includes the processes, tools, events, technology, and actions that are an intentional part of the program implementation.

Outputs - includes types, levels, and targets of services to be delivered by the program (e.g., what are direct products of program activities).

Outcomes (SMART Objectives) - includes the specific changes in program participants’ behaviors, knowledge, skills, status, and level of functioning as a result of activities.

Impact - includes the intended or unintended change, most often a SMART objective.

By developing a logic model, organizations are able to think about the program in ways that prompt the clarity and specificity required for success, and often demanded by funders and potential partners. A logic models allows for:

An inventory of what organizations have and what they will need to operate the program effectively;

A strong case for how and why the program will produce desired results;

A clear sense of the activities necessary to achieve program success;

A method for program management;

Clear description of the data necessary for program assessment; and,

A SMART objective to work towards.

Below is an example of just one piece of the “Women’s Circle” logic model. To review: first the story of “Women’s Circle” was written to highlight what makes the organization unique and effective. In the logic model the inputs, activities, and outputs that support the SMART objective are tied together.

Example: Logic Model for the Women’s Circle Case Study

Inputs

Activities

Outputs

Outcomes (SMART Objective)

Impact

Program Coordinator

Facilitators (2)

Meeting Space

Referral relationships

Implementation materials

Monitoring & Evaluation Tools

Staff Training

Sessions of SISTA

Collect data from participants

Collect data from facilitators

Graduates from SISTA program

Participant demographics

Participant attendance (from sign-in sheets)

Data on sessions (from facilitators)

Between 2011 and 2012 Women’s Circle will graduate 400 women from the SISTA DEBI.

Decrease new cases of HIV among women of color in “Women’s Circle’s” home city by 2% in 1 year.

For a detailed explanation of how to develop a logic model, along with examples and checklists, refer to the Logic Model Development Guide written by the W.F. Kellogg Foundation.

Resource

Need assistance in creating a logic model for your program?

Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.

Evaluation Plans

A well-developed evaluation plan will provide meaningful feedback about a program’s efforts. It will focus an organization’s resources and efforts on evaluating a program efficiently based on utility, relevance, and practicality and not the collection of useless and boring data that provide no clear conclusions.

Using a logic model to guide the development of the evaluation plan helps determine specific aspects of the evaluation component. A well-developed logic model can help save time during other phases of program implementation and evaluation. A general rule of thumb is to have at least one evaluation point for each outcome and/or impact listed on the logic model. The logic model can help guide evaluation plan development by providing the following:

Focus - helps the evaluation team determine what needs to be evaluated.

Questions - help the evaluation team determine what evaluation questions to ask.

Indicators - help decide which data to collect in order to answer the questions about program efficacy.

Timing - helps decide when to collect data during the program implementation.

Data Collection - helps determine the data sources, methods, instruments, and samples to be used to collect and manage the data.

The chart below provides a portion of the Women’s Circle evaluation plan. The series of charts you complete for each evaluation question is part of the evaluation plan.

Example: Evaluation Plan for the Women’s Circle Case Study

Evaluation Question: Do participants who enroll in Women’s Circle complete the program?

Process Objective

Measure

Data Collection Method/Source

Who will collect the data?

Timeframe for Data Collection

Between 2011 and 2012 Women’s Circle will graduate 400 women from the Women’s Circle Program.

Resource

Need assistance in evaluation?

Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.

Role of Data

Data simply means "pieces of information." It can be numbers, words, sounds, or images. Sign in sheets, intake forms, satisfaction surveys- all of these items can provide useful information for an evaluation. Data makes your story stronger and more convincing. Data is collected through program evaluation.

Why is adding data important?

It gives your audience a better idea of how many people you serve, services you provide, etc.

It gives credibility to your work

It makes your organization look good

It is important to funders and often required (NHM&E requirements for CDC funded CBOs)

Beyond funders’ requirements, there are pieces of data your organization may be specifically interested in. Your program logic model can help determine the specific information you would like gathered to both prove program outcomes and enhance your story. They may overlap with your funder’s requirements. Below we refer back to the “Women’s Circle” story to show the data the CBO feels is important to highlight how they are unique and effective.

Example: Data-infused Unique Story for the Women’s Circle Case Study

“At Women’s Circle we are devoted to preventing HIV and supporting those who are positive. It was started 1995 by Linda John, an HIV positive woman, who saw few opportunities for other positive women of color find support in their struggle.

Women’s Circle began as a weekly coffee club. We have grown a lot since then! Now we have:

Classes on HIV, like SISTA, with over 400 participants each year

5 support groups, that gave 80 women hope in 2010

Tested over 600 women for HIV in the past year, with over 100 testing positive

An outreach team that has visited 30 beauty parlors monthly and distributed over 1,000 female condoms and 2,000 pamphlets about HIV among black women

Although Women’s Circle is in a city that’s 85% white, 95% of our clients are of color. We are really connected to the community!

At the office we joke that once a woman comes in, she’s with us for life! Someone will come in for testing, or a support group, and keep coming back for the other programs. 75% of our clients come back at least 3 times. Women say they feel like Women’s Circle is another home. They love the classes and often bring their friends. Among new clients 45% have been referred by a friend. I love working for a place that really helps women in our community”

Need assistance in evaluation? Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation, or call us at (202) 218-3993.

Data Collection

By following your program’s evaluation plan you will be able to collect relevant and pertinent data for your program. The sources of data and method of data storage is unique to each organization. A data collection plan will help ensure that your organization is collecting data that will help answer your evaluation plan questions. The data collection plan will include information for each SMART objective that lists where the information will be extracted from, who will be in charge of collecting the information, and the timeframe for information collection. As seen in the unique story infused with data, information can strengthen what makes an organization unique and effective.

Creating your Data Collection Plan

Now that you have determined what pieces of data demonstrate how your organization is unique and effective, formatted those data points into SMART objectives, and aligned them with your program’s activities it is time to implement the program and being to collect the data. The chart below will assist you in determining how and when to collect each piece of data. There should be a chart for each SMART objective. The chart below has been filled in with information from the “Women’s Circle” example. The series of charts you have completed for each SMART objective is a data collection plan.

NHM&E

Funders often require you record specific data points. The CDC is an example. The CDC collects specific pieces of data called the National HIV Monitoring and Evaluation variables (NHM&E) from funded community based organizations. Though these specific data are required by the CDC, some of the required variables may be pieces of data you want to include in your story. For example, those who are implementing SISTA are required to report the total number of clients attending the sessions. You can use the data entered into this variable for your story. NOTE: Not every chart will have a NHM&E variable.

Example: Women’s Circle Case Study Data Collection Plan

SMART Objective: Between 2011 and 2012 Women’s Circle will graduate 400 women from the SISTA DEBI

Process Objective

Measure(s)

Data Collection Method(s)/Source

Who will collect the data?

Timeframe for Data Collection

NHM&E variable

Between 2011 and 2012 Women’s Circle will graduate 400 women from the Women’s
Circle Program.

If you are unsure whether or not you are required to collect NHM&E data contact your CDC or health department project officer. To identify the required NHM&E variables for each DEBI, refer to the NHM&E site for the specific DEBI. If you have questions on NHM&E data collection guidance contact the NHM&E Service Center.

Community Data Resources

You may have some data points which are about your community, and not your programs. For example Women’s Circle states “Although Women’s Circle is in a city that’s 85% white, 95% of our clients are of color. ” Below are some sources of community data.

EpiInfo, free data collection and analysis software, Centers for Disease Control and Prevention

Need assistance in evaluation?

Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.

Presenting Your Story

Now that you have determined the story you’d like to tell about your organization and have the data to support that story, it is time to write your story in a way to grab others’ attention. You want to tell your story to people or organizations who can:

Provide more funding

Donate

Volunteer

Change policy

Who Should Hear Your Story?

Who

Why you should tell them

What they care about

Clients

Retention, you care about them

What is going well in the program, data that applies to them

Staff

Improve program delivery, motivate staff, reduce turnover

Staff would like to know if they're doing a good job, or where they can improve

Funders

Keep your current funding or get additional funding

Program is doing what they requested, money is being saved/spent wisely, population is being reached, story is unique