How to Measure the Performance of Your Outreach Programs

Transcription

1 How to Measure the Performance of Your Outreach Programs April 2006

2 About DeHavilland Associates DeHavilland Associates is a consulting and communications firm that helps its corporate, nonprofit, and association clients understand and connect with the K-12 community. The company designs, manages, and evaluates outreach campaigns and partnership programs for its clients and creates original initiatives to help educators and education stakeholders establish and strengthen dialogue for the benefit of public education. For more information, visit About the Business/Education Partnership Forum Created and managed by DeHavilland Associates, the Business/Education Partnership Forum is an online clearinghouse for anyone interested in learning how to build effective business/education partnerships. This clearinghouse offers news and announcements, links to resources, a directory of organizations involved in business/education partnerships, and a monthly newsletter. For more information, visit About the Effective Education Partnerships Conference The Effective Education Partnerships Conference (EEPC) is an annual two-day meeting for the practitioners of business/education partnerships, including chambers, business coalitions, individual businesses, school and district leaders, and others interested in hearing case studies and effective practices from others in the field. EEPC will be held July 10-11, 2008 in Fairfax, VA. For more information, visit All content 2007, DeHavilland Associates. All rights reserved.

3 Measuring the Performance of Your Outreach Programs A primer on promoting your eduction outreach program through an array of channels Today s schools operate in an era of accountability, grappling with the requirements and consequences of tracking and reporting results. Those who manage education outreach initiatives find themselves under similar pressure: they are increasingly being required to demonstrate how effectively they use resources, and how they provide a return to the sponsoring organization. For education outreach programs, this mandate is a positive step forward. It indicates that education outreach is now being considered to be just as important as any other HR, PR, or corporate relations function and, as a result, is being held to the same performance and accountability standards. This is a key step on the road to an ongoing, if not increasing, investment in such programs. It also means that education outreach programs will create ever-greater impacts on education due to the cycle of continuous improvement that this type of measurement and analysis makes possible. This paper will provide an introduction to campaign tracking and measurement, help you to identify the metrics appropriate to your program model, and provide formulas and commentary for using them effectively. The First Step in Performance Measurement The first and most important step in setting up an effective measurement component comes as the overall campaign is being planned. Measurement must be included as an integrated component from the outset: campaign goals should be determined in part by what can be measured in both quantitative and qualitative terms, and measurement requirements should be integrated into each campaign component (such as collecting addresses from prospective users so they can be surveyed later). Some principles to consider in designing your measurement strategy: Clearly identify each audience you wish to reach. While your education outreach initiative will certainly center on teachers and students, many campaign sponsors also wish to reach other audiences such as parents, customers, the media, legislators and regulators, and employees in order to maximize returns on their efforts. Determine your desired outcomes for each audience. How many members of each audience do you wish to reach? What do you want those people to know, think, or do as a result of your campaign? Next to each desired outcome for each campaign audience, identify the way(s) in which that outcome can and will be measured. You can use quantitative measurements to determine things like media impressions, inquiry rates, contest entry submissions, or volunteer hours logged. Tools such as audience surveys can be used to capture data such as opinions and perceptions, including perceived value and engagement on the part of participants as well as perceptions of the program and sponsoring company on the part of external audiences. Make sure you have systems in place to capture quantitative data, such as number of website visits, inquiries received, or media impressions. Some types of measurements can be impossible to calculate after the fact. Build survey tools you ll use to capture qualitative data such as participants perception of the value and experience of the program. To minimize the potential for survey bias, bring in a qualified and independent reviewer to look over and revise your surveys. And, if at all possible, pre-test your surveys with a sample of your audience to make sure they can clearly understand the questions. Establish benchmarks for any measurements intended to reflect your program s impact. If you want to your program s impact on student knowledge of global warming, for example, you need to determine what they knew before being exposed to the program. Similarly, if you want to measure the effect of your program on your customers perception of your company, you ll need to capture data on their opinions and feelings before your program is launched. Measuring Outreach Performance April

4 The remaining sections of this paper identify key areas you may wish to consider for tracking and measurement, and provide introductory information and some equations to help you build your systems and tools. Acquisition Most campaigns involve an acquisition component, whether that involves recruiting teachers to request materials, students to participate in a contest, or employees or community members to participate with volunteer hours. Determining which tools and channels are the most effective and cost-efficient allows you to use your money wisely and to predict future participation rates. Response rate is a fundamental metric for any mass marketing initiative that requires action on the part of interested audience members. There are several factors that can affect response rate, including list selection, messaging, and creative elements, and by instituting a tracking program and testing various combinations of those elements (a method known as split run testing), you will be able to determine which combination of marketing elements offers you the greatest return. A campaign s response rate does not have to be restricted to a single composite measurement: it can be very helpful to measure response rate by channel, which can help you determine the most productive means of reaching your target audience. Channels such as online marketing, advertising, direct mail, telemarketing, and conference exhibitions allow you to communicate with prospective users in different ways, and you may find that some channels are particularly effective in generating response to your campaign. Of course, response rates only offer a partial picture: the costs associated with each channel, or even with each vehicle within a channel, can vary widely, and those costs must be taken into account when considering the relative value of using a particular channel or vehicle. Determining cost per prospect, particularly when calculated by source, will help you identify the best outreach opportunities for your campaign. And, while cost per prospect by source offers valuable data, don t forget that there are few channels or marketing vehicles that offer total access to the market. For example, online marketing may offer you the lowest cost per prospect: however, you shouldn t necessarily commit your entire outreach budget there, since this channel can only offer you access to a portion of your total audience. A prudent outreach plan takes into account not only response rates and costs, but also the percentage of the market that a channel or vehicle allows you to reach. To fully penetrate the market, you must call on a combination of marketing opportunities. If your campaign incorporates a two-step acquisition process, whereby prospective users request information and then must take an additional step to register or to participate, calculating that conversion rate is essential to generating a thorough analysis. Online targets, for example, may request information at a very high rate but convert poorly, lessening the true value of this channel despite any perceived cost efficiencies. Exposure While acquisition focuses on specific actions taken by targeted audience, exposure is a broader measure of understanding and awareness made possible by your campaign. This may be more appropriate for information campaigns targeted at your primary audience, such as a stay in school in-school advertising initiative, or it may be used in targeting secondary audiences, such as customers or the public. You can use reported media numbers, such as impressions, to gauge exposure among targeted audiences. Impressions are simply the number of readers or viewers reported by the media outlet carrying your message, such as the ratings claimed by television programs or readership statistics highlighted by magazines or newspapers. For most media, these are estimates: you won t be able to definitively state how many people saw your public service announcement during a commercial break. Online media, however, can offer specific data on page views, time spent per page, and more. Experience As teachers, students, and other relevant audiences (parents, volunteers, etc.) work through your program, you ll want to know how they interacted with it, what they thought of it, and what they got out of it. You can capture quantitative and/or qualitative data through methods such as: Focus Groups Involves recruiting a small number of representatives of your target audience(s) and 2 Measuring Outreach Performance April 2006

5 soliciting their thoughts. This is an early-stage tool, best used for exploring market context and understanding which questions to ask in subsequent measurement and evaluation efforts. It should not be used as an evaluation tool in and of itself the data set is too small, and the dynamics of a focus group work against the gathering of reliable data. Observations Involves live observations of the program being implemented with its target audiences. The observer watches the program in action, taking notes and possibly recording the session, and reports to program managers on what he or she has witnessed. To minimize bias, researchers recommend that an independent observer, not a program representative, conduct and report on the observations. Surveys Surveys are one of the mainstays of program evaluation: they allow evaluators to collect a significant amount of data in a consistent format, and can be administered in print, online, or over the phone. Interviews These can be conducted in person or over the phone and used as a primary data collection tool or as a way to supplement and further explore results obtained through other techniques. Examples of this include interviewing teachers after an observation session or calling a group of survey respondents to ask for clarification on unclear responses. Note that these techniques are relevant whether you are determining effectiveness and approach during program development (i.e., formative evaluation) or if you are analyzing the experiences and outcomes of program participants (i.e., summative evaluation). Each of these techniques can also be used to capture both quantitative and qualitative information from program participants, including teachers, students, and others (such as volunteers or parents) involved in the program in some way. Typical measurement and evaluation topics include: Demographics who is participating in the program, and what traits do they have in common, such as grade level, location, technology access, memberships, or interests? What characteristics are shared by their schools or their communities? Reach how many students did each teacher or facilitator engage in the program? Did participants reach parents, community members, or other school attendees with information on the program, by design or through their own initiative? Usage were all of the materials used? If not, why not? Where materials used as intended? If not, how were they used, and what were the results? If they were used as intended, what were the results? User Experience Did teachers find the materials to be clear and easy to use? Where students engaged? Did they have fun? Were volunteers engaged? Did they enjoy their experience? Did all parties find the program relevant to their interests? Value Were the materials valuable to all audiences? Did they help teachers fulfill classroom requirements? Did they help students understand targeted subject matter better than other available materials? Promotion How did participants hear about the program? Program Improvement What suggestions do participants have for improving this program? What would make it more complete, relevant, or engaging? Before deciding on specific questions to ask, be sure you have a clear understanding as to what information you need from participants, and how that information will be used to analyze or improve your program. Cluttering your data gathering tools with questions that generate unused information will make them unnecessarily long and may reduce participation rates. Finally, if at all possible, pre-test your questions with members of your audience to make sure there is no confusion over the language or focus of any of your questions. Outcomes No measurement/evaluation effort would be complete without an analysis of program outcomes. This can include simple summaries of quantitative outcomes number of website visits, number of volunteers signing up to participate as well as changes in both qualitative and quantitative measurements demonstrated by measurements taken before and after program implementation. These numbers will give you a sense of current year accomplishments and can also serve as benchmarks for next year s program. Measuring Outreach Performance April

6 The data collected through all of these types of efforts can provide you with incredible value: not only can you use it to justify your program s continuation, if not expansion, in future budget cycles, but it can also provide you with the information you need to improve future outcomes: reach more teachers, attract more volunteers, change the lives of more students. It is for these reasons that including a measurement and evaluation component within your education outreach efforts may be one of the most important steps you take. Brett Pawlowski is President of DeHavilland Associates, a consulting firm specializing in campaign design and communications and evaluation strategy in the education outreach market. He has spent several years working with organizations on their education outreach strategy and is a recognized voice in the industry. 4 Measuring Outreach Performance April 2006

Community/School Partnerships: A National Survey February 2007 About DeHavilland Associates DeHavilland Associates is a consulting and communications firm that helps its corporate, nonprofit, and association

Introduction Measuring and evaluating results will provide you and your associates with the vital information you need for making key strategic and tactical decisions prior to, at and after the trade shows,

Investigating Effective Lead Generation Techniques by BNET Editorial There is always some degree of turnover in any customer base. To keep a business growing, it is essential to generate new prospective

A Model for Making Sense Out of Marketing ROI Measurements M easuring and improving marketing ROI is a process. It's not a one-shot deal. It's an ongoing effort, an integral part of well-oiled marketing

MARKETING 360: DEVELOPING AN INTEGRATED MARKETING PLAN How a multimedia platform can drive the success of your business If you ve ever heard a good piano player, you know that a solo act can create beautiful

Marketing Strategy and Performance Benchmarks Social Media Strategy Research Summary Report Ascend2 Research Conducted in Partnership with Leading Marketing Solution Providers Social Media Strategy Research

MID-YEAR PREPARATION 10 Simple Steps to Get You Ready for Year-End IMPLEMENT SIMPLE STRATEGIES NOW THAT WILL MAZIMIZE YOUR YEAR-END FUNDRAISING EFFORTS. Did you know that most organizations receive half

TSN SOLUTIONS COMMUNICATIONS MEDIA EDUCATION PRODUCTIONS EVENTS SALES TSN TEAM Thank you for taking the time to read this marketing and communications solutions document. I look forward to bringing our

Communications and Stakeholder Engagement Created in Partnership between the Forum of Regional Associations of Grantmakers and TCC Group Outline of Presentation I. Introduction II. The Value of Communicating

Public Relations, Publicity, and Corporate Advertising PR, Publicity, and Corporate Advertising Tools for communicating information about an organization, its products, and its services in order to benefit

Stakeholder Engagement Working Group Stakeholder Engagement: Planning and Budget Resource December 2015 Purpose: This resource was developed as part of the Society of Practitioners of Health Impact Assessment

SECTION E: COMMUNICATION SCHOOL DISTRICT COMMUNICATION GUIDELINES INDIVIDUAL TRUSTEE SCHOOL STAFF MEMBER PARENT SCHOOL COUNCIL TEACHER PRINCIPAL AREA SUPERINTENDENT CHIEF SUPERINTENDENT BOARD OF TRUSTEES

Marketing & Communications Marketing Plan Messaging Targeted Marketing Media Public Relations Community Engagement What is marketing? Everything you do to recruit and retain students Why communicate? Tell

Using Survey Research to Evaluate Communications Campaigns Prepared for the Communications Consortium Media Center Belden Russonello & Stewart November 2004 Introduction When should a non-profit communications

February 20, 2013 Division of Communications Strategic Plan The Division of Communications is team of talented, dedicated communications professionals who work to tell the story of Eastern Michigan University.

Email Marketing 101 Good Fast or Low cost marketing We ve all heard the old adage about how, when it comes to marketing, you can get what you need done well, delivered quickly or produced inexpensively;

Internship Guide This guide provides a roadmap for any company, no matter the size of the business or the industry. Whether you re just getting started or already have an internship program that you want

Introduction In Understanding Giving: Beliefs & Behaviors of Colorado s Donors, we explored what, how, and why Coloradans give. The report was based on a wealth of research, most of which was conducted

A Primer in Internet Audience Measurement By Bruce Jeffries-Fox Introduction There is a growing trend toward people using the Internet to get their news and to investigate particular issues and organizations.

Institute for Financial Literacy Financial Certification: A Study of the Impact on Professionals' Financial Literacy Levels and Competency A Report Published by the Institute for Financial Literacy, Inc.

SALEM AREA ONE COMMUNITY INITIATIVE Mission Statement To develop a safe, respectful, all-inclusive community through education, advocacy, and coordinated actions that include an effective response to prejudice

2015-2020 IIA Global Strategic Plan 2015-2020 IIA Global Strategic Plan 2015 2020 As used in this document, The IIA Global Strategic Plan, The IIA refers to and is representative and inclusive of the global

This communications and outreach plan template aims to lay out the major issues for consideration in the planning of a communications strategy. You may find all of it useful or only certain elements and

white paper December, 2003 Introduction In early 2003, Democratic presidential hopeful and former Vermont Governor Howard Dean faced a challenge: How could he make the most of a small staff and a limited

Integrating media relations into your BtoB marketing communications plan Media relations, a subset of public relations, should play an important role in your overall marketing communications plan. Utilizing

OUTREACH OPPORTUNITIES WITH PEP and why you can t afford to ignore them OVERVIEW Public Education Programming, or PEP, is a unique program offering nonprofits and government agencies statewide advertising

time the group group What Makes Us Different PRICEPROMISE DIGITALPUBLICATIONS STRATEGICPLANNING We do this work because we love it, and price isn t part of our equation when it comes to quality products.

For Release Upon Delivery 10:00 a.m., April 15, 2008 TESTIMONY OF BARRY R. WIDES DEPUTY COMPTROLLER FOR COMMUNITY AFFAIRS OFFICE OF THE COMPTROLLER OF THE CURRENCY BEFORE THE COMMITTEE ON FINANCIAL SERVICES

There are many activities that can support the funding needs of an organization. The diversity of funding sources can be essential to an organization s funding stability. The following activities can be

Building Confidence by Building Awareness Increasing Public Awareness The Task of Deposit Guarantee Schemes The Bank Deposit Guarantee Fund (Romania) And The International Association of Deposit Insurers

1 of 12 3/27/2008 3:25 PM EXHIBITOR magazine March 2005 MEASUREMENT efore 2003, The Coca-Cola Co. gauged its trade show success on opinions, rather than hard facts. The only measure we had was qualitative

The Role of Advertising in Legal Marketing ALM Webinar Mary K Young January 23, 2008 Presentation Overview Why Should Law Firms Advertising? Necessary Conditions for Successful Law Firm Advertising Role

Department: Department Overview/Narrative APS Department Management Plan School & Community Relations The School and Community Relations Department is responsible for media relations, community outreach,

WHITEPAPER The Truth Behind Web Inquiry Management What Works, What Doesn t and Where You Stand The Truth Behind Web Inquiry Management Challenges of Marketing In the world of sales and marketing, there

BERGENFIELD PUBLIC SCHOOL DISTRICT PUBLIC RELATIONS PLAN 2016-2021 Approved by the Bergenfield Board of Education December 21, 2015 Table of Contents Introduction... 3 Rationale... 4 Public Relations Goals...

Professional Diploma in M a r k e t i n g About managing the doing of marketing for management roles Fully sponsored by the HRD Fund for Banking & Financial Sector employees only Professional Diploma in

Print article Close window Measuring Print Advertising By Bill McBride Over the past 50 years, great progress has been made in developing print communications research techniques that help us understand

Top 6 Strategies to Build Your Marketing Communication Plan Introduction With the plethora of new marketing tools available it can be difficult to know whereto begin. Do you start with a company Facebook

OFFICE OF INSPECTOR GENERAL UNITED STATES POSTAL SERVICE The Postal Service s Use of Social Media Management Advisory August 1, 2013 Report Number August 1, 2013 The Postal Service s Use of Social Media

TOUCHPOINT STRATEGIES MAXIMIZING THE POTENTIAL OF EVERY CONSUMER INTERACTION Executive Summary Those of us in the healthcare industry know the environment is constantly changing. One of the more recent

St. Thomas the Apostle Catholic Church, Naperville, IL INTRODUCTION Since 2005, St. Thomas has experienced rapid growth in the area of external communications technology. Parishioners and members of the

DO YOU YOU HAVE THE Our experts can get you there SEARCH ENGINE OPTIMIZATION SEARCH ENGINE REPUTATION MANAGEMENT EMAIL SOCIAL MEDIA WEBSITE DESIGN Rise to the top of search engine results! We ll help you

School of Journalism & Graphic Communication 2010-2020 Strategic Plan Mission Statement: The mission of the School of Journalism & Graphic Communication (SJGC) is to develop its students to assume meaningful

Producing a Club Newsletter - 6 Steps for VPPR s Today, creating a newsletter is simple and cost-effective. If you have access to a computer and the Internet, you can create a professional-looking newsletter

AUDIENCE PROFILE: The characteristics of the people who make up an audience of an advertising medium in terms of age, sex, region, education, socio-economic group, occupation or any other demographic aspects.

How to Measure Exhibition Success A Workbook for Marketers EXHIBITION AND EVENT ASSOCIATION OF AUSTRALASIA How to Measure 1 discovered it handled it bought it felt it learned it measured it How to Measure

Successful Steps and Simple Ideas to Maximise your Direct Marketing Return On Investment By German Sacristan, X1 Head of Marketing and Customer Experience, UK and author of The Digital & Direct Marketing

How to Measure Exhibition Success A Workbook for Marketers Exhibitions Business doesn t get any more direct than this The measure of success Given the importance of exhibitions in the marketing mix, one

Sample Outline for a Strategic Communication Action Plan The following is an example of an outline for a strategic communication action plan for XXXX County Schools based on the recommendations from an

Online Media Research Peter Diem/Vienna Moscow, April 4 th, 2013 Basics of Practical Online Research 1. Online Research about the Internet In this case the aim of online surveys is to clarify facts about

SUMMER BOOT CAMP APPLICATION Background: The advertising & media industry, a multi-billion dollar international business has long asked the question: Where can diverse talent be found? In response to this

Summary of ESD S Proposal for the NIST s New York MEP Designation The NY MEP Center hopes to continue providing manufacturing extension services to enhance the productivity and technological performance

Attachment B Texas System of Care Social Marketing Plan Overview Texas made an early commitment to the system of care approach and has made steady progress over the past 15 years. In the communities where

Appendix H3 Public Outreach H3.1 Public Outreach Components There are a variety of components that will be needed in development of a comprehensive outreach campaign for the SMMP. Components will include:

Primer on Measuring the ROI of Integrated Marketing and Branding Initiatives Academic Impressions Webinar Elizabeth Scarborough 04 December 2008 How many students did we get because of that billboard weput

1 CHAPTER 7-2/E DEVELOPING AND UTILIZING ELECTRONIC MEDIA Summary In summary, the term electronic media encompasses television, radio and telephone. Television and radio are commonly referred to as broadcast