Series Introduction

The increasing emphasis on government accountability at the state and federal levels has increased interest in and use of outcome data. Moreover, research has found that high performing states in integrated employment generally have a clear and visible data collection system that provides individual outcome data (Hall et al, 2007). But what are the most important elements in designing and using a system? Stakeholders have raised questions regarding creating effective data collection systems, identifying variables with the most utility for influencing policy, and using data as a strategic planning tool. This series is intended to shed light on the successes and challenges of collecting data on day and employment services across several states and to provide strategies for other states as they examine their own data collection systems and their impact on their employment priorities for individuals with ID/DD. During the Spring and Summer of 2008, ICI researchers conducted interviews with state and local key informants who had been recommended as being knowledgeable about their state's data collection system. State policy documents and state websites were also used as resources.

Background

In 1994 the statewide employment coordinator at the Bureau of Developmental Services initiated the development of an employment data collection system in New Hampshire. Interested in finding out the number of individuals the bureau served who were employed in the community, the coordinator requested each Area Agency in the state to report the number of people employed and the average number of hours each individual worked. Each Area Agency collected the data for the six-month period between January and June and sent it to the bureau, where over the course of several months state staff entered the information into a large database. As a result of the initial data collection, the bureau learned that more than 900 individuals had jobs in the community and that many of these individuals held more than one job during the data collection period.

Data system basics

Based upon the experience of the first data collection, bureau staff identified preliminary lessons learned about the data collection process. They used the lessons learned as building blocks in the development of a robust data collection system called the Employment Data Survey, or ED Survey.

The design of the data system

New Hampshire is divided into 10 Area Agencies. Area Agencies are responsible for providing day and employment services for individuals who have a developmental disability or acquired brain disorder and who reside in their region. As part of this they are charged with the collection of the ED Survey data from employment providers. In NH, Area Agencies have the option to provide services directly to individuals or to contract with vendors. Regardless of whether the employment services are provided by the Area Agency or a contracted vendor, the employment providers must submit the data to the Area Agency in their region that is assigned to administer the survey, enter the data into the statewide employment database, and export the data to the bureau. The only exception to this is for individuals who direct their own services. For these individuals, the ED Survey is completed by the case managers who monitor their service funds. The case manager submits the information to the Area Agency and the Area Agency enters the data into the employment database. Further data is collected on all jobs in NH, including those in sheltered workshops.

The ED survey is designed to yield reliable information through a standardized system of data collection. The first time employment data is collected on an individual, both the demographic and job-specific sections of the survey must be completed. For subsequent collection periods Area Agencies print out the last employment survey for each individual and asks provider staff to update the data. If there are no changes in an individual's employment status, the forms do not need to be changed. The data reported on hours worked and wages earned is reported as an average for the six month period. Typically the changes that need to be made are in the dollar amounts that people earn, and these changes take only a small amount of time for employment providers to complete. All changes are made by hand directly to the survey form.

After all necessary changes have been made, the employment surveys are returned to the Area Agency to be entered into the statewide employment database. There are typically one or two Area Agency staff members responsible for entering data into the state database. These staff members help to ensure that the employment data collected is of good quality. When employment vendors supply data that looks inaccurate or incomplete they contact the providers for clarification prior to exporting the data to the bureau. One Area Agency reported that typically they need to do very little follow-up to clarify data, because the survey is completed by provider vocational directors and managers who are well acquainted with individuals' employment status.

Area Agencies have 40 days after the end of the data reporting period to export their data to the state. To ensure that the exported data remains confidential, the bureau assigns each individual a code number. Once the exported data is received by the bureau, the ED system is automatically linked to the coded individual database at the bureau.

What data elements are collected

The Employment Data Form has two sections. Section one is focused on demographic information, which is used to create a profile of each individual who has a job. The section is only completed the first time an individual obtains a job but can be updated if there is a change in an individual's profile.

Section two contains the core elements of the Employment Survey Data. If an individual worked at more than one job during the data collection period, a separate version of section two is completed for each job. The second section of the survey has two question styles: short answer and multiple choice.

Please see the Appendix for a detailed overview of the employment variables NH collects.

Who it is collected on

One lesson learned was that many individuals held concurrent jobs during the data collection period. If the bureau wanted to collect data not only on the number of people with jobs but also data about each job placement, they would need
to create a data collection system that could collect data on each job. For this reason employment data is collected on all individuals receiving developmental services who have had paid employment during the given collection period and on each job an individual obtains or maintains during a collection period.

Frequency of data collection

A second lesson learned was that data needs to be collected more than once per year. The data collections and analysis between 1994 and 1996 took about one year to complete. Bureau staff felt that the one year completion time resulted in outdated information and made it difficult to use the data to make policy and administrative decisions. The need for up-to-date data led the state to develop a system that collects data twice per year. One collection period covers employment services provided between January and June, and the second collection period covers employment services provided between July and December.

Standardization across the state

Over time New Hampshire has worked to standardize their data collection system so that all bureau, Area Agency, and employment provider staff working on the ED survey have a shared understanding of the information collected. To ensure the development of a common set of expectations and definitions related to the survey, "Guidelines for the Employment (ED) Survey: Clarification of Some Questions" was developed. This document functions as a frequently asked questions page and answers questions related to who is responsible for completing specific elements of the survey, clarifies the meaning of commonly used integrated employment terminology, and clarifies the intent of specific survey questions.

For example, it was determined that the multiple choice answers provided for the question on work environment needed clarification because the quality of the data collected for this question relied heavily upon a common understanding of integrated employment terminology. The following choices related to work environment were defined in the guidelines document:

Individual job: works alone on the job or with co-workers who do not receive developmental or behavioral services

Job share: two people who receive developmental or behavioral services and who share a job

Enclave: more than two people receiving services together

Sheltered workshop: outside contract work done at the Area Agency or vendor agency. Could involve one or more individuals.

An example of a question whose specific intent needed clarification was, "What is the average number of hours worked per week that there is no paid support?" The guidelines define this question as the average number of hours per week when the individual is working with no job coach on site. They are working independently or with natural supports.

Linked systems

The ED survey is loosely linked to the funding system and overall evaluation of services in NH. The code number assigned to an individual within the ED Survey corresponds to the individual's funding identification number. The use of the same code number allows the state to connect funding with employment outcomes, although the bureau does not regularly run funding outcome reports on employment survey data.

The ED survey results are also used to assess how NH compares to other states and how NH compares to its own past outcomes at any point in time in terms of the number of people who have jobs. The data is also used to evaluate Area Agencies and compare their employment services against one another.

Data sources

The source of the majority of the data for the ED survey comes from the records of employment services providers. This is true whether the Area Agency provides the employment service or the Area Agency contracts withan employment vendor. In the case of individuals who have self-directed services the case manager collects the information from the individual receiving services or from their families.

Spotlight on Data Collection Strategies Used by One Area Agency

One Area Agency staff member described the details of their local data collection process and the strategies they use to ease the burden on providers. Strategies include providing ample time for providers to complete the data, creating checklists, sending reminders, and engaging the providers in the data collection process through the use of humor.

Typically the Area Agency mails the ED Survey forms for each individual to employment providers four weeks before they plan on entering the data into the statewide database. The one-month lead time is especially important to providers who support 50 or more people in jobs.

Another strategy used by this Area Agency to make the data collection process easier for providers is the creation of a checklist of individuals on whom each provider should report data. Area Agency staff attaches a cover letter to the ED Survey listing each individual in alphabetical order and the name of the individuals last reported employer. Providers can then use the cover sheet as a check list to mark off the individuals whose employment situation has changed and therefore whose ED Survey form needs updating.

The use of email reminders to let providers know when to expect the ED Survey forms and how much time remains for them to complete the forms helps to keep providers on track with the data collection. These reminders are especially important because the months in which the data collection occurs, January and June, are typically busy times of the year for providers.

Lastly, this agency used humor to engage the providers in the data collection process. Noting that the employment data collection is commonly referred to as the ED Survey, agency staff uses an image of a horse wearing eyeglasses to call to mind the image of the 1960s television comedy, Mr. Ed.

How the data is used, shared, and analyzed

The information collected by the ED Survey on the number of people employed and their jobs is used to produce reports designed to guide statewide employment policy and practice.

The bureau produces two different reports on the data every six months. The first is a report on all types of employment and the second is a report on all employment except sheltered workshops. Both reports use the same format and report on the same specific six-month period. Data is reported for each of the 10 regions in the state and for the state as a whole. Each question in the survey is analyzed as part of the reports. Traditionally the report has focused on statewide and regional employment outcomes, but recently the bureau has developed the capacity to produce reports that provide a breakdown of the average hours worked and wages earned by individuals. There is also a regional and statewide report on the names and addresses of all the businesses that employed individuals during the six-month period.

The bureau is the only entity that has access to the information collected by all 10 Area Agencies and therefore the only one that can run the statewide report. In the past year and one half all Area Agencies have obtained technology that allows them to analyze data for their agency, but it was unclear how widely Area Agencies were making use of this opportunity.

The ED Survey database is designed to make the process of analyzing the employment data straightforward. The Statewide Employment Coordinator is able to create each report by selecting the specific report she would like to run on the computer, and the computer program is designed to create a standardized report of the information. The number of individuals employed is reported by region. Additionally, data is reported for each region on the number of people employed based upon responses to the demographic, short answer, and multiple choice questions on the survey.

The bureau disseminates each statewide report to bureau employees and the Area Agency directors. Some Area Agencies share the reports with their providers but some do not. Additionally, the ED survey data is shared at the twice-annual statewide conferences on employment. The data is presented in numeric and chart formats. The data is also shared periodically with family support councils. These are family advisory councils to the Area Agencies.

Using the reports at the statewide level

While New Hampshire had not emphasized specific employment outcomes, some providers described the employment outcomes data collection system as an important factor in the state's focus. The reports have allowed NH to see that, while generally over time more people were getting jobs, the rate of job attainment was not keeping pace with the movement of individuals off the waiting list and transitioning from school. The Statewide Employment Coordinator notes that despite supporting approximately 50 percent of the individuals they serve in integrated employment, the reports were a "wake-up call that New Hampshire needed to be doing more to get people jobs." The reports were also described as a catalyst for challenging the bureau to do a better job of expanding opportunities for integrated employment. For example, the reports showed that only about 700 businesses in the state employed people supported by the bureau, despite the fact that New Hampshire has approximately 30,000 businesses. This data suggested the need to expand connections with the business community.

The reports also serve as important sources for information for legislative reports, the design of the state's Medicaid Infrastructure Grant (MIG), and requests for data from national organizations such as the Institute for Community Inclusion's National Survey of Day and Employment Programs.

Using the reports at the Area Agency level

The biannual collection of data has increased the emphasis many Area Agencies place on employment outcomes. It was reported that some use the ED Survey report as a way to motivate employment providers to improve outcomes. A staff person at one Area Agency noted that the first item she looks at is the average rate of pay and the number of people working in each region. One Area Agency administrator noted that she pays close attention to not only the average rate of pay but also job tenure and work environment data to help her assess the quality of the employment placements in her region. Both women said that they wanted to see if their region was producing better employment outcomes than other areas in the state.

For Area Agencies that are involved in the first stages of NH's MIG, there is also a heightened awareness about the importance of the ED Survey and its connection to efforts to improve employment outcomes. In these regions there is a recognized need to identify strategies to more closely link provider outcomes to the regions' data report.

Other Area Agencies use the longitudinal database as a source of information on individuals' employment histories and to customize individuals' resumes based upon previous employment experiences.

Using the reports at the local level

Contacted members of the provider community have spoken positively about the data collection and genuinely appreciated that the bureau returned the data to their organization. Comments such as, "[The data] certainly had an impact on us because... without the data, we would not have been able to say whether [we were improving]," expressed providers' appreciation for the feedback that came through regular contact about the data.

An analysis of the system: Successes and challenges

What's working well

Stakeholders identified factors that help the ED Survey process to operate smoothly. These factors include: strong connections between bureau, Area Agency, and provider staff; regular collection of data increases Area Agencies' interest in the data reports; and the ease of report development.

Strong connections between bureau, Area Agency, and provider staff are one factor that helps the ED Survey process to operate smoothly. One example of this is in the response to providers' questions that are not addressed in the ED Survey guidelines. If Area Agency staff is unable to answer a question for the provider they are able to contact bureau staff. Bureau staff is noted for responding to questions in a timely manner, which in turn helps to ensure that the data collection process is completed in the allotted time. Another example of how strong connections benefit the data collection process is that one Area Agency staffer felt that she needed to conduct minimal follow-up with providers because of her relationships with providers.

Bureau staff felt that the level of interest that Area Agencies show in the data reports was related to the regular collection of data. Staff noted that many Area Agencies are so interested to see the results of the ED Survey that they will contact the bureau office and inquire as to when the next report will be released.

Lastly bureau staff felt that the ED Survey process operated smoothly because the system of data reporting and report generation is so ingrained in practice that staff now spends about one week per year creating the data reports. This is in contrast to the years when the system was being developed and refined when considerable amounts of bureau time and resources were dedicated to instituting the ED Survey.

Challenges within the system

NH has faced several challenges in the development of the ED Survey. Some of those challenges occurred during the initial development and refinement stages of the survey, while others are current issues impacting the system.

Early challenges

One of the greatest challenges faced by bureau staff during the development and refinement stages of the survey was the state's culture of local control. Traditionally, Area Agencies operated with limited oversight from the bureau with the expectation that they use the freedom they had been granted to develop innovative supports. Given this history, some Area Agencies initially resisted participation in the ED Survey because it was developed by the state offices with minimal input at the local level.

Bureau staff discussed their awareness that, by instituting the ED Survey, they were working outside the culture of local control but felt strongly that, in order for a reliable system of data collection to be developed and coordinated across the state, the bureau needed to have full authority over the system. The bureau periodically meets with Area Agencies and providers to refine the survey and discuss the data, which has resulted in a close collaboration on this data system. Over time Area Agencies have responded positively to the ED Survey, and it has become embedded in state practice.

One important outcome of the bureau having full responsibility for the ED Survey was standardization of the data collection process. When the data collection process first started, Area Agencies needed support from the bureau to make sure the data they were supplying was accurate. If the bureau had not had the final authority for the survey data it would have been much more difficult to guarantee that the data being reported by Area Agencies was comparable. Specific ways in which the bureau helped to ensure the accuracy of the data was responding to Area Agency questions and examining the information to ensure that responses made sense within the context of the state's employment system. Another example of why it was important for the bureau to have full control over the development and implementation of the survey was an early discrepancy that arose between Area Agencies' data. Several Area Agencies initially wanted to submit data on individualized volunteer positions, while others did not want to include this data in the survey. However, because the focus of the data collection is on employment, the bureau made the final decision that only data on paid employment be included in the survey.

Over time, as the details of the data collection system became engrained in practice, multiple stakeholders, including employment providers and Area Agencies, have helped to refine the ED Survey. This kind of joint effort at improving the system has added to a sense of ownership of the data across stakeholder groups.

Current challenges at the bureau level

Despite the importance that the bureau places on employment outcomes, at times the employment data gets less attention due to competing staff responsibilities. At the time of these interviews there was not a bureau staff member solely devoted to employment outcomes and data, but a plan was being developed to hire a statewide employment specialist to work directly with Area Agencies, including on their employment outcome data.

Another challenge faced at the bureau level is that they are not able to use the ED Survey as a method to evaluate providers, only Area Agencies. Changes to the survey to allow the bureau to use the survey results to measure the quality of provider employment outcomes could lead to new information about innovations in employment supports at the local level.

Current challenges at the Area Agency level

An administrator at one Area Agency noted that within their local Area Agency, staff wanted to produce consistently better employment outcomes and suggested producing another version of the ED Survey report that focused on longitudinal data across the state and within each Area Agency, as opposed to the current report format that highlights a specific point in time. The administrator also noted that a longitudinal report by provider would be helpful. Longitudinal reports would allow the Area Agency to more closely measure the region's progress against other Area Agencies as well as the progress of individual employment providers.

Another challenge Area Agency staff identified was the difficulty in using ED Survey data to measure service quality. The ED Survey report was noted for providing Area Agencies with a lot of information but that it wasn't necessarily usable at the Area Agency level. However, the usability of the report as a point-in-time measurement of outcomes and the reliance on presenting the information in table format without written analysis of what the results mean make the report less practical at the local level. It was suggested that the development of a report focusing solely on outcome measures would allow Area Agencies to more closely link outcomes to service quality.

Current challenges at the local level

Challenges exist for both providers and individuals and their families at the local level. Individuals and families are not currently able to access the ED Survey reports, but the state has discussed putting the reports online so that individuals and their families can view the information.

One provider noted that they would like to see the data collection move to an online format. Noting that while most of the changes are simple to make, sometimes staff is challenged by missing forms. When a form is missing they need to create a new one for an individual. Also, because forms are filled out for each job an individual holds, if the person lost their job in between data collection periods, providers need to use the old form as a termination form and then create new forms for each new job the person has. Moving to an online system would eliminate the need to create new survey forms.

The provider also indicated that they would like to see an online system that does not necessitate the creation of a new form for each job an individual has, but uses one form with space to detail each job. Lastly, while her Area Agency shares the ED report with the provider's director, the information is not automatically shared by the Area Agency with the staff that completes the ED Survey.

Moving forward: Areas for future development

Future plans to revise the ED system include the development of reports that link service funding to employment outcomes. Additionally, there is a plan to post the data online.

Lessons learned and implications for other states

NH has been collecting employment data for many years. They reported that one lesson learned was that it takes time to develop a good data collection system. Bureau staff would advise states who were developing a data collection system that they should plan to revise the system on an ongoing basis until the system works well for all stakeholders.

Other suggestions for ensuring the development of a reliable data collection system include:

Understand your state agency's culture and develop your system around this parameter

Make data a priority at the state, regional, and local level

Set goals for employment outcomes and design a data collection system to measure the employment outcomes valued by your state

Include individuals who are knowledgeable about your state's integrated employment services in the design of your data system

Consider the development of a secure online system to reduce the burden on those responsible for reporting data

Conclusion

New Hampshire's ED Survey has been in existence for nearly 15 years and has matured to meet the state's data collection needs during this time. The survey has become ingrained in state practice and has allowed NH to effectively measure employment outcomes at the state and Area Agency levels. While the system is well developed, stakeholders at all levels have a continued desire to refine the ED Survey to ensure that it continues to meet the state's evolving needs.

Acknowledgments

The authors would like to thank the stakeholders in New Hampshire for their contributions to and review of this brief.

This project was supported, in part, by cooperative agreement #90DN0216, from the Administration on Developmental Disabilities, Administration for Children and Families, US Department of Health and Human Services. The opinions contained herein are those of the grantee and project participants and do not necessarily reflect those of the Administration on Developmental Disabilities.

For more information please contact:

Jean Winsor Research Associate Institute for Community Inclusion University of Massachusetts Boston jean.winsor@umb.edu 703.542.6799

Appendix

Demographic Data

Data
Requested

Choices

Individual's
Name

NA

Date of
Birth

NA

State
ID

NA

Area Agency

NA

Gender

Female, Male

Funding
Level

NA

Medical,
physical, and/or behavioral challenges that impact a person's ability
to work? Choose all that apply

Copyright 1992-. ICI is
affiliated with the University of Massachusetts Boston and Boston Children's Hospital . Permission to use, copy, and distribute all documents
on this site, in whole or in part, for non-commercial use and without
fee, is hereby granted provided that appropriate credit to the University of Massachusetts Boston be included in all copies. Alternate formats are available upon request.