To link to the entire object, paste this link in email, IM or documentTo embed the entire object, paste this HTML in websiteTo link to this page, paste this link in email, IM or documentTo embed this page, paste this HTML in website

Fiscal Research Division
Fiscal Brief
Executive Summary
Education
TTeenn Quuees ss t tt i iioonns ss t ttoo BBeet tt t tteer rr PPi ii l lloot tt PPr rrooggr rraamms ss
August 8, 2008
Changes needed in pilot program design to produce clear, useful results.
The majority of pilot programs in North Carolina have
failed to produce clear evidence of success or failure.
This has made it difficult for members to determine
whether or not to expand or discontinue the programs.
The General Assembly has expressed a strong desire
to receive clear, objective evaluations of new
programs. However, most pilot programs have been
designed in ways that make quality evaluation
impossible.
The goal of this memo is to help policymakers avoid
the pitfalls that have undermined past pilot programs.
Policymakers should ask the following ten questions
to ensure that new pilot programs will be able to
provide clear results:
1. What is the problem that needs solving?
2. How does the program address the identified
problem?
3. What is the cost of the program if it is
successful?
4. Is there a budget or spending plan?
5. What criteria will be used to determine the
program’s success or failure?
6. What alternative programs/ solutions might
also address the problem?
7. Does the design of the program allow for
meaningful evaluation?
8. Are there problems in the program design that
will affect validity?
9. Is there sufficient time to observe effects?
10. Are there enough units of study to ensure
statistical significance?
With clearer results, policymakers will better be able
to determine which programs work and which
programs do not.
Introduction
Pilot programs are new initiatives implemented on a
limited basis as a test or trial. Ideally, the small- scale
pilot program will provide data showing whether or
not the new program has potential to succeed on a
larger scale, or whether it should be discontinued.
The State of North Carolina has demonstrated an
admirable willingness to try out new initiatives by
funding new pilot programs. Unfortunately,
policymakers have learned little from these efforts.
North Carolina’s pilot programs have generally
included provisions and funding for program
assessment. Unfortunately, these pilot program
assessments have often provided ambiguous results,
making decisions on program expansion difficult for
policymakers. The primary reason is that the pilot
programs themselves have been designed in ways that
inadvertently preclude meaningful assessment.
Common problems with pilot programs include:
• Unclear goals – what does it mean for a
program to “ work”?
• Unclear criteria – what measurements will be
used to determine if a program is successful?
• No control group – results of the program are
not compared against an independent group
not affected by the pilot program;
• Selection bias problems – sites that are in the
program are systematically different than
those that are not;
• An inadequate timeframe in which to observe
outcomes – some pilot programs have been
discontinued before results can be observed;
• An inadequate number of pilot sites – the
number of sites is insufficient to produce
meaningful data.
The goal of this memo is to help policymakers avoid
the pitfalls that have undermined past pilot programs.

Fiscal Research Division
Fiscal Brief
Executive Summary
Education
TTeenn Quuees ss t tt i iioonns ss t ttoo BBeet tt t tteer rr PPi ii l lloot tt PPr rrooggr rraamms ss
August 8, 2008
Changes needed in pilot program design to produce clear, useful results.
The majority of pilot programs in North Carolina have
failed to produce clear evidence of success or failure.
This has made it difficult for members to determine
whether or not to expand or discontinue the programs.
The General Assembly has expressed a strong desire
to receive clear, objective evaluations of new
programs. However, most pilot programs have been
designed in ways that make quality evaluation
impossible.
The goal of this memo is to help policymakers avoid
the pitfalls that have undermined past pilot programs.
Policymakers should ask the following ten questions
to ensure that new pilot programs will be able to
provide clear results:
1. What is the problem that needs solving?
2. How does the program address the identified
problem?
3. What is the cost of the program if it is
successful?
4. Is there a budget or spending plan?
5. What criteria will be used to determine the
program’s success or failure?
6. What alternative programs/ solutions might
also address the problem?
7. Does the design of the program allow for
meaningful evaluation?
8. Are there problems in the program design that
will affect validity?
9. Is there sufficient time to observe effects?
10. Are there enough units of study to ensure
statistical significance?
With clearer results, policymakers will better be able
to determine which programs work and which
programs do not.
Introduction
Pilot programs are new initiatives implemented on a
limited basis as a test or trial. Ideally, the small- scale
pilot program will provide data showing whether or
not the new program has potential to succeed on a
larger scale, or whether it should be discontinued.
The State of North Carolina has demonstrated an
admirable willingness to try out new initiatives by
funding new pilot programs. Unfortunately,
policymakers have learned little from these efforts.
North Carolina’s pilot programs have generally
included provisions and funding for program
assessment. Unfortunately, these pilot program
assessments have often provided ambiguous results,
making decisions on program expansion difficult for
policymakers. The primary reason is that the pilot
programs themselves have been designed in ways that
inadvertently preclude meaningful assessment.
Common problems with pilot programs include:
• Unclear goals – what does it mean for a
program to “ work”?
• Unclear criteria – what measurements will be
used to determine if a program is successful?
• No control group – results of the program are
not compared against an independent group
not affected by the pilot program;
• Selection bias problems – sites that are in the
program are systematically different than
those that are not;
• An inadequate timeframe in which to observe
outcomes – some pilot programs have been
discontinued before results can be observed;
• An inadequate number of pilot sites – the
number of sites is insufficient to produce
meaningful data.
The goal of this memo is to help policymakers avoid
the pitfalls that have undermined past pilot programs.