Participants at the March 1997 focus group meeting, reflecting on the state of evaluation
research on drug courts, generally believed that the development and implementation of drug courts had far outpaced research on
their effectiveness. The first drug court programs, developed in the late 1980s and early
1990s, were truly "grassroots" efforts, born out of frustration with "revolving door"
justice and a sense that there had to be a better way to deal with the crush of cases
resulting primarily from a crack cocaine epidemic and stepped-up law enforcement
efforts that focused heavily on arrests for possession of illegal drugs. These programs,
responding to a perceived crisis in the courts, operated on shoestring budgets
with little or no capacity to document their effectiveness with independent evaluations.
Yet the results were readily apparent to observers, and other jurisdictions -- responding to
the same sorts of pressures -- began developing their own drug courts. From only two
drug courts operating in 1990, the number grew to ten in 1992, 41 by 1994, and more than 160 at the start of 1997. As Professor John Goldkamp observed at one of the focus group meetings, the diffusion of ideas about drug
courts has spread far faster than the empirical knowledge of what is happening in the
field.

So great was the need that many jurisdictions embraced drug courts before their
effectiveness could be demonstrated empirically. Yet some drug court practitioners did
seek to quantify what they were seeing in the courtroom and in reports of drug test
results, and began collecting data on a variety of outcome measures. And,
as the concept began to take hold, a few (very few) federally funded evaluation studies
were undertaken. By early 1997, approximately
20 evaluation studies of drug courts -- most of them undertaken locally, often with
minimal resources -- had been completed. Reviewing this body of research, the General
Accounting Office (GAO) found that the drug court programs studied in these 20
evaluations varied widely with respect to eligibility requirements, percentage of eligible
persons who completed the programs, length and nature of the treatment provided,
frequency of drug testing and court appearances, and other variables. Further, the
20 studies themselves varied widely in their objectives, their scope, and the
methodologies they employed. Only 6 of the 20 studies
involved both a comparison group and an assessment of how participant and non-
participant arrest rates compared after program completion. Moreover, many of the drug
court programs covered in the 20 studies were still relatively new at the time the
evaluations were conducted. In light of these limitations, the GAO report, published in
July 1997, concluded that while existing evaluations provided some information about
the courts studied, they did not provide a sufficient basis for drawing firm conclusions
about drug court impact.

The GAO study was particularly concerned with what existing evaluations revealed about six specific issues: (1) criminal profile of drug court participants compared with that of similar offenders processed through the
traditional adjudication system; (2) completion rates of drug court participants; (3) differences in characteristics between drug court
completers and noncompleters; (4) sanctions imposed on persons who failed to complete drug court programs compared with those similarly situated but processed through the traditional criminal justice system; (5) drug use and
criminal recidivism rates of program and nonprogram participants; and (6) costs and benefits of drug courts to the criminal justice system. Participants at the two focus group
meetings were concerned with most of these issues, too. The types of information that
these participants identified as essential for routine data collection and reporting by all
drug courts (see pages 4-5 and 18-19) should provide a foundation for future evaluations
that will address the issues of interest to drug court practitioners and those targeted by
GAO.

Participants at the focus group meetings were also concerned, however, with a range of
evaluation questions -- and with issues about the conduct and funding of evaluations --
that were not central to the GAO study. They addressed the following four main sets of
questions:

1) What is the current state of knowledge about the impact and effectiveness of drug
courts? What do we think we know, and on what basis?

2) Who are the primary audiences for evaluations of drug courts? What are likely to be
the principal concerns of these stakeholders with respect to evaluation?

3) What should be the primary objectives of local-level evaluation research on drug
courts? What should be the main objectives of nationally funded evaluation research?

4) What are the principal obstacles to sound evaluation research on drug courts, and how
can these obstacles be overcome?

A. The Current State of Knowledge About Drug Court Impact and
Effectiveness

The Drug Court Clearinghouse and
Technical Assistance Project at American University, which is funded by DCPO,
periodically publishes information about the activities and performance of drug courts
throughout the Nation. The Clearinghouse's 1997 Summary Assessment of the Drug
Court Experience, based on information compiled from surveys, observations of AU staff
during the course of providing technical assistance to drug court programs, and the
experiences reported by drug court officials, provides an overview of practitioners'
accumulated knowledge and beliefs about drug courts' impact and effectiveness. It points
to a number of areas in which the drug court approach seems to be more effective than
traditional case processing, including:

>Reductions in drug use

>Reductions in recidivism

>Intensive supervision of defendants in pretrial status and of convicted offenders

>Capacity to promptly respond to noncompliance/relapse

>Capacity to integrate drug treatment with other rehabilitation services to promote
long-term recovery

>High retention in the program

>Cost-effectiveness

>Reunited families

>Birth of drug-free babies

>Criminal justice resources freed up to handle violent and other serious cases

>Greater credibility for the criminal justice process

The perceptions that experienced practitioners have about the effectiveness of the drug
court approach are important components of an overall evaluation of their impact.
However, it is important to supplement these perceptions and the drug courts' reports of
their activities with other data -- particularly data that enable comparison of the
performance and outcomes of drug court participants with those of nonparticipants who
have similar relevant characteristics.

B. Audiences for Drug Court Evaluations

Participants at the focus group meeting identified a number of different stakeholders who
are likely to be interested in the operations and impact of a drug court, and who are thus
potential consumers of evaluation research. Some, but far from all, of these stakeholders may also be funders of the drug court program or of an evaluation of it. At the local level, they include:

>Residents and businesses in neighborhoods where drug court participants live

>Residents and businesses in neighborhoods where drug sales and related crime take
place

>The education system

>The business community

>The political or policymaker community, including legislators and other elected officials

>The church community

>Social service providers

>Public health officials

>Substance abuse treatment providers

>Police

>Prosecutors

>Defense attorneys

>Court officials -- judges and court administrators

>Jail officials

>Drug court participants

>The media

While virtually all of these stakeholders will be interested in an evaluation that focuses on certain kinds of hoped-for outcomes
(for example, reduced recidivism), some will be extremely interested in potential
evaluation questions that are of little interest to others. Court administrators, for example,
are likely to be very interested in the impact of drug courts on court caseloads and the use
of courtroom time for drug-related cases, while public health officials will probably be
much more interested in the impact of drug courts on reducing parti-cipants' substance
abuse and improving their overall health. Given the wide range of potential focal points
for drug court evaluations, as well as limited resources and serious methodological
difficulties in addressing some types of questions, choices will have to be made with
respect to evaluation priorities at different levels of government and for different types of
drug court programs.

C. Evaluation Objectives and Priorities

1. Local-level Evaluations.At the local level, the fundamental challenge for an evaluator
is to determine the effects of the specific drug court in terms of the goals that have been
set and the outcome measures agreed upon by the drug court leaders and the local-level
policy group responsible for drug court operations. Often this group includes
representatives of current and prospective sources of funding for the drug court. If not, it
should. One of the primary purposes of an evaluation is to provide a sound basis for
making decisions concerning future financial support for a program.

Simply determining the effects is only part of the challenge, however. Optimally (and assuming sufficient resources and access to relevant data about both program participants and comparison group members),
evaluators may also be able to explain the reasons for particular effects, taking account of
the full range of operational components of the drug court. That is, what is it about a
particular drug court that makes it "work" or fail to work? Given the large number of
operational components in a drug court, answering this question will be a formidable
task. For example, which of the following factors, or what combination
of factors, explains a favorable outcome such as significantly reduced recidivism or reduced substance abuse?

Other potential explanatory factors could be listed, but the point should be clear: There
are a great many reasons why a particular drug court program may work or not work, and
explaining why is difficult and methodologically challenging. From a policy and funding
standpoint, however, answering the "why" question is important. Different approaches to
drug court structure and operations can result in significantly different outcomes and are
also likely to require different types and levels of resources.

Given the limited resources available for local-level evaluations, it is probably asking
too much to expect these evaluators to explain the outcomes they find in terms of all of
the potentially relevant variables. However, it is not asking too much to expect that the
evaluators describe the programs in considerable detail, so that a base of knowledge can be developed about the components of drug court programs that succeed (or fail) on key outcome measures.

As noted above, different stakeholders are likely to have different priorities with respect
to evaluation topics. However, there is a core group of potential "client outcomes" of
drug court programs that participants in the focus group meetings identified as especially
important to assess through evaluation -- preferably by comparing the performance of
drug court participants with a group of nonparticipants who are similar on relevant dimensions, such as current charges, prior criminal histories, nature and severity of substance abuse problems, age, race/ethnicity, and sex. They are:

1) Reductions in recidivism

2) Reductions in substance abuse

3) Improvements in participants' life circumstances, including:

>education level

>acquisition of job skills

>actual employment

>income

>reduction in reliance on welfare

>housing situation

>family situation

>physical health

In addition to outcomes for drug court participants compared with similarly situated
nonparticipants, learning about potential effects of the drug court on the operation of the
criminal justice system and on the communities served by the drug court is a high
priority. Here the comparisons are not of groups of participants, but rather between the
drug court and other systems or between the drug court and the pre-drug court operation
of the system. The focus group noted the following potential outcome variables with
respect to the "system effects" of drug courts in comparison with more traditional
criminal justice processes:

>Faster responses to noncompliance with program conditions

>Greater collaboration between and among the institutions and agencies involved in the drug court's operation

>More interaction of courts and other justice system agencies with community groups

>Greater overall cost-effectiveness

The focus groups discussed the development of a framework for analyzing drug court costs and benefits. They recognized that a great many data elements are potentially relevant to a cost-benefit analysis. There was general
consensus that, important as this type of analysis is, it is extremely difficult to carry out
successfully because of the difficulties in quantifying both costs and cost avoidance.

2. National-level Evaluations.National-level evaluation efforts can and should have a different scope and focus from local-level evaluations. Focus group members welcomed
the news that the National Institute of Justice (NIJ), in collaboration with the DCPO, will
arrange for evaluations of established drug courts. National-level evaluations will be sufficiently funded to address a wide range
of questions and permit the use of appropriate methodologies. The NIJ studies will use
carefully selected comparison groups and will seek to obtain postprogram data on
substance abuse and life circumstances as well as on new arrests and convictions.

As drug courts around the Nation develop the kinds of databases envisioned by the focus group (see pages 3-4), it should be
possible for federal agencies to fund more single-court studies that undertake a comprehensive
evaluation of the drug court program under exami-nation. And, as knowledge is
developed about the components of specific drug courts, it should be possible to undertake meaningful cross-jurisdiction studies that permit the assessment of the relative effectiveness and impacts of different approaches to
implementing drug courts. The objective of such studies is not to compare drug courts with each other but, rather, to learn what approaches are most effective with specific target populations.

It is important for federally funded evaluations to look beyond data on recidivism and
substance abuse and to help develop an understanding of how drug courts are affecting the attitudes and practices of justice system and treatment practitioners. There is a strong sense, particularly among drug court judges, that
many of the principles and techniques used in drug courts -- for example, immediate response to noncompliance, collaborative planning and problem-solving, and creativity in devising a range of responses (rewards as well as sanctions) to participants' performance in the program -- are forerunners of broad changes in court and justice system operations. It is important for national-level evaluators to gain a thorough understanding of the evolution of drug courts and their impact on the full
range of justice system processes. Perhaps most important, national-level evaluations should
address the "why" questions, developing knowledge not only about what type of
approaches are most effective in achieving reduced recidivism, reduced substance abuse,
and other goals, but also about why they are effective.

D. Overcoming the Obstacles to Sound Evaluation Research

There are six major obstacles to sound evaluation research on drug courts. These
obstacles, and suggestions for ways to overcome them, are discussed in the remainder of this section of the report.

1. Difficulty in obtaining relevant data on the behavior of comparison group members and of drug court participants who have completed the program or been terminated for other reasons. The identification of appropriate comparison groups -- consisting of persons who are similar in terms of salient characteristics to the persons admitted to the drug court program, and for whom it will be possible to collect essential data -- is a
key element of sound evaluation. Follow-up data on the behavior of participants who
have completed the program is also very important to gain a sense of the long-term
impact of the drug court. Identifying comparable groups could be difficult but not an
insurmountable obstacle. However, obtaining some kinds of data on the behavior and life
circumstances of comparison group members, as well as on drug court participants who
have graduated or left the program for other reasons -- especially over an extended period
of time -- could be extremely difficult. The 1997 GAO report on drug courts recognizes
these difficulties (Report, page 86) and observes that drug courts ultimately "may have to
be quite innovative in their strategies for collecting such data and may need to resort to
sampling and surveying participants and nonparticipants."

Given the current state of drug court management information systems, obtaining data
even on active drug court program participants can be very difficult. Learning about
criminal behavior of program completers and comparison group members is feasible,
because criminal history data are accessible. However, obtaining data on the substance
abuse behavior and life circumstances of persons over whom the drug court program has
no supervising authority is at best extremely difficult and quite expensive. The March
1997 JMI Working Paper Drug Court Evaluations and Related Research, which was
among the materials considered by the evaluation focus group participants, contains a
number of suggestions about possible sources of such data, including searches of criminal
history records and probation department files, searches of statewide substance abuse
treatment databases, and locating individuals to interview. While these strategies can be
useful for a well-funded study such as those now being conducted by NIJ, they are
currently almost certainly beyond the capacity of most locally funded evaluations. Development of the
capacity to conduct comparison group evaluation studies that use follow-up data on drug
use relapse will require close collaboration of justice system and treatment officials at all
levels of government, as well as major investments in improved management information
systems for drug courts.

One type of comparison that is feasible -- and can be valuable for evaluating the impact
of drug courts -- is a before-and-after comparison of drug court participants themselves.
Drug court programs typically obtain a considerable amount of baseline information on
participants' life circumstances -- including criminal history data and information on
substance abuse, employment, housing, family situation, and physical health -- when participants are first
admitted to the program. It is clearly feasible to examine the extent to which retention in the drug court program is correlated with
changes in participants' criminal behavior, substance abuse patterns, income, employment, and
other life circumstances variables.

2. The Changing Nature of Drug Courts.
Drug courts are "moving targets" for an evaluator. Because the field is evolving so
rapidly and practitioners are constantly learning from their own and others' experiences,
almost every drug court is undergoing changes in variables such as the target population,
the points in the process
at which a person can enter the drug court, the components of the testing regimen and
treatment program, and the personnel involved in the program. For evaluators even to
know what it is they are evaluating, they must have a very good understanding of the
history of the program and how all of its components have developed over time. For
evaluations of programs already in operation, the only ways to acquire this type of
information are to review documents, interview the practitioners and policymakers who
have been involved in the program, and perhaps also interview some of the former and
current drug court participants.

3. Lack of a Cadre of Experienced Researchers. One obstacle to obtaining good evaluations is a paucity of researchers who are familiar with both the criminal justice and substance abuse treatment systems and have the necessary methodological skills. This, however, should not be a major barrier.
Most drug courts are located reasonably close to at least one college or university. It
should be possible to interest professors and graduate students in conducting an
evaluation of the drug court, and there are now enough materials on drug courts to enable
them to educate themselves rapidly on the drug court concept and the key issues that need to be addressed through evaluation. Further, they can learn quickly about the drug court concept and about the goals and operational procedures of the local drug court
by spending time observing drug court and interviewing practitioners and program
participants. Other potential evaluators can be found in government agencies and in nonprofit organizations and for-profit consulting firms. DCPO can help educate novice drug court researchers about key evaluation issues and
problems by bringing them together with researchers and practitioners who have been
grappling with issues in the field for several years. Additionally, the AU Drug Court
Clearinghouse has extensive materials on drug courts that will be helpful to potential
evaluators.

4. Resources for Evaluation.This is a major issue, especially for local-level evaluations. It is not realistic to expect a highly sophisticated, comprehensive evaluation of a drug court
program, using comparison groups and analyzing follow-up data on recidivism and drug
abuse relapse, to be undertaken on a shoestring budget. However, very few drug courts
are in a position to obtain the substantial funding necessary for a comprehensive and
methodically sophisticated multiyear evaluation. It is possible to learn a great deal about
drug court program impact and effectiveness from relatively modest evaluation efforts
that are well focused. Experienced researchers who participated in the focus group
commented that one key to sound and affordable evaluation is ongoing interaction
between the researcher and the drug court policy group, optimally beginning while the
drug court is still in the initial planning stages. With early involvement of the prospective evaluator, it
should be possible to collaboratively shape goals, identify relevant outcome measures,
and devise ways to ensure that the data necessary for the evaluation can be obtained at a
cost that the program can bear.

5. Inadequate Management Information Systems.The state of management information
systems in most jurisdictions is generally inadequate. With a few notable exceptions,
most jurisdictions simply do not have the capacity to maintain, in an automated format,
the kind of detailed client information needed to support drug court management or
evaluation. A closely related problem is the difficulty of linking the drug court
management information that does exist with other data sets that could shed light on the
long-term impact of drug courts -- including data systems dealing with employment
services, welfare, medical services, and substance abuse treatment services. There are
problems both with linking the data electronically and sharing the data, because of regulations that govern the disclosure of personal information. Suggestions for developing improved MIS capabilities are discussed in Section III of this report.

6. Lack of Clarity About Evaluation Priorities. The relationship between local-level and national-level evaluations is not clear. In particular, what should be the priorities for local-level evaluations, and how should evaluation at the local level relate to national-level evaluations funded by NIJ and other
Federal agencies? Particularly in view of the need for local drug court programs to use
limited funds to address the treatment and supervision needs of drug court participants, it
is difficult to justify large expenditures of local funds for expensive though
methodologically sophisticated evaluations. Further, some of the questions that DCPO or
GAO might want to have answered for purposes of shaping national policy with respect
to drug court funding might not be of high priority to local-level policymakers and
funding authorities. These issues were not explored in depth at the focus group meetings,
but there was a general sense that they warrant further attention and that it would be helpful for
local-level drug court leaders to have access to the best thinking about evaluation
priorities and to develop a clear sense of the respective responsibilities of local and national-level evaluators. One way to do this would be to establish an "evaluation steering group" that includes practitioners and researchers
involved in both national and local-level evaluations. Such a group, working with DCPO
and NIJ officials, could help bridge gaps, identify problems, and strengthen
national-local linkages with respect to evaluation. Additionally, as State administrative
offices of courts and State alcohol and drug abuse agencies become increasingly involved
in shaping policy for drug courts, it will be important to ensure that these agencies are
involved in developing evaluation plans.