Join Together Online recently posted a story announcing that the Office of National Drug Control Policy and the National Drug Court Institute released their first "National Report Card on Drug Courts." In the report, drug courts received an "A" for, among other things, reducing recidivism. Having spent over 6 years researching drug courts, I was not surprised by the assessment. But I was surprised by the claim that this was the first report card.
In 1997, the U.S. General Accounting Office (now the Government Accountability Office), the independent investigative arm of Congress responsible for evaluating federal expenditures and programs, published its own report card on drug courts. The GAO did not give A's to the programs they looked at. A metastudy of 20 drug treatment court evaluations produced by 16 jurisdictions discovered that 14 of the 20 studies used no comparison between participants' and nonparticipants' arrest rates after program completion, and the GAO concluded the data did not permit any "definitive conclusions concerning overall impact of drug courts" (GAO, 1997:13).
The GAO recommended steps the Department of Justice (DOJ) could take to improve "the methodological soundness of future federally funded impact evaluations" (GAO, 2002:1). Foremost, it recommended collecting data on participants' criminal recidivism and post-program follow-up data on drug use relapse "to the extent possible."
Five years later, the GAO published a follow up report to assess improvements in data collection for national impact studies. No A's here either. The report was titled clearly enough, Drug Courts: Better DOJ Data Collection and Evaluation Efforts Needed to Measure Impact of Drug Court Programs (GAO, 2002). The GAO summed up their findings: "DOJ continues to lack vital information that the Congress, the public, and other program stakeholders may need to determine the overall impact of federally funded drug court programs and to assess whether drug court programs are an effective use of federal funds" (GAO, 2002:3).
Remarkably, the GAO reported that the Drug Courts Program Office (DCPO), which provides grant funds to and has oversight responsibilities for more than 500 drug courts, had, in 2000, actually eliminated survey questions intended to collect post-program outcome data from the programs to which it provides funds (GAO, 2002:12).
How have things improved so much in the last two years? Maybe they haven't. Most studies of drug courts still fail to utilize the proper control and experimental groups for outcome comparisons. On page 2 the Report Card presents the findings from some of "the most rigorous evaluations conducted among particular drug courts." Every local evaluation listed compares the recidivism rate for an entire control group of offenders not participating in drug treatment courts with drug court graduates -- a mere portion of the entire experimental group of drug court participants. Similarly, the Report Card cites a study by the National Institute of Justice that found a 16.4 percent one year rearrest rate among a nationwide sample of 17,000 drug court graduates. They do not give recidivism data for the rest of the drug court participants.
To follow the report card metaphor, this would be like comparing the entire student population of one school with the honor roll of another and declaring that the second group's higher grades proved that their school did a better job educating them. In some circles this is called creaming, skimming off the very best performers and comparing them to other groups in their entirety, curds and all. In any circle it is bad research methodology, violating the most basic premises of sampling posited in undergraduate textbooks.
In all fairness, this situation is hardly unique to drug treatment courts. Mark Kleiman (2003) recently took researchers to task for the same manipulation of data in an evaluation of a faith-based prison program. In the study, the researchers reported that program graduates had lower recidivism rates than the control group. Politicians jumped on this finding to argue that it proved the program worked. The program group taken as a whole, however, actually had higher recidivism rates than the control group.
And in a meta-analysis of 33 correction-based training programs for adult offenders, Wilson et al. (2000, p. 361-362) found "A full 89 percent of the studies were judged to be of poor methodological quality. Thus, the generally positive findings may result from differential characteristics of the offenders rather than a positive effect of the program activities. That is, these groups could differ before program participation on important variables, such as degree of motivation to make positive life changes."
Importantly, not all drug court studies suffer from these problems, and some good research has found evidence that drug treatment courts can reduce participants' recidivism rates. Studies of adult and juvenile drug courts in Ohio, for instance, have found that participants were less likely to be rearrested than the comparison groups during the follow-up period (Latessa, Listwan, Shaffer, Lowenkamp, & Ratansi 2001; Latessa, Shaffer, & Lowenkamp, 2002). And some of the studies cited in the Report Card do utilize proper methodological rigor. But such well-designed studies are few and far between.
Inadequate research, of course, does not mean that drug courts do not reduce recidivism and drug abuse; only that such outcomes have yet to be demonstrated as irrefutably as advocates and patrons claim. (And these concerns don't even touch on the methodological problems posed by claims of enormous savings from prison diversion when prison and jail populations continue to operate at maximum capacity, incarcerated populations are still growing, and many drug court participants would not have been incarcerated anyway; or the potential for circularity in claims that length of time in treatment is related to treatment success.) Despite their popularity and massive growth, it might be more productive to admit the verdict on drug courts is still out, and in the end no one is served by promotion presented as research.
Kevin Whiteacre lives in Berwyn, IL.
References
Kleiman, M. A. R. (2003). Faith-based fudging: How a Bush-promoted Christian prison program fakes success by massaging data.
Latessa, E. J., Listwan, S. J., Shaffer, D. K., Lowenkamp, C., & Ratansi, S. (2001). Preliminary evaluation of Ohio=s drug court efforts, Center for Criminal Justice Research, Division of Criminal Justice, University of Cincinnati.
Latessa, E. J., Shaffer, D. K., & Lowenkamp, C. (2002). Outcome evaluation of Ohio=s drug court efforts, Center for Criminal Justice Research, Division of Criminal Justice, University of Cincinnati.
United States General Accounting Office (1997). Drug courts: Information on a new approach to address drug-related crime, Briefing report to the Committee on the Judiciary, US Senate, and the Committee on the Judiciary, House of Representatives, May.
United States General Accounting Office (2002). Drug courts: Better DOJ data collection and evaluation efforts needed to measure impact of drug court programs, Report to Congressional Requesters, April.
Wilson, D. B., C. A. Gallagher, & D. L. MacKenzie (2000). A meta-analysis of corrections-based education, vocation, and work programs for adult offenders. Journal of Research in Crime and Delinquency, 37(4)347-368.
