Truth and D.A.R.E.: Is D.A.R.E.’s new Keepin’ it REAL curriculum suitable for American nationwide implementation?

Abstract

The study investigates the effectiveness and appropriateness of the Drug Abuse Resistance Education (D.A.R.E.) Keepin’ it REAL (KiR) curriculum – America’s most prevalent in-school prevention programme – using a modified systematic review procedure as adapted from the Cochrane Handbook. No published studies research the D.A.R.E. version of KiR, and so we consider the Keepin’ it REAL intervention as a whole. After reviewing the abstracts of 677 studies matching relevant keyword searches, 11 studies matched inclusion criteria (e.g. testing effectiveness on substance use). The systematic review yields mixed results for the effectiveness of the Keepin’ it REAL intervention. Concerns remain regarding the appropriateness of the KiR D.A.R.E. programme: (1) KiR has only been tested on a narrow audience and may not be appropriate for D.A.R.E.’s larger audience, (2) KiR may not be effective in reducing substance use among elementary school students and (3) the specific versions of KiR implemented by D.A.R.E. (KiR D.A.R.E. and KiR D.A.R.E. Elementary) have yet to be tested for efficacy. The authors recommend independent, randomised trials for the KiR D.A.R.E. curriculum and the development of a standardised measure and evaluation system for in-school substance use prevention programmes.

Keywords:

• Keepin’ it REAL
• D.A.R.E.
• NREPP
• systematic review
• effectiveness measures

Introduction

Los Angeles Police Department Chief Daryl Gates founded the Drug Abuse Resistance Education (D.A.R.E.) programme in 1983 in an effort to combat drug use and gang violence in Los Angeles. The programme involved having a uniformed police officer come into elementary schools and instruct students on the dangers of underage drinking and drug use. D.A.R.E. grew quickly, promoted by its administrative organisation, D.A.R.E. America. By 2003, D.A.R.E. was instituted in 80% of United States school districts (Armentano, 2003), making it, by far, the most commonly used youth drug prevention programme in the country. D.A.R.E. was and is synonymous with in-school drug prevention programmes.

The viral programme attracted attention not only from community and school leaders but also from academics eager to test the success of the D.A.R.E. programme. While certain studies supported the programme (e.g. Faine & Bohlander, 1988), the vast majority of scientific studies on D.A.R.E. condemned the programme as ineffective (e.g. Clayton, Cattarello, & Johnstone, 1996; Clayton, Cattarello, & Walden, 1991; Dukes, Stein, & Ullman, 1997; Ringwalt, Ennett, & Holt, 1991; Rosenbaum & Hanson, 1998). Many concluded that D.A.R.E. spread because of its word-of-mouth marketing and popularity rather than its effectiveness.

Administrators of the D.A.R.E. programme during its early years were resistant to change and criticism. As more evaluation studies came out against D.A.R.E. throughout the late 1980s and early 1990s, D.A.R.E. proceeded unabated D.A.R.E. America’s unwillingness to change its programme or to formally evaluate its programme fuelled concerns, controversy and scepticism among evidence-focussed academics and policy workers (Rosenbaum, 2007).

Beginning in the early 2000s, D.A.R.E. made significant steps to meet scientific standards for efficacy. In 2001, D.A.R.E. America won a grant from the Robert Wood Johnson Foundation to develop and evaluate a new middle-school drug prevention programme called “Take Charge of Your Life” (RWJF, 2009). This first attempt to base the D.A.R.E. programme in science was unsuccessful; a longitudinal test funded by that grant found that students who participated in TCYL actually increased subsequent drug and alcohol use (Sloboda et al., 2009).

Admitting that the TCYL programme was unsuccessful, D.A.R.E. changed course. Instead of developing its own programme, in 2009, D.A.R.E. partnered with the makers of Keepin’ it R.E.A.L. (KiR), a culturally sensitive prevention programme developed by researchers at the Pennsylvania State University in the late 1980s and early 1990s (Hecht, Colby, & Miller-Day, 2010), to create a version of KiR called Keepin’ it REAL D.A.R.E. (Nordrum, 2014). KiR had already been studied by prevention experts and was recognised as an evidence-based programme in the Substance Abuse and Mental Health Administration’s National Registry for Evidence Based Programs and Practices (NREPP; “Intervention Summary”, 2006). D.A.R.E. initially used KiR for its middle school programming only. By 2012, D.A.R.E. America had adapted an elementary school version, Keepin’ it REAL D.A.R.E. Elementary, which it used for its elementary school programming. Consequently, D.A.R.E. syndicated KiR into its expansive network spanning approximately 75% of American school districts, making KiR the most widely disseminated prevention programme in the United States.

To understand the KiR intervention in the context of this study, it is necessary to understand how over the years, the KiR developers published several versions and adaptations of their programme. The original KiR programme, which was declared “evidence-based”, was intended for middle school students and included three versions: Latino, White/Black and multicultural. The programme consisted of “10 lessons promoting antidrug norms and teaching resistance and other social skills” using a “culturally based narrative and performance framework” (Hecht et al., 2003), and is based on four resistance strategies: Refuse, Explain, Avoid and Leave. The KiR research team developed two adaptations to expand its programme to the 5th grade: KiR-Plus and KiR-Acculturation Enhanced (KiR-AE). These adaptations represented significant changes to the programme, making the programme developmentally appropriate, and including the introduction of new lessons and changing messaging to match the audience (for a full description of the adaptation, see Harthun, Dustman, Reeves, Marsiglia, & Hecht, 2009). When D.A.R.E. adopted KiR in 2009, the KiR D.A.R.E. programme was formed as a version of the KiR middle school interventions (note: the KiR D.A.R.E. version is implemented by uniformed police officers, whereas other versions are implemented by teachers and other school professionals). In 2012, KiR and D.A.R.E. created a new version, KiR D.A.R.E. Elementary, for D.A.R.E. elementary school students (5th and 6th grade students).

While the inclusion of KiR as part of D.A.R.E. made it by definition “evidence-based,” several questions remain on its effectiveness and the appropriateness of its place as the most prevalent in-school prevention programme in the United States.

Research aims

Thus, the purpose of this review is to evaluate the available effectiveness evidence for the D.A.R.E. Keepin’ it REAL curriculum with regard to substance use outcomes (e.g. decreased 30-day recent substance use) in the populations on which it is implemented and to determine if there is adequate available evidence to justify the widespread implementation of the Keepin’ it REAL D.A.R.E. programme. It is hoped that the results of this review may be used by policy makers to determine whether D.A.R.E. has adequate scientific backing to merit its “evidence-based” designation and by the scientific community to identify research and evaluation methods for the D.A.R.E. KiR Curriculum going forward.

Methodology

The systematic review process, as adapted from the Cochrane Handbook (Higgins & Green, 2008), was used to synthesise the data from current literature on the effects of the Keepin’ it REAL intervention on substance use outcomes. Peer-reviewed studies, published after 1990, that analysed the effectiveness of the Keepin’ it REAL intervention in reducing substance use among adolescents were included in the systematic review.

Research databases ERIC, PsycInfo, PubMed and Google Scholar were searched for the keywords: “Keepin’ it REAL”, “D.A.R.E.”, “Drug Abuse Resistance Education” and the name of KiR’s developers and sorted for results later than 1990. Studies were also pulled from the SAMHSA-NREPP listing of the KiR intervention. A total of 677 abstracts were reviewed. Out of the search results, 267 studies had abstracts that were deemed to be relevant to our focus (e.g. studied either the D.A.R.E. or Keepin’ it REAL programme). Of those 267 studies, 130 focussed on older D.A.R.E. interventions and were excluded. Two studies measured the effectiveness of auxiliary programme elements, such as branded media campaigns and were excluded; 124 of the remaining studies did not evaluate the effectiveness of a relevant programme in terms of reducing adolescent substance use. For example, several studies described how adaptations were made or described dissemination practices of the programme but did not address the effectiveness of the programme. Only 11 of the reviewed studies considered the effects on substance use of the Keepin’ it REAL intervention, thereby meeting the inclusion criteria, and were included in the systematic review. Data from included studies were extracted and analysed qualitatively for the strength and quality of evidence supporting or refuting the Keepin’ it REAL intervention. The findings of the present study are a detailed examination and synthesis of the 11 included studies.

It is important to note that the number of variations and adaptations of KiR makes it difficult to research succinctly. Specifically, no published studies exist (as of April 2016) on either the KiR D.A.R.E. or KiR D.A.R.E. Elementary programmes. Therefore, we review all effectiveness studies related to all KiR curriculums, with the caution that programme variations may have different effects. In the Results section, we will focus on KiR middle school interventions, on which KiR D.A.R.E. and KiR D.A.R.E. Elementary were based, as well as elementary school adaptations, as they are best connected with our research aim.

We initially wanted to analyse the intervention quantitatively with an averaged effect size, but as we reviewed the studies, it became clear that a quantitative analysis would be inappropriate. Different versions of KiR are not identical, and differences in the versions of KiR are not easily quantifiable. In this review, we are trying to understand the effectiveness of a programme (KiR D.A.R.E.) by reviewing the observed effects of tested interventions that are similar, but may differ on certain dimensions (e.g. target age, demographics, delivery venue). Taking a simple average of the effect sizes for all KiR versions may give undue weight to versions that are more dissimilar from KiR D.A.R.E. than others, and the ending result may be misleading. Instead, qualitative review is better suited for comparison of programme versions that are similar but are not identical. The reasons for a qualitative rather than quantitative review will be discussed further in the Discussion section.

Results

“Table 1: Summary of Included Studies” represents a summary of the included studies.

Table 1. Summary of included studies.

Study title	Authors and date	Study design	Statistical test(s)	Results/findings of substance use outcomes
Culturally grounded substance use prevention: An evaluation of the Keepin’ it REAL curriculum*	Hecht et al. (2003)	6035 7th grade students randomly placed into four interventions: treatment as usual (TAU), KiR-Mexican/Mexican-American, KiR-Black/White, and KiR-Multicultural	Adjusted mean difference in effect sizes at 2, 8, and 14 months post-intervention	Mexican/Mexican American version showed significant results for all three waves (ES = −0.091, SE = 0.041; ES = −0.127, SE = 0.048; ES = −0.168, SE = 0.064), while the Black/White and Multicultural versions only significant result at 14-month wave.
Mexican/Mexican American adolescents and Keepin’ it REAL: An evidence-based, substance abuse prevention program*	Kulis et al. (2005)	Analysis of 2003 study’s data set for Mexican/Mexican-American students	Adjusted mean difference in effect size 14 months post-intervention	Latino (Mexican/Mexican-American) and multicultural version had statistically significant results (ES = −0.17, SE = 0.06; ES = −0.15, SE = 0.06). Black/White version not significant
The Drug Resistance Strategies intervention: Program effects on substance use*	Hecht et al. (2006)	Analysis of data set from 2003 study using growth models	Two different growth models	Mexican/Mexican-American and Multicultural versions slowed growth of substance use, while Black/White version did not
Promoting reduced and discontinued substance use among adolescent substance users: Effectiveness of a universal prevention program*	Kulis et al. (2007)	Analysis on subpopulation of participants in 2003 study that had used substances before baseline assessment	Multiple regression and chi-square analysis	The researchers did not distinguish between versions of the intervention. The program significantly increased reduced use and discontinuation rates
Differences by Gender, Ethnicity, and Acculturation in the Efficacy of the Keepin’ it REAL Model Prevention Program	Kulis et al. (2007)	Multilevel or hierarchical analysis of 2003 data set using pre-test/post-test method	Pre-test/post-test using fixed effects for school and controlling for baseline covariates	Greater effect among less linguistically acculturated Latinos, compared to insignificant results among more linguistically acculturated Latinos and Non-Latino Whites
Immediate and Short-Term Effects of the 5th Grade Version of the Keepin’ it REAL Substance Use Prevention Intervention	Hecht et al. (2009)	Randomized trial of 1566 students in multicultural 5th grade adaptation of the KiR program or TAU and evaluated in pre- and post-tests on substance use norms, expectancies, resistance strategies, and intentions	Random coefficients model with fixed effects including a linear trend component and an interaction between condition and linear time	Significant effects for use of resistance strategies (p < 0.001), but no significant difference of lifetime (p = 0.80) or recent (p = 0.10) substance use. These results represent effects at Wave 3, or the end of 6th grade
When to Intervene: Elementary School, Middle School, or Both? Effects of Keepin’ it REAL on Substance Use Trajectories in Mexican Heritage Youth	Marsiglia et al. (2011)	Randomized trial (from same trial as 2008 study) involving 1670 Mexican heritage students (84% of the total sample). Participants received KiR-Multicultural or KiR-AE in either 5th grade, 7th grade, both, or neither (control)	Six-wave growth model, following students until 8th grade, to explain use trajectories	No benefit from receiving the intervention only in 5th grade; only a significant effect if intervention in 5th and 7th grade. Intervention in 5th and 7th grade was as effective as the intervention only in 7th grade for inhalants and marijuana and less effective for other substance use outcomes
Effects of the 5th and 7th Grade Enhanced Versions of the Keepin’ it REAL Substance Use Prevention Curriculum	Elek et al. (2010)	Randomized trial of 1984 elementary school students (same trial as 2008 study) placed in either the KiR-AE or KiR-Plus intervention or TAU at 5th, 7th or 5th and 7th grade	Mean differences in effect size	Neither the KiR-AE or KiR-Plus interventions jad significant positive results. 5th grade KiR intervention were found to be significantly harmful
The Effectiveness of Adapted Versions of an Evidence-Based Prevention Program in Reducing Alcohol Use among Alternative School Students+	Hopson and Steiker (2010)	Trial of 70 students in adaptations of KiR in alternative schools, tested before, after, and 6-week post	Repeated measures MANOVA using nonparametric Friedman’s analyses	Multivariate analysis showed a significant difference between the two groups’ alcohol use and intentions, with F(4, 120) = 3.52 (p = 0.02)
Short-term effects on substance use of the Keepin’ it RERAL pilot prevention program: Linguistically adapted for youth in Jalisco, Mexico	Marsiglia et al. (2014)	Random control trial of linguistically adapted version of KiR in Guadalajara, Mexico, and participants tested for recent substance use before, after, and 8-month post	A series of t-tests and ordinary least squares to determine program effects	Reduced the frequency of alcohol (ES = 0.22, SE = −0.09) and cigarette use (ES = −0.18, SE = 0.09) and showed some indication (p < 0.10) of reducing the amount of cigarettes consumed (ES = −0.15, SE = 0.08)
Evidence for Site-Specific, Systematic Adaptation of Substance Prevention Curriculum with High Risk Youth in Community and Alternative School Settings+	Holleran Steiker et al. (2014)	73 participants in community and alternative school settings randomly assigned to KiR adaptation, original KiR, or control and tested for substance use	MANOVA analyses across the three different interventions for beer, wine, liquor, and marijuana use	Adapted KiR intervention significantly more effective than both the original KiR intervention and the control at reducing beer use (ES = 0.11) and wine use (ES = 0.11)
Long-Term Effects of the Keepin’ it REAL Model Program in Mexico: Substance Use Trajectories of Guadalajara Middle School Students	Marsiglia et al. (2015)	431 8th grade students (10 classrooms) from two public schools in Guadalajara, given Mantente REAL (translated into Spanish) or a control condition and tested for substance use before, after, and 8-month post	Three wave growth models	The KiR program retarded growth of alcohol use frequency for females and growth of marijuana use for males. The intervention did not slow cigarette use growth

*Study is referenced in SAMHSA’s NREPP as evidence that KiR is evidence-based.

+Independent study, i.e. not co-authored by one of KiR’s original developers.

American middle school studies

The first large-scale randomised trial of the Keepin’ it REAL middle school intervention occurred in the late 1990s and early 2000s, and the results were first published in 2003 (Hecht et al., 2003). The study, which was authored by a research team including KiR developers Drs. Marsiglia and Hecht, involved 35 urban middle schools from the Phoenix Arizona area. The participant schools were block randomised and assigned to one of four different interventions: KiR-Mexican/Mexican American, KiR-Black/White, KiR-multicultural and control or treatment as usual (TAU). A total of 6035 middle school participants were enrolled in the study and filled out at least one of the questionnaires. Of the 6035 participants, 55% were Mexican or Mexican-American, 19% were other Latino, 17% were White Non-Hispanic and 9% were African-American, and 82% qualified for federal lunch assistance. The interventions were conducted in schools in 1998–1999, and the Keepin’ it REAL interventions were supported by public service announcements (PSAs) outside of school; 57% of the experimental students saw one or more KiR PSAs, whereas only 30% of the control students saw one or more KiR PSA. Among adolescents, alcohol and drug use tends to increase with increases in age, and so substance use prevention programmes tend not to report absolute percentages of substance use before and after the intervention. Instead, to compare the effect of the programme with the control group, the researchers reported the results of the study in terms of mean difference in effect size between intervention and control after accounting for differences at baseline.

The students were asked to fill out four questionnaires concerning their recent alcohol, tobacco, marijuana and the overall substance use (use within the past 30 days), on which the study’s results are based. The questionnaires were filled out at four waves: (1) before the intervention, (2) two months after the intervention, (3) eight months after the intervention and (4) fourteen months after the intervention.

The researchers found that participating in the Mexican/Mexican–American intervention was related to a statistically significant mean difference in effect size from the control in recent substance use during the first 2-months post-intervention (−0.091 SE = 0.041), during the first 8-months post-intervention (ES = −0.127 SE = 0.048) and throughout the 14-months after intervention (ES = −0.168 SE = 0.064). The other two versions of the KiR programme only showed significant results for recent substance use in the 14-month post-intervention questionnaire (Black/White: ES = −0.149 SE = 0.063; Multicultural: ES = −0.159 SE = 0.052). In terms of specific substances, the intervention showed the greatest effect size for recent use of alcohol (statistically significant at the 2- and 14-month post-test for all three versions) and less consistently significant for recent use of cigarettes and marijuana.

The authors ran statistical analyses of the effectiveness of culturally matched programmes (i.e. Hispanic version among Hispanic students, Black/White version among Black/White students), but the authors’ analysis yielded insufficient evidence that culturally matched programmes were more effective than non-culturally matched programmes. The authors did not report the quantitative results of that analysis.

Using the same data set from the 2003 study, the researchers published a 2005 report (Kulis et al., 2005) regarding the effect of the KiR intervention on the sub-population of students who were of Mexican descent, as the programme had been developed for a Hispanic audience. Those sub-study participants were 3402 students of Mexican heritage from 35 urban, Phoenix-area middle schools.

Again, the study presented adjusted mean differences in substance use outcomes from the 14-month post-intervention questionnaire (results from the 2-month post-questionnaire and 8-month post-questionnaire were not addressed). The results indicated that both the Latino and Multicultural version of the intervention had significant effects on relevant substance use outcomes in that sub-population from the original study. The Latino version showed a statistically significant mean difference in recent substance use (ES = −0.17, SE = 0.06) and recent marijuana use (ES = −0.24, SE = 0.09). The multicultural version showed significant mean differences in recent substance use (ES = −0.15, SE = 0.06), recent alcohol use (ES = −0.24, SE = 0.09) and recent marijuana use (ES = −0.16, SE = 0.08). None of the three versions had a statistically significant adjusted mean difference in effect size for recent tobacco use.

The Non-Latino version, which was identified in the 2003 study as the White/Black version, showed no statistically significant mean differences among these primarily Hispanic students in recent substance use, recent alcohol use, recent cigarette use or recent marijuana use. It might be expected that the Black/White version would not be effective with a primarily Hispanic audience and could be considered inferential evidence of the need for different versions. While the 2005 study broke down effectiveness of the different interventions among the Hispanic population, the authors did not describe the effectiveness of the different interventions among the Black/White or other ethnicity students, despite a sufficient population of Black/White students (n = 1576) to perform powerful statistical analyses.

Despite using the same data set, the results of the 2005 study (Kulis et al., 2005) contradict with the conclusions of the 2003 study, which stated that culturally matched programmes were not more effective than non-culturally matched programmes. The 2005 study reports that at the 14-month post-test, the Hispanic and multicultural versions were more effective among Hispanic students than the Black/White intervention (among Hispanic students in the sample, Latino: −0.20, SE = 0.08, Multicultural: −0.15, SE = 0.06 and Black/White: −0.16, SE = 0.10). This apparent contradiction has not been addressed.

In 2006, the researchers published another paper (Hecht, Graham, & Elek, 2006) using the data set from the 2003 study. While the 2003 study reported 6035 middle school participants, the 2006 study of the same original population reported 6298 middle school participants. The reason for this inconsistency is not explained. In the 2006 study, the researchers employed two growth models to statistically analyse the effects of the programme. The first model considered outcomes from the 2-, 8- and 14-month post-test, while the second model considered outcomes from all measures (pre-test and 2, 8 and 14 month post-tests). Both models showed that the Mexican/Mexican–American version and the multicultural version significantly slowed the growth of recent drug use. The Black/White version, however, did not show any statistically significant results among the predominantly Hispanic student population.

In the 2007 study (Kulis, Nieri, Yabiku, Stromwall, & Marsiglia, 2007), these same researchers again analysed the original 2003 data set focussing on the sub-population of middle school students of all races and ethnicities who had used substances at pre-test (n = 1364). Based on the nature of the population, the researchers published results for reduced substance use and discontinuation of use. The report did not specify whether the participants in the experimental group went through the Mexican/Mexican–American, White/Black or Multicultural versions, making it difficult to draw conclusions about the effectiveness of specific versions of the intervention. The results showed significant reductions in alcohol use compared to the control group at the 2-month post-test (ES = 0.54, SE = 0.26), in alcohol abstinence (ES = 0.51, SE = 0.26) and in abstinence from all substances (ES = 0.47 with SE = 0.23). There were no statistically significant findings for reduced or discontinued use of tobacco or marijuana.

The 2007 study (Kulis et al., 2007) again reviewed the 2003 data set to examine gender differences in the programme effects of KiR among non-linguistically acculturated Hispanics, linguistically acculturated Hispanics and non-Latino Whites. The researchers hypothesised that there would be greater observable differences among males and females in the non-linguistically acculturated group because of strong cultural norms against female substance abuse. Indeed, the authors found that the effects of the programme were more pronounced in males than females among non-linguistically acculturated. However, there was no difference in effect across genders among linguistically acculturated Hispanics or non-Latino Whites.

Across the five studies that analysed the 2003 data set, which measured the effect of the KiR middle school intervention, there is little evidence that the Black/White version was effective. The Mexican/Mexican–American and Multicultural versions were effective among a primarily Hispanic audience. All studies were conducted by a similar team of authors, including the programme developers, Drs. Hecht and Marsiglia.

Elementary school adaptations

In the late 2000s, the KiR team tested the efficacy of an expanded programme that could be implemented in 5th grade involving two new adaptations of KiR: KiR-Plus and KiR-Acculturation Enhanced (KiR-AE). Three included studies relate to this elementary school version and are based on the same trial: 2009 (Hecht et al., 2009), 2011 (Marsiglia, Kulis, Yabiku, Nieri, & Coleman, 2011), and 2010 (Elek, Wagstaff, & Hecht, 2010). Overall, the trial involved 1984 5th and 7th graders from lower-income primarily Hispanic 29 elementary and K-8 schools. Each school received either the control (TAU) or one of six experimental treatments: KiR-AE 5th grade, KIR-AE 7th grade, KiR-AE 5th and 7th grade, KiR-Plus 5th grade, KiR-Plus 7th grade, or KiR-Plus 5th and 7th grade. Substance use was measured at six waves, following the students from 5th grade to 8th grade. The 2009 study (Hecht et al., 2009) published the short-term results (up to Wave 3, or the end of 6th grade) of the 5th grade intervention, and found that while the treatment significantly increased the number of REAL refusal strategies employed, it did not decrease either recent or lifetime substance use rates. The authors followed up with the 2011 (Marsiglia et al., 2011) study when the six wave data set was complete. The authors found that receiving the 5th grade intervention alone did not significantly decrease substance use. Further, receiving both the 5th grade and 7th grade treatments was only approximately as effective as receiving the 7th grade intervention alone for marijuana and inhalants, but worse for tobacco and alcohol. The authors published the 2011 study (Elek et al., 2010), which studied mean differences in effect sizes at the 8th grade level. Virtually none of the experimental groups showed a significantly greater reduction than the control group (difference in effect size), indicating that the programme was either not effective on those substance use measures, or perhaps that the study was under-powered to detect true differences.

Mexican middle school adaptations

In the late 2000s, the creators of the KiR programme adapted the programme linguistically to be tested in middle schools in Guadalajara, Mexico. The programme was tested in two public middle schools in Mexico, involving 431 students, and substance use outcomes were measured in three waves: pre-test, immediate post-test and 8-months-post-test. The 2014 (Marsiglia et al., 2014) study uses a series of t-tests and ordinary least-squares regresions on the pre-test and immediate-post-test to determine programmatic effects from the linguistically adapted version of KiR. The researchers found that the programme reduced the frequency that alcohol (ES = −0.22, SE = 0.09) and cigarettes (ES = −0.18, ES = 0.09) were used and the amount of cigarettes used (ES = −0.15, SE = 0.08). The 2015 study (Marsiglia, Kulis, Booth, Nuño-Gutierrez, & Robbins, 2015) reported on the same data set but with access to all three waves. The researchers reported that the KiR programme retarded the growth of alcohol use frequency for females and marijuana use for males, with no effect on tobacco use.

Alternative school adaptations

The only independent trials of the KiR programme (i.e. trials not co-authored by KiR developers; denoted in Table 1 with a plus sign) were conducted with student populations from Texas alternative schools. The first study in an alternative school setting (Hopson & Steiker, 2010) used an adapted version of the Keepin’ it REAL curriculum appropriate for alternative high school students. The research was conducted in four Texas alternative schools: two disciplinary schools (otherwise known as last chance schools) and two Popular Innovation Schools. Participants in the study were older than other trials (average age 16) and predominantly Hispanic (49%). The schools were block randomised and two schools received the adapted KiR intervention while the other two served as controls. The researchers found a significant difference in use rates and alcohol-related norms among students in the adapted KiR intervention versus the control. However, the study was limited by a small sample size (70 participants at pre-test, continued by attrition down to 41 at 6-months-post-test).

In 2014 (Holleran Steiker, Hopson, Goldbach, & Robinson, 2014), the team of researchers recruited 73 participants from Texas alternative school settings, including a juvenile justice day programme, a homeless shelter, four alternative high schools, low-income housing programmes, and LGBTQ youth centre and a youth group on the Texas–Mexico border. The participants were predominantly Hispanic (54%), majority female (56%), and high school age (average age 16.2). The researchers adapted the KiR middle school curriculum to match their site-specific needs and placed participants into one of three interventions: Adapted KiR Group, Original KiR group and a control group (Total N = 73 including: Adapted KiR N = 20, Original KiR N = 20, Control N = 33). Beer, Wine, Liquor and Marijuana use were measured at pre-test, post-test and 6-months-post-test. The researchers found with a MANCOVA that the Adapted KiR group was more effective than either the control or the Original KiR treatment; however, the small sample size makes inference difficult. The F-Test for “Group × Time” was significant at the 0.05 alpha level for Beer (ES = 0.11) and Wine Use (ES = 0.11) and significant at the 0.10 alpha for Liquor (ES = 0.09) and Marijuana use (ES = 0.09).

While these alternative school trials provide additional evidence regarding the effectiveness of the KiR programme, their applicability to our research aim of studying KiR D.A.R.E. is limited. These studies have a relatively small sample size, utilise an alternative school setting as opposed to a traditional school setting and target significantly older students than D.A.R.E. (high school versus 5th–7th grade).

Discussion

Synthesis

These studies are limited in their ability to affirm or refute KiR D.A.R.E.’s efficacy and yield mixed results regarding the overall effectiveness of KiR, broadly defined, in reducing substance use among students. Just three large trials have been performed on the KiR programme, each on a different adaptation or version of the programme (American middle school, American elementary school, Mexican middle school). Two small-scale trials have been tested in Texas alternative school settings, though their scope is limited by the small sample size and idiosyncratic adaptations made to the curriculum. The trials that involved American students both were all implemented with a predominantly Hispanic, urban, low-income population in the American Southwest and Texas. The programme was developed specifically for a Hispanic audience (Gosin, Marsiglia, & Hecht, 2003; Hecht et al., 2003), and there is evidence that a Hispanic adaptation of KiR is effective with a predominantly Hispanic audience, as expected. However, just one study showed that the Black/White version is effective with this predominantly Hispanic audience. No research exists on whether the programme can be successful with a predominantly African American or predominantly White audience.

The research also finds that a KiR adaptation for elementary school students was unsuccessful at reducing substance use. Implementation of the KiR programme in 5th grade alone was ineffective, and implementation of the KiR programme in 5th and 7th grade was not even as effective as implementation of the KiR programme in 7th grade, alone. There is evidence that the programme can be adapted for a Mexican audience, though it is unclear whether this has any translation to an American audience (which is the aim of this paper).

KiR D.A.R.E. or KiR D.A.R.E. Elementary interventions have never been directly tested, and so a discussion of how the KiR D.A.R.E. interventions differ from tested KiR interventions is warranted, especially given that variations of the programme show different levels of success. The authors did not have access to copies of the different adaptations of KiR, and so we cannot definitively describe the magnitude of changes between versions, i.e. the magnitude of differences between KiR’s original, multicultural middle school intervention, which showed efficacy in a randomised control trial, and KiR D.A.R.E. or KiR D.A.R.E. Elementary. For evidence of how much KiR has changed in its creation of a D.A.R.E. version, we can only turn to secondary sources. In a commentary on health message dissemination (Hecht et al., 2010), KiR developer Hecht and his team state that KiR and D.A.R.E. America worked together to “DARE-ify” the programme and making it appropriate “for [D.A.R.E. police] officer implementation”. However, the commentary provides little available information on how those resulting changes may have affected the effectiveness of the original KiR intervention – or for which subgroups of students. Lesson topics, a short description (fewer than 10 words) of each lesson’s main theme, have been presented in the literature (Burnett, 2013) and on the Keepin’ it REAL website (http://www.kir.psu.edu), and so we can observe differences in lesson topic among versions of the programme (see Table 2). The lesson topics for the KiR multicultural middle school intervention and the KiR D.A.R.E. intervention are identical. However, there are significant differences between the lesson topics for the KiR multicultural middle school intervention and KiR D.A.R.E. Elementary. Without access to the curriculum, it is difficult to conclude the usefulness of comparing lesson topics among KiR versions (clearly, there are vastly different ways to cover a topic as broadly defined as “Avoid” or “Risks”). We also note that D.A.R.E. America has stated that its modifications constitute a new version of KiR called Keepin’ it REAL D.A.R.E., which may imply that significant changes were made. For these reasons, it seems at least possible that the KiR D.A.R.E. intervention differs substantially from KiR interventions that have been rigorously tested. Given that previous adaptations have differed enough from the core middle school KiR programme to render the intervention ineffective, policy makers should be cautious if not sceptical of the claim that KiR D.A.R.E. is, itself, an evidence-based programme. The variability of effectiveness among KiR versions further concerns that KiR D.A.R.E. and KiR D.A.R.E. may not be appropriate for nationwide implementation.

Table 2. Comparison of lesson topics among KiR versions.

Intervention	KiR original multicultural middle school intervention	KiR D.A.R.E. (middle school)	KiR D.A.R.E. elementary
Source	KiR Website	Burnett (2013)	Burnett (2013)
Lesson 1 Topic	Options and Choices	Options and Choices	Introduction to D.A.R.E.’s Keepin’ it REAL program
Lesson 2 Topic	Risks	Risks	Drug Information for Responsible Decision Making
Lesson 3 Topic	Communication and Conflict	Communication and Conflict	Risks and Consequences
Lesson 4 Topic	Refuse	Refuse	Peer Pressure
Lesson 5 Topic	Explain	Explain	Dealing with Stressful Situations
Lesson 6 Topic	Avoid	Avoid	Basics of Communication
Lesson 7 Topic	Leave	Leave	Nonverbal Communication and Listening
Lesson 8 Topic	Norms	Norms	Bullying
Lesson 9 Topic	Feelings	Feelings	Helping Others
Lesson 10 Topic	Support Network	Support Network	Getting Help from Others and Review

Part of the systematic review process is to identify possible biases in the research base. One potential source of bias is the status of the authors. Nine of the 11 studies that fit the inclusion criteria and most studies on KiR that did not meet the inclusion criteria are co-authored by KiR developers Dr. Michael Hecht and/or Dr. Flavio Marsiglia (the independent trial is indicated with a footnote). Therefore, many of the evaluations of the efficacy of this programme cannot be considered independent evaluations.

The analysis of these studies also raises concerns about the widespread use of SAMHSA’s NREPP, which has been discussed in the literature in the past (Gandhi, Murphy-Graham, Petrosino, Chrismer, & Weiss, 2007). Four studies (as indicated with an asterisk) reviewed in the systematic review were used by SAMHSA to justify KiR’s standing as an evidence-based programme in the NREPP in 2006. KiR’s standing as a SAMHSA NREPP evidence-based programme likely served as an impetus for D.A.R.E.’s adoption of KiR in 2009, and D.A.R.E. actively advertises itself to schools and communities as an evidence-based programme based on its NREPP evidence-based designation. However, KiR’s “evidence-based” designation is based upon its effectiveness as a middle school intervention, and D.A.R.E. serves primarily an elementary school audience. Not only are the studies inconclusive about the effectiveness of all versions of KiR (the Mexican/Mexican-American and Multicultural versions show more evidence basis than the Black/White version), but also none of the studies indicate that the programme will be effective for an elementary school (5th and 6th grade) audience. In fact, three studies showed that previous adaptations of the KiR programme for elementary school audiences (KiR-Plus and KiR-AE) had insignificant or even counterproductive results in elementary school (5th grade) participants. The failure of previous KiR adaptations in elementary schools motivates scepticism of the effectiveness of the KiR D.A.R.E. programme in elementary schools. The validity of this concern can be tested once KiR D.A.R.E. Elementary is studied in randomised trials.

These results raise concerns about the appropriateness of KiR generally and particularly its various versions, given that this programme continues to be the most commonly used substance use prevention programme in the United States.

Limitations

Limitations of the Studies Reviewed. The only published studies on the KiR intervention use a narrow demographic population (urban, predominantly-Hispanic, low-income students). It is difficult to conclude the generalisability of results without having results of studies performed on other populations. All but one of the available studies for review derived from a single experiment and all involved the same experimental team. There is need for more studies on a broader range of subject populations and by a wider range of investigators. To reduce risk of reporting bias, future research should standardise the measures and statistical test(s) used in their studies.

Recommendations for future research

Most immediately, the field should look to scientifically study the KiR D.A.R.E. and KiR D.A.R.E. Elementary interventions in randomised trials performed by independent researchers for various populations. Only through randomised trials performed by independent researchers can, we confirm that the KiR D.A.R.E. and KiR D.A.R.E. Elementary programmes are effective for the national audience that D.A.R.E. serves. D.A.R.E. reaches millions of young people every year – from White, Black, Hispanic, Asian, low-income, middle-income, high-income, urban, rural and suburban backgrounds – and it should be a priority of the field to investigate the effectiveness of the programme for the populations it reaches.

Further, researchers in the field should begin considering elements for a standard measure of effectiveness for in-school prevention programmes. The field should consider what measures for youth substance use are most relevant in warding off drug abuse and addiction and what measures policy makers consider when choosing a prevention programme.

Conclusion

D.A.R.E. is the most prevalent in-school substance use prevention programme in the United States, reaching approximately 75% of school districts. Therefore, it is important for D.A.R.E. to employ a prevention programme substantiated by a wealth of scientific evidence. The systematic review revealed major shortfalls in the evidence basis for the KiR D.A.R.E. programme. Without empirical evidence, we cannot conclusively confirm or deny the effectiveness of the programme. However, we can conclude that the evidence basis for the D.A.R.E. version of KiR is weak, and that there is substantial reason to believe that KiR D.A.R.E. may not be suited for nationwide implementation.

Aside from synthesising the results of studies on the Keepin’ it REAL intervention, this review demonstrates the need for a standardised measure and evaluation system for in-school prevention programmes. As it stands, researchers choose their own measures and statistical tests when studying in-school prevention programmes, increasing the risk of bias in studies. A major body for evaluating prevention programmes based on efficacy studies is the National Registry of Evidence Based Practices and Programs, which has a low standard for its designation of “evidence-based” prevention programmes and does not distinguish between evidence-based universally and/or evidence-based for a specific population. The combination of these two effects could cause ineffective or suboptimal in-school prevention programmes to be identified as “evidence-based” and disseminated widely. Ideally, the use of a standard measure could also account for differences in variations and adaptations. In the case of KiR, it seems that the programme has been deemed “evidence-based” due to the success of one or two of its middle school interventions. However, the evidence-based designation extends to all versions of KiR, including ones without scientific evidence, such as the KiR White/Black Version, KiR-AE for 5th graders and KiR-Plus for 5th graders, or ones that have yet to be studied, like KiR D.A.R.E. or KiR D.A.R.E. Elementary. A standardised measure and evaluation system would be better equipped to address variations in effectiveness between different versions and adaptations of the same programme and would also help schools decide among prevention programmes, which use different metrics for efficacy. Previous work towards a standardised measure and evaluation system (e.g. Flay et al., 2005) and advancing the replication of prevention programmes (e.g. Valentine et al., 2011) should be continued.

Declaration of interest

The authors declare no conflict with this research.