Table Of ContentJOURNALOFAPPLIEDBEHAVIORANALYSIS 2007, 40, 659–672 NUMBER4 (WINTER2007)
TREATMENT INTEGRITY OF SCHOOL-BASED INTERVENTIONS
WITH CHILDREN IN THE JOURNAL OF APPLIED
BEHAVIOR ANALYSIS 1991–2005
LAURA LEE MCINTYRE
SYRACUSEUNIVERSITY
FRANK M. GRESHAM
LOUISIANASTATEUNIVERSITY
AND
FLORENCE D. DIGENNARO AND DEREK D. REED
SYRACUSEUNIVERSITY
We reviewed all school-based experimental studies with individuals 0 to 18years published in
theJournalofAppliedBehaviorAnalysis(JABA)between1991and2005.Atotalof142articles
(152 studies) that met review criteria were included. Nearly all (95%) of these experiments
provided an operational definition of the independent variable, but only 30% of the studies
providedtreatmentintegritydata.Nearlyhalfofstudies(45%)werejudgedtobeathighriskfor
treatmentinaccuracies.Treatmentintegritydataweremorelikelytobeincludedinstudiesthat
used teachers, multiple treatment agents, or both. Although there was a substantial increase in
reporting operational definitions of independent variables, results suggest that there was only
a modest improvement in reported integrity over the past 30years of JABA studies.
Recommendations for research andpractice arediscussed.
DESCRIPTORS: treatment integrity, child studies, school interventions, applied behavior
analysis
_______________________________________________________________________________
The field of applied behavior analysis has specification of an independent variable as well
always rested on the fundamental principle that accurate independent variable application, de-
the empirical demonstration of measurable finitive conclusions regarding the relation
changes in behavior must be related to between an independent variable and a depen-
systematic and controlled manipulations in the dentvariablearecompromised.Thebestwayto
environment. That is, the observed changes in ensure accurate application of the independent
the dependent variable (behavior) must be variable is to measure the extent to which
attributed to changes in the independent vari- treatment is implemented as intended.
able (some environmental event). Without this Documentation of independent variable im-
empirical demonstration, a true science of plementation has been discussed in the litera-
human behavior is an impossibility (Skinner, ture under the rubric of treatment fidelity
1953). Without objective and documented (Moncher & Prinz, 1991) or treatment integrity
(Gresham, Gansle, & Noell, 1993; Gresham,
We thank Heidi Olson-Tinker, Lisa Dolstra, Veronica Gansle, Noell, & Cohen, 1993; Peterson,
McLaughlin, and Mai Van for assistance with the initial Homer, & Wonderlich, 1982; Yeaton &
preparationofthisarticle.WealsoaregratefultoMichael
Sechrest, 1981). Treatment integrity refers to
J.Vance for assistancewith datacollection.
Address correspondence to Laura Lee McIntyre, De- thedegreetowhichtreatmentsareimplemented
partment of Psychology, 430 Huntington Hall, Syracuse as planned, designed, or intended and is
University,Syracuse,NewYork13244(e-mail:llmcinty@
concerned with the accuracy and consistency
syr.edu).
doi:10.1901/jaba.2007.659–672 with which interventions are implemented
659
660 LAURA LEE MCINTYRE et al.
(Peterson et al.). Therefore, treatment integrity (Smith, Glass, & Miller, 1980). Having this
is necessary but insufficient for demonstrating knowledge would focus a behavior analyst’s
a functional relation between intervention efforts on problem solving with teachers and
procedures and behavior change (Gresham, students (i.e., change the intervention or di-
1989). rectly work to improve teachers’ implementa-
A number of studies have been published in tion of the current plan). Finally, recent
recent years that have examined variables legislation, such as the No Child Left Behind
associated with adequate treatment integrity Act (U.S. Departmentof Education, 2002) and
(DiGennaro, Martens, & Kleinmann, 2007; Individuals with Disabilities Improvement Act
DiGennaro, Martens, & McIntyre, 2005; (2004), necessitates that school-based practi-
Mortenson & Witt, 1998; Noell, Witt, Gil- tioners and teachers be accountable for their
bertson, Ranier, & Freeland, 1997; Noell et al., practices. As a result, there has been a recent
2000; Sterling-Turner, Watson, Wildmon, push for evidence-based practices in academic
Watkins, & Little, 2001; Witt, Noell, LaFleur, settings as well as demonstrations of accurate
& Mortenson, 1997). Most of these studies plan implementation over time.
have focused on schools as the primary setting How common is the measurement of
for investigation. Investigating the degree to treatment integrity in the behavior analysis
which interventions are carried out with in- literature? Several reviews of the literature
tegrity in schools is valuable for several reasons. suggest that the measurement of treatment
First, research suggests that teachers fail to integrity is uncommon (Gresham, Gansle, &
implement interventions with accuracy despite Noell, 1993; Peterson et al., 1982; Wheeler,
receiving high levels of initial training (e.g., Baggett, Fox, & Blevins, 2006). Peterson et al.
DiGennaro et al., 2005; Noell et al., 2000). reviewed539studiespublishedintheJournalof
This is a waste of time and resources for both AppliedBehaviorAnalysis(JABA)between1968
teachers and consultants if, after training, the and1980;theyfoundthatonly20%ofthe539
interventions are not implemented as intended. studies reported data on treatment integrity,
Second, findings also suggest that student and over 16% of these studies did not provide
problem behaviors are negatively correlated an operational definition of the independent
with treatment accuracy, such that low levels variable. There were no trends suggesting an
of problem behavior are associated with high improvement in treatment integrity over time.
levels of treatment integrity (DiGennaro et al., Gresham, Gansle, and Noell provided an
2005, 2007; Wilder, Atwell, & Wine, 2006). update of Peterson et al.’s review by examining
Thus, a teacher’s failure to implement recom- 158 studies published in JABA between 1980
mended interventions may result in poor out- and 1990 that were child studies (,19 years of
comes for students, in that behaviors will not age). Of these 158 studies, only 32% provided
improve in the desired direction. Third, the an operational definition of the independent
extent to which teachers implement plans with variable and only 16% (25 studies) systemati-
accuracy influences a behavior analyst’s ability cally measured and reported levels of treatment
to effectively conduct formative evaluations. integrity.
Specifically,abehavior analystwill beunable to Wheeleretal.(2006)focusedonintervention
determine if a student’s resistance to treatment studies of children with autism published
isaresultofanineffectiveinterventionoralack between 1993 and 2003. Of the 60 studies
of intervention implementation (Moncher & included in the review, more than half (60%)
Prinz, 1991) because the treatment’s effect size were published in JABA, with the remaining
is positively correlated with internal validity studies (n 5 26) drawn from eight other
SCHOOL-BASED INTERVENTIONS 661
journals (e.g., Research in Developmental Dis- rigor of our experimental procedures (Baer,
abilities, Journal of Autism and Developmental Wolf, & Risley, 1968; Johnston & Penny-
Disorders).TheresultsofWheeleretal.’sreview packer,1993;Kazdin,1973).Thebasicconcern
were consistent with previous studies. Of these isthatwhendataarenotcollectedregardingthe
60studies,only18%(n511)reporteddataon status of the independent variable, researchers
treatment integrity. On the other hand, nearly and practitioners alike cannot objectively con-
all (92%) included operational definitions of clude that the independent variable was im-
independent variables. Closer analysis of the plemented as planned or intended (Kennedy,
results of Wheeler et al.’s review provides some 2005; Moncher & Prinz, 1991). This problem
insight into the treatment-integrity reporting may be especially problematic in practice
trends for child-based autism treatment studies. settings (Wilder et al., 2006), such as interven-
Although most of the included studies were tions that are implemented in schools.
published in JABA, only 14% (n 5 5) included The current article updates and extends the
treatment integrity data. This figure is lower findings of the Peterson et al. (1982) and
than what others have reported (e.g., Gresham, Gresham, Gansle, and Noell (1993) reviews by
Gansle, & Noell, 1993) for JABA studies. In another15 years.Allschool-basedinterventions
with children (,19 years old) published in
contrast, studies published in Research in
JABA between 1991 and 2005 were reviewed
Developmental Disabilities, Focus on Autism
for possible inclusion. The clinical relevance of
and Other Developmental Disabilities, Journal
investigating treatmentintegritycombinedwith
of Autism and Developmental Disorders, and the
the importance of demonstrating that the
Journal of Positive Behavioral Interventions in-
independent variable was accurately applied in
cluded treatment integrity data in 25% to 33%
school-based intervention research serves as the
of studies. Studies that met inclusionary criteria
basis of this study.
published in Education and Treatment of
Children and the Journal of Early Intervention
reported treatment integrity 50% and 100% of METHOD
the time, respectively. In contrast, the three Criteria for Review
studies published in Education and Training in
Atotalof995articles(excludingbookreviews
Mental Retardation and Developmental Disabil-
and remembrances) were reviewed to determine
ities and the Journal of Developmental and possible inclusion. Five features of each study
Physical Disabilities did not report treatment were considered. First, the study had to be
integrity data. Although these findings are experimental, in that the effects of intervention
limited due to the scope of Wheeler et al.’s onbehaviorwereexamined(i.e.,thestudyhadto
review criteria, they are helpful in placing manipulate some aspect of the environment to
treatment integrity reporting in JABA in create changes in a dependent variable). Because
context. we were evaluating school-based intervention
Basedontheforegoingreviews,itisclearthat studies, articles that were assessment only (e.g.,
the majority of treatment outcome studies functional analysis, preference assessment) were
published in JABA and other behavioral excluded. If a study contained an initial
journals either did not measure or did not functional analysis followed by an intervention,
report levels of treatment integrity. As can be the intervention experiment was included.
derived from the above discussion on the Second, participants had to be younger than
importance of treatment integrity, the failure 19 yearsold,aninclusionarycriterionpreviously
to gather data on the integrity of independent employed by Gresham, Gansle, and Noell
variables may compromise the precision and (1993). Third, studies without a clear baseline
662 LAURA LEE MCINTYRE et al.
or control condition were excluded from fur- given the following criterion: ‘‘If you could
ther review. Studies that were not true experi- replicate this treatment with the information
mental designs (e.g., AB designs) were exclud- provided, the intervention is considered opera-
ed. Fourth, all studies had to be conducted in tionally defined.’’ This criterion was proposed
school settings; however, school was liberally by Baer et al. (1968) and later used by
defined to include a continuum of school Gresham, Gansle, and Noell (1993) in their
placements, including residential programs. review. Those studies that referred to more
Inpatient hospital units (e.g., Neurobehavioral extensive sources (e.g., book chapters, manuals,
Unit at the Kennedy Krieger Institute) and or technical reports) were coded as ‘‘footnote’’
outpatient clinics were excluded. Fifth, brief (i.e., contained directions to contact the author
reports of three or fewer pages in length were or see published details elsewhere).
excluded, as outlined by Peterson et al. (1982). Monitoring treatment integrity. Studies were
Becausearticles of three or fewerpages typically coded according to their inclusion of treatment
do not provide sufficient methodological detail integrity data. Studies that systematically mon-
(e.g., lengthy descriptions of independent itored and reported treatment integrity on at
variables or integrity monitoring), we chose
least one independent variable were coded
to exclude these studies so we would not
‘‘yes.’’ Specifically, this included studies that
artificially underestimate independent variable
(a) specified a method of measurement (observ-
operational definition and integrity reporting.
er present, videotaping of sessions, component
Thus, a total of 142 articles met these
checklist)and(b)reporteddataaspercentageof
inclusionary criteria over the 15-year period.
implementation (i.e., percentage of implemen-
Because some of the articles contained multiple
ted steps in the intervention). Studies that
experiments, a total of 152 studies met
monitored treatment integrity but failed to
inclusionary criteria for this review. (A full list
report data were coded as ‘‘monitored.’’ For
of articles meeting inclusionary criteria is
example, ‘‘treatment integrity was assessed to
available from the first author.)
ensure the fidelity of this intervention’’ was
coded as ‘‘monitored.’’ Likewise, studies that
Coding
mentioned statements such as ‘‘deviations from
This review focused on the operational
intervention protocol were not observed’’ were
definition of the independent variables and
also coded as ‘‘monitored’’ (no method of
the extent to which these variables were
measurement was described). The key differ-
described,monitored, andmeasured.Following
encebetween‘‘yes’’and‘‘monitored’’categories
the procedural guidelines set forth by Peterson
et al. (1982), the risk for treatment inaccuracies was the provision of percentage data regarding
was also investigated. In addition, we were implementation and a specified data-collection
interested in assessing whether treatment in- method. Studies that made no mention of
tegrity reporting trends varied by publication treatment integrity were coded ‘‘no.’’ We chose
year and by whom the intervention was to replicate Gresham, Gansle, and Noell’s
implemented (treatment agent; e.g., teacher, (1993) treatment integrity coding because,
researcher, etc.). Coding schemes for each of unlike Peterson et al. (1982), this method
these variables are described below. allowed differentiating the categories of ‘‘yes’’
Operational definition of the independent and ‘‘monitored.’’
variable. Each study was coded ‘‘yes,’’ ‘‘no,’’ Risk for treatment inaccuracies. Treatments
or ‘‘footnote’’ in answer to the question: Is the were coded as either no, low, or high risk for
independent variable (treatment) operationally treatment inaccuracies based on the guidelines
defined?Toanswerthisquestion,eachraterwas set forth by Peterson et al. (1982). Treatments
SCHOOL-BASED INTERVENTIONS 663
were coded as ‘‘no risk’’ if the implementation tors. Researchers and research assistants were
of the treatment was reported as monitored or individuals who collected data for the purpose
measured (i.e., monitoring of treatment in- ofthepublishedstudyandwerenotinvolvedin
tegrity was coded as either ‘‘yes’’ or ‘‘moni- other service delivery roles (e.g., classroom
tored’’).Treatments were coded as ‘‘low risk’’ if teacher). Peer tutors were other children,
thetreatmentwasnotreportedtobemonitored typically in the target child’s classroom, who
ormeasuredbutwasjudgedtobeatlowriskfor were not the focus of the intervention. ‘‘Self’’
inaccuracies. Low-risk treatments included was recorded if the intervention was self-
treatments that were (a) mechanically defined administered or self-mediated (e.g., self-moni-
(e.g., computer mediated), (b) permanent toring interventions). ‘‘Multiple’’ was coded if
products (e.g., posting of classroom rules), (c) more than one category of treatment agent was
continuously applied (e.g., noncontingent ac- used. If the treatment agent described in the
cess to preferred items or activities), or (d) study did not fit in any of the aforementioned
single components (e.g., escape contingent on categories, ‘‘other’’ was coded. There were
work completion). Treatments were coded as a small handful of studies that did not specify
‘‘highrisk’’ if the treatmentwas not reported to the treatment agent. In these cases ‘‘not
be monitored or measured but was necessary. specified’’ was coded.
According to Peterson et al. (1982) treatments
in the high-risk category were those in which Rater Training and Interobserver Agreement
‘‘the administrationof the independent variable
A PhD-level behavior analyst (faculty mem-
was not exempted by any of the cases cited in
ber) and four doctoral students with advanced
categoryB[lowrisk],andthepotentialforerror
training in behavior analysis served as raters,
was judged to be high’’ (p. 485). Operationally
witheachratercoding20%ofthestudies.Prior
defined, these included person-implemented
to coding, all raters received four 2-hr training
interventions that included multiple behavioral
sessionstodiscussassignedpracticearticles(i.e.,
components (e.g., contingent reinforcement
JABA articles published prior to 1991) and to
with response cost).
revise ambiguous codes. During these training
Publication year. The publication year of the
sessions, all ratersreached 100% agreement (via
article was recorded (i.e., 1991 to 2005).
consensus) on whether an assigned article met
Treatment agent. The individuals who im-
inclusionary criteria. Five articles were assigned
plemented the intervention were classified into
per training session, yielding a total of 20
one of the following mutually exclusive cate-
training articles used prior to conducting
gories: (a) teacher, (b) professional (nontea-
independent coding. In addition, a random
cher),(c)paraprofessional,(d)parentorsibling, sample of 20% of studies meeting inclusionary
(e) researcher or research assistant, (f) peer criteria was selected for interobserver agreement
tutors, (g) self, (h) multiple, (i) other, or (j) not coding. Studies were coded on five categories:
specified. Examples of the teacher category (a) operational definition of the independent
included early childhood educators, general variable (three categories), (b) integrity assess-
education classroom teachers, or discrete-trial ment (three categories), (c) risk for treatment
instructors. The professional category included inaccuracies (three categories), (d) publication
other nonteacher professionals (e.g., school year(15categories),and(e)treatmentagent(10
psychologists, speech–language pathologists). categories). Percentage agreement was calculat-
Paraprofessionals included support staff such edbydividingthenumberofagreementsbythe
as classroom aides, teaching assistants (non- number of agreements plus disagreements and
teachers), or playground or lunchroom moni- multiplying by 100%. Percentage agreement
664 LAURA LEE MCINTYRE et al.
averaged 93% across the five codes (98% on treatment integrity but were judged to be at
operational definition of the independent vari- low risk for treatment inaccuracies.
able; 87% integrity assessment; 88% risk for Reporting treatment integrity data did not
treatment inaccuracies; 100% publication year; appeartodifferconsistentlybypublicationyear;
92% treatment agent). however, there was ample variability across the
15-year period. Figure 1 depicts the percentage
of studies that included treatmentintegrity data
RESULTS by publication year. On average, treatment
integrity data were included in one third of the
The majority of studies (n 5 144; 95%)
includedstudies(M534%;SD519.23).The
provided operational definitions of treatments,
publication years 1996, 1998, 1999, and 2005
with an additional five studies (3%) reporting
included relatively more studies that reported
references or contact information to allow
treatment integrity data (range, 50% to 67%)
readers to gather more information about the
than the remaining 12 years. Figure 2 shows
interventions (e.g., treatment manuals, pre-
treatment integrity data from 1968 to 2005
viously published studies, etc.). The remaining
based on Peterson et al.’s (1982) review; Gre-
three studies (2%) did not provide operational
sham, Gansle, and Noell’s (1993) review; and
definitions adequate for replication purposes or
thepresentreview. Thesedataarebasedon 834
cite other sources for more information.
studies published in JABA from 1968 to 2005.
Approximately one third (n 5 46; 30%) of
Of these 834 studies, 179 (21%) reported
the studies provided treatment integrity data in
treatment integrity data (range, 0% to 67%).
the form of percentage of implementation.
We were interested in exploring whether
Studies that reported these data showed a
studies that used particular treatment agents
high percentage of integrity (M 5 93%;
(e.g., teachers, researchers) reported treatment
SD 5 9.93). The majority of studies that
integrity data more frequently. As shown in
reported integrity data (n 5 36; 78%) reported
Table 1, there were a variety of reported
procedural fidelity of 90% or greater. Thirteen
treatment agents in the included studies. The
studies(8%)mentionedthattreatmentintegrity
most common were researchers (n 5 52),
was monitored but did not provide data for
teachers (n 5 38), multiple (n 5 19), and
percentage of steps accurately implemented.
professionals (n 5 15). Although only seven
Over 60% of the studies (n 5 93) did not
studies used peer tutors as treatment agents,
report treatment integrity data nor did they
57% (n 5 4) reported treatment integrity data.
report monitoring the implementation of their
Of the 19 studies that used multiple treatment
interventions. agents, nearly a third (n 5 6; 32%) included
Approximately39%ofstudies(n559)were
treatment integrity data. Likewise, for the 38
considered to be at no risk for treatment
studies that used teachers as treatment agents,
inaccuracies, in that the authors reported 37% (n 5 14) reported treatment integrity
treatment integrity data or that treatment data. Studies that used professionals, parents or
integrity was monitored. Just under half of the siblings, researchers, or self-administered treat-
included studies (n 5 69; 45%) were consid- ments had lower reporting of treatment in-
eredtobeathighriskfortreatmentinaccuracies tegrity data (range, 0% to 25%).
in that information on the implementation of
treatments or the assessment of independent
DISCUSSION
variableswasnotincludedbutshouldhavebeen
(Peterson et al., 1982). The remaining 16% of The present review of school-based interven-
studies (n 5 24) did not include information tions with children published in JABA demon-
SCHOOL-BASED INTERVENTIONS 665
Figure1.Percentage ofJABA school-based studies reporting treatmentintegrity databyyear (1991to 2005).
Figure2.PercentageofJABAstudiesreviewedbyPetersonetal.(1982);Gresham,Gansle,andNoell(1993);andthe
current review reporting treatmentintegrity databyyear (1968to2005).
666 LAURA LEE MCINTYRE et al.
Table 1
Treatment Integrity Monitoring byTreatmentAgent
Treatmentagent Yes+datan(%) Monitoredn(%) Non(%) Total
Teacher 14(37) 4(10) 20(53) 38
Professional(nonteacher) 3(20) 1(7) 11(73) 15
Paraprofessional 2(33) 0(0) 4(67) 6
Parentorsibling 0(0) 1(50) 1(50) 2
Researcher 13(25) 4(8) 35(67) 52
Peertutors 4(57) 1(14) 2(29) 7
Multiple 6(32) 2(10) 11(58) 19
Doesnotspecify 3(33) 0(0) 6(67) 9
Self 0(0) 0(0) 2(100) 2
Other 1(50) 0(0) 1(50) 2
Total 46(30) 13(8) 93(61) 152
stratesthatreportingratesoftreatmentintegrity Reasons for low rates of treatment integrity
datahavebeenremarkablystable(andlow)over reporting are not entirely clear; however, low
the past 15 years. Approximately one third reporting may be a function of the editorial
(30%) of studies that met our inclusionary process (i.e., space limitations in journals
criteria reported treatment integrity data. This warrant cutting out treatment integrity data)
figure is slightly higher than the Peterson et al. or may be due to logistics (e.g., lack of skills in
(1982)andGresham,Gansle,andNoell(1993) treatment integrity assessment, lack of re-
reviews of this literature that showed 20% and sources). There may also be a publication bias
16%integrity,respectively.Althoughsomewhat favoring the reporting of treatment integrity
different inclusionary criteria were used in the data when integrity is high. In addition, it is
two earlier reviews, treatment integrity report- plausible that researchers do not view treatment
ing has been remarkably stable over the past integrity data collection as important, especially
37 years (1968 to 2005) (Figure 2). Of interest if interventions produce the desired effects. We
is the large increase in treatment integrity argue that without collecting integrity data, it
reporting that was seen from 1993 to 1994. becomesdifficulttomakeconclusionsregarding
Although attributions about the cause of this intervention results.
increasecannotbemade,thisspikeoccurredthe Having access to treatment integrity data can
year following Gresham, Gansle, and Noell’s help behavior analysts to make decisions about
review. Gresham, Gansle, and Noell reported treatments in school-based settings. If, for
a similar increase from 1982 to 1983 (the year example, an intervention is being implemented
followingPetersonetal.’sreview).Itisplausible accurately yet does not produce the desired
that papers of this nature may increase JABA effects, the behavior analyst will likely modify
authors’ and editors’ awareness of the need to the treatment. If the intervention is being
include treatment integrity data. Alternatively, implementedinaccuratelyanddoesnotproduce
there may be other variables that contributed to the desired effects, the behavior analyst will
the spike in treatment integrity reporting, such likely institute additional training or pro-
as the sharp increase seen from 1997 to 1998. grammed consequences to increase implemen-
To the best of our knowledge, however, tation accuracy. On the other hand, if the
editorial guidelines for preparing manuscripts intervention is not being implemented with
to be submitted to JABA did not change during integrity yet still produces the desired effects,
this time. the behavior analyst will likely change the
Reporting of treatment integrity data has treatment protocol to reflect the modified
been relatively stable and low over the years. intervention.Finally,iftheinterventionisbeing
SCHOOL-BASED INTERVENTIONS 667
implemented with integrity and the desired paraprofessionals, peers, or multiple treatment
treatment outcomes are produced, a causal agents, authors are more likely to report
relation between independent variable manip- treatment integrity data. It may be the case
ulations and changes in the dependent variable thatthese treatmentagentswere judgedtobeat
can be inferred. Thus, we argue that including high risk for procedural inaccuracies and the
regular treatment-integrity assessments is neces- authorsthereforewenttogreatlengthstoensure
sary but insufficient for making treatment- thattheseagentsimplementedthetreatmentsas
related decisions (Gresham, 1989). planned. Although definitive conclusions can-
In contrast to the rates of treatment integrity not be made based on these descriptive data, it
reporting, reporting of operationally defined appearsthatthetreatmentagentusedinschool-
independent variables has dramatically in- based studies may influence the likelihood of
creased, with nearly all (95%) studies including reporting treatment integrity data. What is
detailed descriptions of the interventions. This unknown, however, is how many other authors
figure is consistent with a recent review of collected treatment integrity data but failed to
interventionsforchildrenwithautism(Wheeler report it in their published articles. Failure to
et al., 2006) but is a much improved rate over includeabriefstatementontheextenttowhich
the 34% reported by Gresham, Gansle, and treatments were implemented as planned may
Noell (1993). Including operational definitions be especially problematic for interventions
of independent variables contributes to the judged to be at high risk for treatment
replicability of our science of behavioral inter- inaccuracies(Petersonetal.,1982).Iftreatment
ventions (Bellg et al., 2004). integrity data are not regularly included,
Although treatment integrity measures are inferences based on the study results may be
important for virtually all experimental studies, significantlylimited(Kennedy,2005).Thus,we
including assessment studies and interventions recommend that if treatment integrity data are
conducted in other settings, we chose to sample collected or if intervention implementation is
interventions with children in school settings. monitored,thisinformationshouldbeincluded
This population and setting were selected in published studies.
because it is the focus of our own research; Althoughwehave seenmarkedimprovement
however, this may be of interest to other in descriptions of independent variables, pub-
researchers in its own right. Furthermore, lications in JABA continue to focus on clear
interventions carried out in school settings, in specifications ofthedependentvariables anddo
which treatment agents are less likely to be not include measurements of the independent
researchers with significant training in experi- variables. Indeed, a ‘‘curious double standard’’
mental methods, may be at greatest risk for so aptly recognized by Peterson et al. (1982)
inaccurate implementation of interventions. still remains. This observation continues to be
When treatment integrity is not systematically recognized by various task forces and organiza-
assessed and reported, there is little basis for tionswithinthefieldsofeducation,psychology,
judgments about how closely an implemented andmentalhealth.Forexample,theTaskForce
intervention approximates an intended inter- on Evidence-Based Practice in Special Educa-
vention. Because the current review focused on tion of the Council for Exceptional Children
school settings, the extent to which these stated that the integrity of intervention im-
findings generalize to published studies con- plementation is critical in single-case designs
ducted with other populations is unknown. because the independent variable is implemen-
Our findings suggest that when school-based ted continuously over time (Horner et al.,
interventions are carried out by teachers, 2005).
668 LAURA LEE MCINTYRE et al.
Similarly, other task forces within the tionally related to desired changes in the
American Psychological Association on evi- dependent variable, there has been relatively
dence-based treatments such as Divisions 16 little research that demonstrates this relation
(school psychology), 53 (clinical child/adoles- (Wilder et al., 2006). Furthermore, it may be
cent), and 54 (pediatric) have called for the the case that high levels of treatment integrity
assessment and monitoring of treatment in- are necessary for some interventions but may
tegrity. Furthermore, researchers who submit not be necessary for others. Only a handful of
single-case experimental design grant applica- behavior-analytic studies have addressed this
tions to the U.S. Department of Education’s issue, unfortunately coming to somewhat
Institute of Education Sciences (IES) now must different conclusions. For example, Wilder et
describe ‘‘how treatment fidelity will be mea- al. systematically manipulated different levelsof
sured, frequency of assessments, and what treatment integrity of a three-step prompting
degree of variation in treatment fidelity will be
procedure on children’s compliance. Wilder et
accepted over the course of the study’’ (IES,
al. concluded that the level of treatment
2006,p.50).Theserecommendationshavealso
accuracy had a large impact on children’s
been made by the National Institutes of Health
compliance. Northup, Fisher, Kahng, Harrel,
(NIH). Specifically the NIH Behavior Change
and Kurtz (1997), on the other hand, found
Consortium recommends that treatments be
very little difference between time-out treat-
monitored and reported and that treatment
ments implemented at 100% accuracy and
agents be trained and supervised in the delivery
those implemented at 50% accuracy. Vollmer,
of treatments (Bellg et al., 2004). Monitoring
Roane, Ringdahl, and Marcus (1999) evaluated
and reporting treatment fidelity are especially
the effects of differential reinforcement of
important in clinical treatments that are
alternative behavior and found that degree of
considered to be at high risk for treatment
treatment accuracy did affect treatment out-
inaccuracies or complex in other ways (e.g.,
comes. Because of the small number of studies
multisite).Furthermore, the special NIHreport
that have addressed the varying effects of
on treatment fidelity in research specifies that
treatment integrity on behavior change, we
‘‘it is particularly important that funding
recommend that additional studies include
agencies, reviewers, and journal editors who
treatment integrity variation as an independent
publish behavioral change research consider
variable and consider that various treatments
treatment fidelity issues’’ (Bellg et al., p. 451).
may actually require different levels of treat-
Withtheincreasedattentionpaidtoissues of
mentintegritytoproducedesiredchangesinthe
accurate treatment implementation and report-
dependent variable. Regular documentation of
ing of treatment integrity, both within the field
treatment integrity may help to improve our
of behavior analysis and in other fields (e.g.,
psychology, behavioral medicine), it may be knowledge base in this regard.
particularly important for JABA authors and An additional area of research for behavior-
readers to consider some additional ways to analytic studies may be to separate the compo-
strengthen the influence of behavior analysis in nents of treatment packages to identify the
the larger scientific community. Outlined are variables that are functionally responsible for
severalrecommendationsfortreatmentintegrity producing behavior change. It is plausible that
research and recommendations for practice. some components of a treatment package may
beexcluded,whereasothersmaybenecessaryto
Recommendations for Research and Practice produce treatment effects. Thus, a treatment
Although accurate implementation of the that is implemented with 80% accuracy but is
independent variable is assumed to be func- missing a key ingredient may produce poorer
