Table Of ContentEFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 1
Awareness is Not Enough: A Double-blind Randomized
Controlled Trial of the Effects of Providing Discipline
Disproportionality Data Reports to School Administrators
Kent McIntosh
University of Oregon
Keith Smolkowski
Oregon Research Institute
Cody M. Gion
Lauren Witherspoon
Eoin Bastable
Erik J. Girvan
University of Oregon
CITATION: McIntosh, K., Smolkowski, K., Gion, C., Witherspoon, L., Bastable, E., & Girvan,
E. J. (2020). Awareness is not enough: A double-blind randomized controlled trial of the effects
of providing discipline disproportionality data reports to school administrators. Educational
Researcher.
Author Note
The research reported here was supported by the Institute of Education Sciences, U.S.
Department of Education, through Grant R324A170034 to University of Oregon. The opinions
expressed are those of the authors and do not represent views of the Institute or the U.S.
Department of Education.
Corresponding Author: Kent McIntosh, University of Oregon, 1235 University of Oregon,
Eugene, OR 97403, [email protected]
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 2
Abstract
One commonly-used strategy used in attempts to decrease racial disproportionality in school
discipline across the country is sharing data with school administrators that discipline disparities
are a problem in their schools, with the assumption that it will increase attention to equity and
improve outcomes. The purpose of this study was to assess the effects of providing monthly
disciplinary equity reports to school administrators in 35 schools on levels of (a) disciplinary
equity report viewing, (b) disciplinary equity, and (c) inclusion of disciplinary equity into school
improvement plan goals. We used a small, double-blind randomized controlled trial, in which
half of the schools were randomly assigned to receive either monthly disciplinary equity reports
or monthly general discipline reports. Results showed that schools receiving the equity reports
had significantly increased rates of viewing equity reports, but no meaningful change in
disciplinary equity or equity goal setting.
Keywords: Racial Equity, Performance Feedback, Data, School
Discipline
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 3
Awareness is Not Enough: A Double-blind Randomized Controlled Trial of the Effects of
Providing Discipline Disproportionality Data Reports to School Administrators
The disproportionate use of office discipline referrals (ODRs) for Black students is a
longstanding concern in education (U.S. Government Accountability Office, 2018). Although
there is evidence for effectiveness of some systems-level interventions, such as positive
behavioral interventions and supports (McIntosh, Gion, & Bastable, 2018), implementing these
interventions with fidelity requires administrator commitment and substantial resources; thus,
district-wide implementation can be challenging. Accordingly, there is a tendency to seek
efficient interventions that can be implemented quickly.
To motivate school administrators to take action, districts or state teams in multiple states
have adopted the intervention of sending school administrators regular (e.g., monthly) reports
showing the extent to which their schools have racial disparities in exclusionary discipline
(Santiago-Rosario, 2019). Although minimal in terms of effort, sending equity reports has
intuitive appeal as a way to draw attention to and signal the importance of addressing the
problem (Kluger & DeNisi, 1996). Further, there is evidence that school teams that use data
more often for decision making also have lower racial discipline disproportionality (Tobin &
Vincent, 2011). Even so, at the school level, simply making equity reports available to school
administrators on demand does not necessarily increase their viewing of discipline data
(McIntosh, Eliason, Horner, & May, 2014). Moreover, sharing reports showing racial inequities
could backfire, increasing resistance to equity efforts (Hetey & Eberhardt, 2018; Kluger &
DeNisi, 1996).
Purpose of the Study
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 4
The study’s purpose was to provide a rigorous experimental test of providing disciplinary
equity data reports to school administrators. We assessed the effects of providing monthly
disciplinary equity reports, compared to a control of monthly discipline summary reports, to
school administrators on (a) viewing disciplinary equity reports, (b) equity in school discipline,
and (c) identifying disciplinary equity as a school improvement goal.
Method
Settings and Participants
A total of 35 public K-12 schools and their school administrators within three school
districts in the U.S. Pacific Northwest participated in the study. The 26 elementary, 5 middle, and
4 high schools used the School Wide Information System (SWIS; May et al., 2018) to record and
review discipline data. The average Black-White ODR risk ratio was 2.2 (ranging from 0 to 6.3),
meaning that on average, Black students were more than twice as likely to receive one or more
ODRs than White students.
Measures
Viewing of school discipline reports. Through SWIS, we extracted counts of discipline
data reports (i.e., sets of graphs and tables) generated by school users per month. We
standardized the data by using the metric of reports generated per 20 school days, roughly a
month of school.
Equity in school discipline. We assessed equity in school discipline using the Black-
White ODR rate difference, calculated by subtracting the ODRs per White student per day per
month from the ODRs per Black student per day per month. This metric of disciplinary equity
accounted for enrollment of each student group and varying school days in each month. A rate
difference of zero indicates perfect equity in school discipline, and positive values indicate
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 5
higher ODR rates for Black students. The rate difference was selected because, unlike risk ratios,
it has a symmetrical distribution based on a standard scale (change in rates) and thus is more
appropriate for computation of difference scores without transformation and can be calculated
when one group receives zero ODRs (Girvan, McIntosh, & Smolkowski, 2019).
School improvement plan goal setting. We obtained publicly available, annual school
improvement plans for each of the participating schools from 2016-17 and 2017-18, which
described their formal academic and behavior goals. We coded the plans for whether they
included disciplinary equity as a goal (intercoder agreement = 88%).
Intervention
The intervention was a monthly email to school administrators containing one of the
continuously-available SWIS school discipline data reports (see http://www.pbisapps.org): either
the (a) School Summary Report (control condition), which includes figures and tables showing
general patterns of ODRs; or (b) School Ethnicity Report (intervention condition), which
includes figures and tables showing the extent of racial discipline disproportionality. No other
communication or support was provided.
Procedure
Once recruited, schools were matched into pairs within districts based on their
disciplinary equity (specifically the Black-White ODR risk ratio) and randomly assigned to
condition in January of Year 2. School administrators were emailed the reports on the first day of
each month for the remaining five months of the school year (one school was lost from the
sample due to not receiving its report in time). Figure 1 shows the participant flow diagram. The
school and district administrators, lead author (principal investigator), second author
(methodologist), and fifth and sixth authors were all blinded to condition until all analytic
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 6
decisions were completed. Only the third and fourth authors, who generated and sent the reports,
were aware of condition during the study, and they did not participate in analytic decisions.
Data Analysis
Report viewing and disciplinary equity. Analyses assessed the effects of receiving the
emailed equity reports on (a) rates of viewing disciplinary equity reports and (b) actual discipline
disparities. For both DVs, we subtracted Year 1 data for each month from Year 2 data for the
same month to produce year-on-year monthly change over time. Thus, the analyses directly
addressed the hypothesis that the schools that received equity reports compared to general
reports would experience an immediate increase in their viewing of reports and disciplinary
equity above levels from the same time in the previous year. The supplemental appendix
provides a detailed description of data analysis.
School improvement plan goal setting. We evaluated each school’s school
improvement plan for 2016-17 and 2017-18 to assess the extent to which the intervention
condition group had an increased proportion of schools including goals to increase racial equity
in school discipline. We conducted a chi-square test of proportions across groups to assess
statistical significance.
Results
Report Viewing and Disciplinary Equity
Figure 2 presents the year-on-year mean differences for equity report viewing and
disciplinary equity by condition, with 95% confidence intervals, pre and post intervention.
Results showed a statistically significant increase in equity report views (p = .023, Hedges’ g =
0.65) upon intervention for the intervention schools but no decrease in Black-White ODR
differences, while control schools saw a decrease (p = .155, Hedges’ g = 0.09). The supplemental
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 7
appendix provides detailed results.
School Improvement Plan Goal Setting
Prior to intervention, no schools in either condition included school improvement plan
goals referring to increasing racial equity in school discipline. After intervention, no control
schools and one intervention school reported a school improvement plan goal of increasing
disciplinary equity. A χ2 analysis of post-test proportions showed that the one-school increase in
equity goal setting was not statistically significant (χ2 = 1.09, p > .05).
Discussion
Given the need to address disciplinary equity, district and state teams are increasingly
engaging in the practice of sending school administrators reports of their disciplinary equity. Our
study examined the effects of directly sending monthly disciplinary equity reports to school
principals on (a) accessing data, (b) disciplinary equity, and (c) equity goal setting. The double-
blind randomized controlled trial design with an attention control provided an opportunity to
assess the effects of this common intervention with less susceptibility to reactivity or social
desirability bias.
Regarding data report viewing, although they had already been available to school
administrators, sending equity reports directly to them significantly increased how much they
accessed and viewed the reports. Although the increase in equity report viewing was
encouraging, the provision of reports did not lead to any meaningful changes in disciplinary
equity or school improvement plan goal setting. As such, we view this intervention as ineffective
in improving equity on its own. This study adds to previous research in performance feedback
and equity data that simply sharing data showing inequities could have iatrogenic effects,
especially if recipients perceive that they do not have sufficient control of the context to achieve
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 8
equitable disciplinary outcomes (Bastable & McIntosh, 2020; Hetey & Eberhardt, 2018; Kluger
& DeNisi, 1996).
Limitations
Although tightly controlled and rigorous, this study has limitations worth considering.
First, due to software limitations, we could not assess which users were accessing reports. As a
result, it is unclear whether the increase in viewing came from school administrators or other
school or district personnel. Second, the study was small and may have been under-powered to
detect changes in ODRs.
Implications
Our findings indicate that sending equity reports directly to school administrators
increases attention to data, although such effects do not appear to change equity in school
discipline or goal setting. This study suggests that when provided evidence of disproportionality,
school administrators are unlikely to take important actions, like setting equity goals in their
school improvement plans. Yet it is noteworthy that providing reports increased views, which
has been an elusive outcome (McIntosh et al., 2014). More research is needed to determine how
this common intervention might be supplemented to increase goal setting or disciplinary equity.
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 9
References
Allison, P. D. (2009). Missing data. In R. E. Millsap & A. Maydeu-Olivares (Eds.), The Sage
handbook of quantitative methods in psychology (pp. 72-89). Thousand Oaks, CA: Sage.
doi: 10.4135/9780857020994.n4
Bastable, E., & McIntosh, K. (2020). Understanding educators' commitment to equity in school
discipline. Manuscript submitted for publication.
Girvan, E. J., McIntosh, K., & Smolkowski, K. (2019). Tail, tusk, and trunk: What different
metrics reveal about racial disproportionality in school discipline. Educational
Psychologist, 54, 40-59. doi:10.1080/00461520.2018.1537125
Hetey, R. C., & Eberhardt, J. L. (2018). The numbers don’t speak for themselves: Racial
disparities and the persistence of inequality in the criminal justice system. Current
Directions in Psychological Science, 0963721418763931.
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A
historical review, a meta-analysis, and a preliminary feedback intervention theory.
Psychological Bulletin, 119, 254-284.
May, S. L., Ard, W. I., Todd, A. W., Horner, R. H., Glasgow, A., Sugai, G., & Sprague, J. R.
(2018). School-Wide Information System (v6.0). Educational and Community Supports,
University of Oregon, Eugene, OR. Retrieved from http:/www.pbisapps.org
McIntosh, K., Eliason, B. M., Horner, R. H., & May, S. L. (2014). Have schools increased their
use of the SWIS School Ethnicity Report? PBIS evaluation brief. OSEP TA Center on
Positive Behavioral Interventions and Supports. Eugene, OR.
EFFECTS OF PROVIDING DISCIPLINARY EQUITY REPORTS 10
McIntosh, K., Gion, C., & Bastable, E. (2018). Do schools implementing SWPBIS have
decreased racial disproportionality in school discipline? PBIS evaluation brief. OSEP
TA Center on Positive Behavioral Interventions and Supports. Eugene, OR.
Santiago-Rosario, M. R. (2019). State and district team provision of disciplinary equity data to
school administrators. Unpublished report. University of Oregon, Eugene, OR.
Tobin, T. J., & Vincent, C. G. (2011). Strategies for preventing disproportionate exclusions of
African American students. Preventing School Failure, 55, 192-201.
doi:10.1080/1045988X.2010.532520
U.S. Government Accountability Office. (2018). Discipline disparities for black students, boys,
and students with disabilities. Report to congressional requesters, U.S. House (Report
GAO-18-258). Washington, DC: Author.
