Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis

0.11 MB·

by  Björn Ross

Measuring the Reliability of Hate Speech Annotations:
The Case of the European Refugee Crisis
Bjo¨rnRoss MichaelRist GuillermoCarbonell
BenjaminCabrera NilsKurowsky MichaelWojatzki
ResearchTrainingGroup”User-CentredSocialMedia”
DepartmentofComputerScienceandAppliedCognitiveScience
UniversityofDuisburg-Essen
[email protected]
Abstract and report it to the relevant authorities. It would
alsomakeiteasierforresearcherstoexaminethe
Someusersofsocialmediaarespreading diffusionofhatefulcontentthroughsocialmedia
racist, sexist, and otherwise hateful con- onalargescale.
tent. For the purpose of training a hate Fromanaturallanguageprocessingperspective,
7
1 speechdetectionsystem,thereliabilityof hatespeechdetectioncanbeconsideredaclassifica-
0 the annotations is crucial, but there is no tiontask: givenanutterance,determinewhetheror
2
universally agreed-upon definition. We notitcontainshatespeech. Trainingaclassifierre-

n collectedpotentiallyhatefulmessagesand quiresalargeamountofdatathatisunambiguously
a
J asked two groups of internet users to de- hatespeech. Thisdataistypicallyobtainedbyman-
7 terminewhethertheywerehatespeechor uallyannotatingasetoftextsbasedonwhethera
2 not,whethertheyshouldbebannedornot certainelementcontainshatespeech.

and to rate their degree of offensiveness. Thereliabilityofthehumanannotationsisessen-
]
L Oneofthegroupswasshownadefinition tial,bothtoensurethatthealgorithmcanaccurately
C priortocompletingthesurvey. Weaimed learnthecharacteristicsofhatespeech,andasan
s. to assess whether hate speech can be an- upperboundontheexpectedperformance(Warner
c
notated reliably, and the extent to which andHirschberg,2012; WaseemandHovy,2016).
[
existingdefinitionsareinaccordancewith As a preliminary step, six annotators rated 469
1
v subjectiveratings. Ourresultsindicatethat tweets. Wefoundthatagreementwasverylow(see
8 showingusersadefinitioncausedthemto Section3). Wethencarriedoutgroupdiscussions
1 partially align their own opinion with the tofindpossiblereasons. Theyrevealedthatthere
1
8 definition but did not improve reliability, isconsiderableambiguityinexistingdefinitions. A
0 whichwasverylowoverall. Weconclude givenstatementmaybeconsideredhatespeechor
1. thatthepresenceofhatespeechshouldper- notdependingonsomeone’sculturalbackground
0 hapsnotbeconsideredabinaryyes-or-no andpersonalsensibilities. Thewordingoftheques-
7
decision,andratersneedmoredetailedin- tionmayalsoplayarole.
1
: structionsfortheannotation. Wedecidedtoinvestigatetheissueofreliability
v
i furtherbyconductingamorecomprehensivestudy
X 1 Introduction
across a large number of annotators, which we
r
a presentinthispaper.
Social media are sometimes used to disseminate
Ourcontributioninthispaperisthreefold:
hatefulmessages. InEurope,thecurrentsurgein
hatespeechhasbeenlinkedtotheongoingrefugee
• To the best of our knowledge, this paper
crisis. Lawmakers and social media sites are in-
presentsthefirstattemptatcompilingaGer-
creasinglyawareoftheproblemandaredeveloping manhatespeechcorpusfortherefugeecrisis.1
approachestodealwithit,forexamplepromising
• We provide an estimate of the reliability of
to remove illegal messages within 24 hours after
hatespeechannotations.
theyarereported(Titcomb,2016).
Thisraisesthequestionofhowhatespeechcan • Weinvestigatehowthereliabilityoftheanno-
bedetectedautomatically. Suchanautomaticdetec- tationsisaffectedbytheexactquestionasked.
tionmethodcouldbeusedtoscanthelargeamount
1Available at https://github.com/UCSM-DUE/
oftextgeneratedontheinternetforhatefulcontent IWG_hatespeech_public
2 HateSpeech 3 CompilingAHateSpeechCorpus
For the purpose of building a classifier, Warner Aspreviouslymentioned,thereisnoGermanhate
andHirschberg(2012)definehatespeechas“abu- speech corpus available for our needs, especially
sivespeechtargetingspecificgroupcharacteristics, not for the very recent topic of the refugee crisis
such as ethnic origin, religion, gender, or sexual in Europe. We therefore had to compile our own
orientation”. Morerecentapproachesrelyonlists corpus. We used Twitter as a source as it offers
ofguidelinessuchasatweetbeinghatespeechif recent comments on current events. In our study
it“usesasexistorracialslur”(WaseemandHovy, we only considered the textual content of tweets
2016). These approaches are similar in that they thatcontaincertainkeywords,ignoringthosethat
leave plenty of room for personal interpretation, containpictures orlinks. Thissection provides a
since there may be differences in what is consid- detailed description of the approach we used to
ered offensive. For instance, while the utterance selectthetweetsandsubsequentlyannotatethem.
“therefugeeswillliveoffourmoney”isclearlygen- To find a large amount of hate speech on the
eralising and maybe unfair, it is unclear if this is refugee crisis, we used 10 hashtags3 that can be
alreadyhatespeech. Moreprecisedefinitionsfrom used in an insulting or offensive way. Using
lawarespecifictocertainjurisdictionsandthere- thesehashtagswegathered13766tweetsintotal,
foredonotcaptureallformsofoffensive,hateful roughlydatingfromFebruarytoMarch2016. How-
speech, see e.g. Matsuda (1993). In practice, so- ever, these tweets contained a lot of non-textual
cialmediaservicesareusingtheirowndefinitions contentwhichwefilteredoutautomaticallybyre-
which have been subject to adjustments over the moving tweets consisting solely of links or im-
years(Jeong,2016). AsofJune2016,Twitterbans ages. Wealsoonlyconsideredoriginaltweets,as
hatefulconduct2. retweets or replies to other tweets might only be
Withtheriseinpopularityofsocialmedia,the clearlyunderstandablewhenreadingbothtweets
presenceofhatespeechhasgrownontheinternet. together. Inaddition,weremovedduplicatesand
Posting a tweet takes little more than a working near-duplicatesbydiscardingtweetsthathadanor-
internet connection but may be seen by users all malisedLevenshteineditdistancesmallerthan.85
overtheworld. toanaforementionedtweet. Afirstinspectionof
Alongwiththepresenceofhatespeech,itsreal- theremainingtweetsindicatedthatnotallsearch
life consequences are also growing. It can be a termswereequallysuitedforourneeds. Thesearch
precursorandincentiveforhatecrimes,anditcan term#Pack(verminorlowlife)foundapotentially
besoseverethatitcanevenbeahealthissue(Bur- largeamountofhatespeechnotdirectlylinkedto
nap and Williams, 2014). It is also known that therefugeecrisis. Itwasthereforediscarded. As
hate speech does not only mirror existing opin- a last step, the remaining tweets were manually
ionsinthereaderbutcanalsoinducenewnegative readtoeliminatethosewhichweredifficulttoun-
feelings towards its targets (Martin et al., 2013). derstandorincomprehensible. Afterthesefiltering
Hate speech has recently gained some interest as steps, our corpus consists of 541 tweets, none of
a research topic on the one hand – e.g. (Djuric whichareduplicates,containlinksorpictures,or
etal.,2014;BurnapandWilliams,2014;Silvaet areretweetsorreplies.
al., 2016) – but also as a problem to deal with in Asafirstmeasurementofthefrequencyofhate
politicssuchastheNoHateSpeechMovementby speechinourcorpus,wepersonallyannotatedthem
theCouncilofEurope. based on our previous expertise. The 541 tweets
The current refugee crisis has made it evident weresplitintosixpartsandeachpartwasannotated
that governments, organisations and the public bytwooutofsixannotatorsinordertodetermine
shareaninterestincontrollinghatespeechinsocial ifhatespeechwaspresentornot. Theannotators
media. However,thereseemstobelittleconsensus were rotated so that each pair of annotators only
onwhathatespeechactuallyis. evaluatedonepart. Additionallytheoffensiveness
ofatweetwasratedona6-pointLikertscale,the
2“Youmaynotpromoteviolenceagainstordirectlyattack
samescaleusedlaterinthestudy.
orthreatenotherpeopleonthebasisofrace,ethnicity,national
origin,sexualorientation,gender,genderidentity,religious
affiliation,age,disability,ordisease. Wealsodonotallow 3#Pack, #Aslyanten, #WehrDich, #Krimmigranten,
accounts whose primary purpose is inciting harm towards #Rapefugees, #Islamfaschisten, #RefugeesNotWelcome,
othersonthebasisofthesecategories.”,TheTwitterRules #Islamisierung,#AsylantenInvasion,#Scharia
Evenamongresearchersfamiliarwiththedefini- AppliedCognitiveScienceoftheFacultyofEngi-
tionsoutlinedabove,therewasstillalowlevelof neeringattheUniversityofDuisburg-Essen.
agreement(Krippendorff’sα =.38). Thissupports
5 PreliminaryResultsandDiscussion
our claim that a clearer definition is necessary in
order to be able to train a reliable classifier. The
Since the surveys were completed by 56 partici-
lowreliabilitycouldofcoursebeexplainedbyvary-
pants, they resulted in 1120 annotations. Table 1
ingpersonalattitudesorbackgrounds,butclearly
showssomesummarystatistics.
needsmoreconsideration.
Def. Nodef. p r
4 Methods
Participants 25 31
Age(mean) 33.3 30.5
Inordertoassessthereliabilityofthehatespeech
Gender(%female) 43.5 58.6
definitionsonsocialmediamorecomprehensively,
HateSpeech(%yes) 32.6 40.3 .26 .15
we developed two online surveys in a between-
Ban(%yes) 32.6 17.6 .01 -.32
subjectsdesign. Theywerecompletedby56par-
Offensive(mean) 3.49 3.42 .55 -.08
ticipantsintotal(seeTable1). Themaingoalwas
toexaminetheextenttowhichnon-expertsagree
Table1: Summarystatisticswithpvaluesandef-
upon their understanding of hate speech given a
fectsizeestimatesfromWMWtests. Notallpar-
diversity of social media content. We used the
ticipantschosetoreporttheirageorgender.
Twitterdefinitionofhatefulconductinthefirstsur-
vey. Thisdefinitionwaspresentedatthebeginning, Toassesswhetherthedefinitionhadanyeffect,
and again above every tweet. The second survey wecalculated,foreachparticipant,thepercentage
did not contain any definition. Participants were oftweetstheyconsideredhatespeechorsuggested
randomlyassignedoneofthetwosurveys. to ban and their mean offensiveness rating. This
Thesurveysconsistedof20tweetspresentedin allowedustocomparethetwosamplesforeachof
a random order. For each tweet, each participant thethreequestions. PreliminaryShapiro-Wilktests
wasaskedthreequestions. Dependingonthesur- indicatedthatsomeofthedatawerenotnormally
vey,participantswereasked(1)toanswer(yes/no) distributed. WethereforeusedtheWilcoxon-Mann-
if they considered the tweet hate speech, either Whitney(WMW)testtocomparethethreepairsof
basedonthedefinitionorbasedontheirpersonal series. TheresultsarereportedinTable1.
opinion. Afterwardstheywereasked(2)toanswer Participantswhowereshownthedefinitionwere
(yes/no)ifthetweetshouldbebannedfromTwitter. more likely to suggest to ban the tweet. In fact,
Participantswerefinallyasked(3)toanswerhow participants in group one very rarely gave differ-
offensivetheythoughtthetweetwasona6-point ent answers to questions one and two (18 of 500
Likertscalefrom1(Notoffensiveatall)to6(Very instancesor3.6%). Thissuggeststhatparticipants
offensive). If they answered 4 or higher, the par- in that group aligned their own opinion with the
ticipants had the option to state which particular definition.
wordstheyfoundoffensive. We chose Krippendorff’s α to assess reliabil-
After the annotation of the 20 tweets, partici- ity,ameasurefromcontentanalysis,wherehuman
pants were asked to voluntarily answer an open codersarerequiredtobeinterchangeable. There-
question regarding the definition of hate speech. fore,itmeasuresagreementinsteadofassociation,
Inthesurveywiththedefinition,theywereasked whichleavesnoroomfortheindividualpredilec-
if the definition of Twitter was sufficient. In the tions of coders. It can be applied to any number
surveywithoutthedefinition,theparticipantswere of coders and to interval as well as nominal data.
askedtosuggestadefinitionthemselves. Finally, (Krippendorff,2004)
sociodemographicdatawerecollected,including Thisallowedustocompareagreementbetween
age,genderandmorespecificinformationregard- bothgroupsforallthreequestions. Figure1visu-
ingtheparticipant’spoliticalorientation,migration alisestheresults. Overall,agreementwasverylow,
background, and personal position regarding the ranging from α =.18 to .29. In contrast, for the
refugeesituationinEurope. purposeofcontentanalysis,Krippendorffrecom-
Thesurveyswereapprovedbytheethicalcom- mends a minimum of α =.80, or a minimum of
mitteeoftheDepartmentofComputerScienceand .66forapplicationswheresomeuncertaintyisun-
0.3 In the future, finding the characteristics that
make users consider content hateful will be use-
fulforbuildingamodelthatautomaticallydetects
0.2 Group
hatespeechanduserswhospreadhatefulcontent,
a
h No definition
p andfordeterminingwhatmakesusersdisseminate
Al
Definition
0.1 hatefulcontent.
Acknowledgments
0.0
This work was supported by the Deutsche
1 2 3
Question Forschungsgemeinschaft (DFG) under grant No.
GRK 2167, Research Training Group ”User-
Figure 1: Reliability (Krippendorff’s α) for the CentredSocialMedia”.
differentgroupsandquestions
References
problematic(Krippendorff,2004). Reliabilitydid
Peter Burnap and Matthew Leighton Williams. 2014.
not consistently increase when participants were HateSpeech,MachineClassificationandStatistical
shownadefinition. Modelling of Information Flows on Twitter: Inter-
pretation and Communication for Policy Decision
Tomeasuretheextenttowhichtheannotations
Making. InProceedingsofIPP2014,pages1–18.
usingtheTwitterdefinition(questiononeingroup
one)wereinaccordancewithparticipants’opinions NemanjaDjuric,RobinMorrisJingZhou,MihajloGr-
bovic,VladanRadosavljevic,andNarayanBhamidi-
(questiononeingrouptwo),wecalculated,foreach
pati. 2014. Hate Speech Detection with Comment
tweet,thepercentageofparticipantsineachgroup
Embeddings. In ICML 2014, volume 32, pages
whoconsideredithatespeech,andthencalculated 1188–1196.
Pearson’s correlation coefficient. The two series
Sarah Jeong. 2016. The History of Twitter’s Rules.
correlatestrongly(r=.895,p<.0001),indicating
VICEMotherboard.
thattheymeasurethesameunderlyingconstruct.
KlausKrippendorff. 2004. ReliabilityinContentAnal-
ysis: Some Common Misconceptions and Recom-
6 ConclusionandFutureWork
mendations. HCR,30(3):411–433.
Thispaperdescribesthecreationofourhatespeech Ryan C Martin, Kelsey Ryan Coyier, Leah M VanSis-
corpusandoffersfirstinsightsintothelowagree- tine,andKellyLSchroeder. 2013. AngerontheIn-
ment among users when it comes to identifying ternet: thePerceivedValueofRant-Sites. Cyberpsy-
chology,behaviorandsocialnetworking,16(2):119–
hateful messages. Our results imply that hate
22.
speechisavagueconceptthatrequiressignificantly
betterdefinitionsandguidelinesinordertobeanno- Mari J Matsuda. 1993. Words that Wound - Criti-
cal Race Theory, Assaultive Speech, and the First
tatedreliably. Basedonthepresentfindings,weare
Amendment. WestviewPress,NewYork.
planningtodevelopanewcodingschemewhichin-
cludesclear-cutcriteriathatletpeopledistinguish Leandro Silva, Mainack Mondal, Denzil Correa,
Fabr´ıcioBenevenuto,andIngmarWeber. 2016. An-
hatespeechfromothercontent.
alyzingtheTargetsofHateinOnlineSocialMedia.
Researcherswhoarebuildingahatespeechde-
InProceedingsofICWSM2016,pages687–90.
tectionsystemmightwanttocollectmultiplelabels
JamesTitcomb. 2016. FacebookandTwitterpromise
foreachtweetandaveragetheresults. Ofcourse
to crack down on internet hate speech. The Tele-
thisapproachdoesnotmaketheoriginaldataany
graph.
morereliable(Krippendorff,2004). Yet,collecting
WilliamWarnerandJuliaHirschberg. 2012. Detecting
the opinions of more users gives a more detailed
Hate Speech on the World Wide Web. In Proceed-
pictureofobjective(orintersubjective)hatefulness.
ingsofLSM2012,pages19–26.ACL.
Forthesamereason,researchersmightwanttocon-
sider hate speechdetection a regression problem, Zeerak Waseem and Dirk Hovy. 2016. Hateful Sym-
bolsorHatefulPeople? PredictiveFeaturesforHate
predicting,forexample,thedegreeofhatefulness
Speech Detection on Twitter. In Proceedings of
ofamessage,insteadofabinaryyes-or-noclassifi-
NAACL-HLT,pages88–93.
cationtask.