Table Of ContentThe Springer Series on Challenges in Machine Learning
Hugo Jair Escalante · Sergio Escalera
Isabelle Guyon · Xavier Baró
Yağmur Güçlütürk · Umut Güçlü
Marcel van Gerven Editors
Explainable and
Interpretable Models
in Computer Vision and
Machine Learning
The Springer Series on Challenges in Machine
Learning
Serieseditors
HugoJairEscalante,INAOE,Puebla,Mexico
IsabelleGuyon,INRIA,UniversitéParisSud,UniversitéParisSaclay,Paris,France
andChaLearn,Berkeley,CA,USA
SergioEscalera,UniversityofBarcelona,Barcelona,Spain
Thebooksofthisinnovativeseriescollectpaperswrittenbysuccessfulcompetitions
inmachinelearning.Theyalsoincludeanalysesofthechallenges,tutorialmaterial,
datasetdescriptions,andpointerstodataandsoftware.Togetherwiththewebsites
ofthechallengecompetitions,theyofferacompleteteachingtoolkitandavaluable
resourceforengineersandscientists.
Moreinformationaboutthisseriesathttp://www.springer.com/series/15602
Hugo Jair Escalante (cid:129) Sergio Escalera
Isabelle Guyon (cid:129) Xavier Baró (cid:129) Yag˘mur Güçlütürk
Umut Güçlü (cid:129) Marcel van Gerven
Editors
Explainable and Interpretable
Models in Computer Vision
and Machine Learning
123
Editors
HugoJairEscalante SergioEscalera
INAOE UniversityofBarcelona
Puebla,Mexico Barcelona,Spain
IsabelleGuyon XavierBaró
INRIA,UniversitéParisSud,Université OpenUniversityofCatalonia
ParisSaclay,Paris,France Barcelona,Spain
ChaLearn
UmutGüçlü
Berkeley,CA,USA
RadboudUniversityNijmegen
Nijmegen,TheNetherlands
Yag˘murGüçlütürk
RadboudUniversityNijmegen
Nijmegen,TheNetherlands
MarcelvanGerven
RadboudUniversityNijmegen
Nijmegen,TheNetherlands
ISSN2520-131X ISSN2520-1328 (electronic)
TheSpringerSeriesonChallengesinMachineLearning
ISBN978-3-319-98130-7 ISBN978-3-319-98131-4 (eBook)
https://doi.org/10.1007/978-3-319-98131-4
LibraryofCongressControlNumber:2018962179
©SpringerNatureSwitzerlandAG2018
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Foreword
“Toomuchofablackboxtome”.Thatisanoftenheardandlong-standingcriticism
of data-drivenmachine learning methods, in general, and (deep) neuralnetworks,
inparticular.Nevertheless,astoundingresultshavebeenobtainedwiththeseblack
boxes.
Interestingly, one could argue that this is, to a large extent, not in spite of but
ratherthankstotheirblackboxnature:researchersnolongeraimatfullcontrolover
themodelintrinsics,acommonpracticeinthehand-craftedfeaturesera.Instead,the
dataleadstheway,optimisingthewholesysteminanend-to-endmannerforthetask
athand,yieldingsuperiorresults.
The flip side of the coin is that, giventhe complexityof the modelsused these
days,with millionsofparameters,itishardto understandthe processesinsidethe
box. As a consequence,the question rises whether such systems can be trusted at
all–especiallywhenitcomestosafety-criticalapplicationssuchasself-drivingcars
ormedicalimageinterpretation.
Three more observations further add to this scepticism. First, networks often
struggle to generalise beyond the circumstances seen at training time. Yet, they
keepmaking(oftenwrong)predictionswithhighconfidenceforout-of-distribution
samples. Second, there is the issue with adversarial examples, where it has been
shownthataddingrelativelylowamountsofnoisesufficestochangetheoutputof
aneuralnetworkinanarbitrarypredefineddirection.Finally,withartificialsystems
reachingorevensurpassinghumanperformance,thelong-standingcriticismofthe
blackboxapproachnowbecomesmorerelevantthanever.
After the initial enthusiasm at the start of the third summer of AI about the
goodperformanceobtainedwithdeeplearning,moreandmoreconcernsareraised
along the lines sketched above. As a countermeasure, we need more research
towardsmodelexplainabilityandinterpretability.Letusbuildanewgenerationof
machine learning models that are capable not only of predicting the output with
high accuracybut also of explainingthe producedresult and enablingresearchers
to interpret the learned models. This is a challenging endeavour, with several
openresearchquestions:Howtovisualiseorcommunicatemodelexplanationsand
interpretations with the user? How to avoid a misguided feeling of trust? How
v
vi Foreword
to evaluate model explanations and interpretations? How to avoid or deal with
subjectivity in this matter? Within this book, a fine collection of the current state
oftheartinthisdirectionisbroughttogether,highlightingdifferentapproachesto
tackletheproblem.
KULeuven,Flanders,Belgium TinneTuytelaars
June2018
Preface
Researchprogressincomputervisionandpatternrecognitionhasledtoavarietyof
modellingtechniqueswith (almost) human-likeperformancein a variety of tasks.
A clear example of this type of models is neural networks, whose deep variants
dominatethe arenasof computervision amongother fields. Althoughthis type of
modelshasobtainedastoundingresultsinavarietyoftasks(e.g.facerecognition),
theyarelimitedintheirexplainabilityandinterpretability.Thatis,ingeneral,users
cannotsaytoomuchabout:
(cid:129) Whatistherationalebehindthedecisionmade?(explainability)
(cid:129) Whatinthemodelstructureexplainsitsfunctioning?(interpretability)
Hence,while goodperformanceis a criticalrequiredcharacteristicforlearning
machines,explainability/interpretabilitycapabilitiesarehighlyneededifonewants
to take learning machines to the next step and, in particular, include them into
decision support systems involving human supervision (for instance, in medicine
orinsecurity).Becauseoftheircriticalimportance,thereisaresearchtrendwithin
thecomputervisionandmachinelearningcommunitiesinstudyingbothaspects.In
fact,inrecentyears,muchworkhasbeendevotedtodefiningwhatisexplainability
and interpretability in the context of models and how to evaluate these aspects,
proposing and analysing mechanisms for explaining recommendations of models
andinterpretingtheirstructure.
All this progress puts us in perfect time to compile in a single book the
latest research advanceson explainableand interpretablemodelsin the contextof
computervisionandmachinelearning.Thebookisdividedintofourpartsthatcover
complimentaryandrelevanttopicsaroundthissubject.
Part I focuseson generalnotionsand conceptsaroundexplainabilityand inter-
pretability.F.Doshi-VelezandKimelaborateonconsiderationsfortheevaluationof
interpretablemachinelearningmodels.Theyprovideadefinitionofinterpretability,
principlesforevaluationandataxonomyofevaluationapproaches.Theyconclude
with recommendations for researchers in the field. In the same line, Ras et al.
elaborateonissuesregardingdeeplearningandexplainability,tryingtobridgeagap
vii
viii Preface
betweenexpertusersandlay/averageusers.Theydiscusstherelationbetweenusers
lawsandregulations,explanationsandmethodsinthecontextofexplainability.
The secondpart of the bookis devoted to chaptersthat focuson explainability
andinterpretabilityfromthemachinelearningpointofview.Goudetetal.describe
Causal GenerativeNeuralNetworks,a methodologyto infercausal relationsfrom
observationaldata.Morespecifically,theyprovideameanstoestimateagenerative
modelofthejointdistributionofobservedvariables.Sincecausalityistheultimate
explanatory mechanism desired for most modelling techniques, this methodology
can have a great impact into the field. Loza Mencia et al. contribute to the book
with a chapter on rule-based methods for multi-label classification. The chapter
emphasises the interpretability characteristics of rule-based approaches to multi-
label classification, and two approachesfor learning predictive rule-based models
are reviewed in detail. Rieger et al. study the relationship between performance
and quality of explainability in the context of deep neural networks. They aim to
determinewhetherexplanationsexhibitasystematic biasandhowthestructureof
theneuralnetworkcanbeadaptedtoreducesuchbias.
The third part of the book focuses on explainability and interpretability in
computer vision. Akata el al. describe a novelmethodologyfor generatingexpla-
nations in image-object classification. The key features of the proposed method
are a relevance loss that conditions sentence generation on the image category
and, on the other hand, a discriminative loss inspired on reinforcement learning
that relies on a sentence classifier. N. Fatema and R. Mooney present a novel
methodologyto generateexplanationsforensemblesof visualquestionanswering
(VQA) systems. In addition, two evaluation protocols are described and used to
evaluate explanations generated by their ensemble. This is among the first works
dealing with explanation of ensembles of VQA systems.J. Kim and J. Canny
describeamethodologyforgeneratingvisuallyinterpretableimagesinthecontext
ofautonomousvehicledriving.Themethodologycomprisestwosteps:aCNNwith
attentionmodelthathighlightspotentiallysalientregionsin imagesanda filtering
stepthataimsatremovingspurioussalientregions.Themethodologyisextensively
evaluated,comprisingqualitativeandquantitativeassessments.
Last but not least, Part IV covers methodologies related to explainability and
interpretability in the context of first impressions and job candidate screening.
Liemet al. elaborateon the gapbetweenmachinelearning(andcomputerscience
in general) and psychology in the context of job candidate screening. Through a
detailed review, the authors try to fill an understanding gap between both areas.
Liem at al. describe their solution to the job candidate screening competition. H.
KayaandA.Salahdescribethewinningmethodologyofthejobcandidatescreening
competition.Theauthorsfocuson theexplanatorycharacteristicsoftheir solution
anddiscussthe potentialbiasof theirmodel.Similarly,Aakuretal. describetheir
winning methodology for the explainable job candidate screening challenge. The
authorsprovideandetaileddescriptionoftheirmethodandanindepthanalysisof
theirresults.
Preface ix
To the best of our knowledge, this is the first compilation of research on this
topic.Wewerefortunatetogather11chaptersofextraordinaryqualitythat,together,
captureasnapshotofthestateoftheartinthisprettymuchimportanttopic.
Puebla,Mexico HugoJairEscalante
Barcelona,Spain SergioEscalera
Paris,France IsabelleGuyon
Barcelona,Spain XavierBaró
Nijmegen,TheNetherlands Yag˘murGüçlütürk
Nijmegen,TheNetherlands UmutGüçlü
Nijmegen,TheNetherlands MarcelvanGerven
June2018
