Table Of ContentThe Information Retrieval Series
Andrea Esuli
Alessandro Fabris
Alejandro Moreo
Fabrizio Sebastiani
Learning
to Quantify
The Information Retrieval Series
Volume 47
SeriesEditors
ChengXiangZhai,UniversityofIllinois,Urbana,IL,USA
MaartendeRijke,UniversityofAmsterdam,TheNetherlandsandAholdDelhaize,
Zaandam,TheNetherlands
EditorialBoardMembers
NicholasJ.Belkin,RutgersUniversity,NewBrunswick,NJ,USA
CharlesClarke,UniversityofWaterloo,Waterloo,ON,Canada
DianeKelly,UniversityofTennesseeatKnoxville,Knoxville,TN,USA
FabrizioSebastiani ,ConsiglioNazionaledelleRicerche,Pisa,Italy
InformationRetrieval (IR) deals with access to and search in mostly unstructured
information,in text, audio, and/orvideo, either from one large file or spread over
separateanddiversesources,instaticstoragedevicesaswellasonstreamingdata.
Itispartofbothcomputerandinformationscience,andusestechniquesfrome.g.
mathematics,statistics, machinelearning,databasemanagement,orcomputational
linguistics. Information Retrieval is often at the core of networked applications,
web-baseddatamanagement,orlarge-scaledataanalysis.
The Information Retrieval Series presents monographs, edited collections, and
advancedtextbooksontopicsofinterestforresearchersinacademiaandindustry
alike. Its focus is on the timely publication of state-of-the-art results at the
forefrontofresearchandontheoreticalfoundationsnecessarytodevelopadeeper
understandingofmethodsandapproaches.
Thisseriesisabstracted/indexedinEICompendexandScopus.
Andrea Esuli • Alessandro Fabris •
Alejandro Moreo • Fabrizio Sebastiani
Learning
to Quantify
AndreaEsuli AlessandroFabris
IstitutodiScienzaeTecnologie DipartimentodiIngegneria
dell’Informazione dell’Informazione
ConsiglioNazionaledelleRicerche UniversitàdiPadova
Pisa,Italy Padova,Italy
AlejandroMoreo FabrizioSebastiani
IstitutodiScienzaeTecnologie IstitutodiScienzaeTecnologie
dell’Informazione dell’Informazione
ConsiglioNazionaledelleRicerche ConsiglioNazionaledelleRicerche
Pisa,Italy Pisa,Italy
ThisworkwassupportedbyIstitutodiScienzaeTecnologiedell’Informazione
ISSN1871-7500 ISSN2730-6836 (electronic)
TheInformationRetrievalSeries
ISBN978-3-031-20466-1 ISBN978-3-031-20467-8 (eBook)
https://doi.org/10.1007/978-3-031-20467-8
©TheEditor(s)(ifapplicable)andTheAuthor(s)2023.Thisbookisanopenaccesspublication.
Open Access This bookis licensed under the terms of the Creative Commons Attribution 4.0Inter-
nationalLicense(http://creativecommons.org/licenses/by/4.0/),whichpermitsuse,sharing,adaptation,
distribution andreproduction inanymediumorformat,aslong asyougive appropriate credit tothe
originalauthor(s)andthesource,providealinktotheCreativeCommonslicenseandindicateifchanges
weremade.
Theimages or other third party material in this book are included in the book’s Creative Commons
license,unlessindicatedotherwiseinacreditlinetothematerial.Ifmaterialisnotincludedinthebook’s
CreativeCommonslicenseandyourintendeduseisnotpermittedbystatutoryregulationorexceedsthe
permitteduse,youwillneedtoobtainpermissiondirectlyfromthecopyrightholder.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Policymakers orcomputerscientistsmaybe
interestedinfindingtheneedleinthe
haystack(...), butsocialscientistsaremore
commonlyinterestedin characterizingthe
haystack.
(Daniel J. Hopkinsand Gary King,2010)
Preface
Inanumberofapplicationsinvolvingclassification,thefinalgoalisnotdetermining
which class (or classes) individual unlabelled instances belong to, but estimating
theprevalence(or“relativefrequency”,or“priorprobability”)ofeachclassinthe
unlabelleddata.Inrecentyearsithasbeenpointedoutthat,inthesecases,itwould
make sense to directly optimise machine learning algorithms for this goal, rather
than(somehowindirectly)justoptimisingtheclassifier’sabilitytolabelindividual
instances.Thetaskoftrainingestimatorsofclassprevalenceviasupervisedlearning
isknownaslearningtoquantify,or,moresimply,quantification.Itisbynowwell
knownthatperformingquantificationbyclassifyingeachunlabelledinstanceviaa
standard classifier and then counting the instances that have been assigned to the
class (the Classify and Count method) usually leads to biased estimators of class
prevalence,i.e.,topoorquantificationaccuracy;asaresult,methods(andevaluation
measures)thataddressquantificationasataskinitsownrighthavebeendeveloped.
This book covers the main applications of quantification, the main methods that
havebeendevelopedforlearningtoquantify,themeasuresthathavebeenadopted
forevaluatingit,andthechallengesthatstillneedtobeaddressedbyfutureresearch.
The book is divided in seven chapters. Chapter 1 sets the stage for the rest
of the book by introducing fundamental notions such as class distributions, their
estimation,anddatasetshift,byarguingforthesuboptimalityofusingclassification
techniques for performing this estimation, and by discussing why learning to
quantify has evolved as a task of its own, rather than remaining a by-product of
classification.Chapter2providesthemotivationforwhatistocomebydescribing
the applications that quantification has been put at, ranging from improving clas-
sificationaccuracyin domainadaptation,to measuringandimprovingthefairness
ofclassificationsystemswithrespecttoasensitiveattribute,tosupportingresearch
anddevelopmentin the social sciences, in politicalscience, epidemiology,market
research,andothers.InChapter3wemoveontodiscusstheexperimentalevaluation
of quantification systems; we look at evaluation measures for the various types
of quantification systems (binary, single-label multiclass, multi-label multiclass,
ordinal),butalso atevaluationprotocolsforquantification,thatessentially consist
in ways to extract multiple testing samples for use in quantification evaluation
vii
viii Preface
from a single classification test set. Chapter 4 is possibly the central chapter of
the book, and looks at the various supervised learning methods for learning to
quantifythat have been proposedover the years, be they of an aggregativenature
(i.e., methods that require the classification of all individual unlabelled items as
an intermediate step) or of a non-aggregative nature (i.e., methods in which no
classificationofindividualitemsisperformed).InChapter5welookatanumberof
“advanced”(orniche)topics in quantification,includingquantificationforordinal
data,cross-lingualquantificationoftextualitems,quantificationfornetworkeddata,
and quantification for streaming data. Chapter 6 looks at other aspects of the
“quantificationlandscape”thathavenotbeencoveredinthepreviouschapters,and
discusses the evolutionof quantificationresearch,from its beginningsto the most
recent quantification-based “shared tasks”, the landscape of quantification-based,
publicly available software libraries, and other tasks in data science that present
important similarities with quantification. Chapter 6 also presents the results of
experiments, that we have carried out ourselves, in which we compare many of
themethodsdiscussedinChapter4onacommontestinginfrastructure.Chapter7
concludesthebook,pointingtopotentialfuturedevelopmentsinthequantification
arena.
The bookismostly addressedto researchersin data science thatmightwantto
come up to speed with the state of the art in learning to quantify, but it can be
useful also to researchers and scientists that operate in other disciplines and that
applytechniquesfromdatascience to theirownapplicationdomains.Indeed,itis
ourexperiencethatmanypotentialusersofquantificationtechniques(whooperate
inthefieldstoucheduponinChapter2,andpossiblyinotherstoo)donotusethem,
thussettlingforsuboptimal“classifyandcount”techniques,forthesimplefactthat
theyarenotawareoftheirexistence,andoftheexistenceofquantificationasatask
ofitsown;itisalsothosepotentialusersthatwehopewillbeinspiredbythisbook.
We thus hope that the availability of a book that surveys all aspects of the
quantification workflow and presents them in a hopefully accessible form, will
increasetheinterestinthissubjectonthepartofresearchersandpractitionersalike,
andwillcontributetomakingquantificationbetterknowntopotentialusersofthis
technologyandtoresearchersinterestedinadvancingthefield.
Pisa,Italy AndreaEsuli
Padova,Italy AlessandroFabris
Pisa,Italy AlejandroMoreo
Pisa,Italy FabrizioSebastiani
Acknowledgments
The work of Andrea Esuli, Alejandro Moreo, and Fabrizio Sebastiani has been
supported by the SoBigData++ project, funded by the European Commission
(Grant871042)undertheH2020ProgrammeINFRAIA-2019-1,bytheAI4Media
project, funded by the European Commission (Grant 951911) under the H2020
Programme ICT-48-2020, and by the SoBigData.it and FAIR projects, funded
by the Italian Ministry of University and Research under the NextGenerationEU
program. The authors’ opinions do not necessarily reflect those of the European
Commission. The work by Alessandro Fabris was supported by MIUR (Italian
Ministry for University and Research) under the “Departments of Excellence”
initiative(Law232/2016).
ix
Contents
1 TheCaseforQuantification ................................................ 1
1.1 ClassDistributionsandTheirEstimation.............................. 2
1.2 TheSuboptimalityofClassifyandCount ............................. 3
1.3 NotationalConventions................................................. 5
1.4 QuantificationProblems................................................ 6
1.5 DatasetShiftandQuantification........................................ 8
1.5.1 Types of Dataset Shift and Their Relation
toQuantification............................................... 11
1.6 QuantificationandBiasMitigation..................................... 14
1.7 StructureofThisBook.................................................. 16
2 ApplicationsofQuantification.............................................. 19
2.1 ImprovingClassificationAccuracy .................................... 19
2.1.1 WordSenseDisambiguation.................................. 21
2.2 Fairness.................................................................. 22
2.2.1 ImprovingFairness............................................ 22
2.2.2 MeasuringFairness............................................ 23
2.3 SentimentAnalysis ..................................................... 24
2.4 SocialandPoliticalSciences........................................... 25
2.5 MarketResearch ........................................................ 27
2.6 Epidemiology ........................................................... 28
2.7 EcologicalModelling................................................... 29
2.8 ResourceAllocation .................................................... 31
3 EvaluationofQuantificationAlgorithms.................................. 33
3.1 MeasuresforEvaluatingSLQ,BQ,andMLQ......................... 34
3.1.1 PropertiesofEvaluationMeasuresforSLQ,BQ,
andMLQ....................................................... 35
3.1.2 Bias ............................................................ 37
3.1.3 AbsoluteErroranditsVariants............................... 37
3.1.4 RelativeAbsoluteErroranditsVariants ..................... 38
xi
