Federated Learning: A Comprehensive Overview of Methods and Applications

531 Pages·2022·13.823 MB·English

by  Heiko Ludwig (editor), Nathalie Baracaldo (editor)

Heiko Ludwig
Nathalie Baracaldo   Editors
Federated
Learning
A Comprehensive Overview of Methods
and Applications
Federated Learning
Heiko Ludwig • Nathalie Baracaldo
Editors
Federated Learning
A Comprehensive Overview of Methods
and Applications
Editors
HeikoLudwig NathalieBaracaldo
IBMResearch–Almaden IBMResearch–Almaden
SanJose,CA,USA SanJose,CA,USA
ISBN978-3-030-96895-3 ISBN978-3-030-96896-0 (eBook)
https://doi.org/10.1007/978-3-030-96896-0
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerland
AG2022
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Machine learning has made great strides over the past two decades and has been
adoptedinmanyapplicationdomains.Successfulmachinelearningdependslargely
onaccesstoqualitydata,bothlabeledandunlabeled.
Concerns related to data privacy, security, and sovereignty have caused public
and technical discussion on how to use data for machine learning purposes
consistentwithregulatoryandstakeholderinterests.Theseconcernsandlegislation
have led to the realization that collecting training data in large central repositories
maybeatoddswithmaintainingprivacyfordataowners.
While distributed learning or model fusion has been discussed since at least
a decade, federated machine learning (FL) as a concept has been popularized by
MacMahanandotherssince2017.Inthesubsequentyears,muchresearchhasbeen
conducted – both, in academia and the industry – and, at the time of writing this
book,thefirstviablecommercialframeworksforfederatedlearningarecomingto
themarket.
Thisbookaimstocapturetheresearchprogressandstateoftheartthathasbeen
made in the past years, from the initial conception of the field to first applications
and commercial use. To get this broad and deep overview, we invited leading
researchers to address the different perspectives of federated learning: the core
machinelearningperspective,privacyandsecurity,distributedsystems,andspecific
applicationdomains.
The book’s title, Federated Learning: A Comprehensive Overview of Methods
and Applications, outlines its scope. It presents in depth the most important
issuesandapproachestofederatedlearningforresearchersandpractitioners.Some
chapters contain a variety of technical content that is relevant to understand the
intricaciesofthealgorithmsandparadigmsthatmakeitpossibletodeployfederated
learninginmultipleenterprisesettings.Otherchaptersfocusonprovidingclarityon
howtoselectprivacyandsecuritysolutionsinawaythatcanbetailoredtospecific
usecases,whileotherstakeintoconsiderationthepragmaticsofthesystemswhere
thefederatedlearningprocesswillrun.
Giventheinherentcross-disciplinarynatureofthetopic,weencounterdifferent
notational conventions in different chapters of the book. What might be parties in
v
vi Preface
federatedmachinelearningmaybecalledclientsinthedistributedsystemsperspec-
tives.Intheintroductorychapterofthisbook,welayouttheprimaryterminology
weuse,andeachchapterexplainshowthediscipline-specificterminologymapsto
the common one when it is introduced, if this is the case. With this approach, we
makethisbookunderstandabletoreadersfromdiversebackgroundswhilestaying
truetotheconventionsofthespecificdisciplinesinvolved.
Taken as a whole, this book enables the reader to get a broad state-of-the-art
summaryofthemostrecentresearchdevelopments.
Editingthisbook,andwritingsomeofthechapters,requiredthehelpofmany,
who we want to acknowledge. IBM Research gave us the opportunity to work in
thisexcitingfield,notjustacademicallybutalsotoputthistechnologyintopractice
andmakeitpartofaproduct.Welearnedinvaluablelessonsonthejourney,andwe
havemuchtothanktoourcolleaguesatIBM.Inparticular,wewanttoacknowledge
ourdirector,SandeepGopisetty,forgivingusthespacetoworkonthisbook:Gegi
Thomas,whomadesureourresearchcontributionsmaketheirwayintotheproduct:
andourteammembers.
Thechapterauthorsprovidethesubstanceofthisbookandwerepatientwithus
withrequestsforchangestotheirchapters.
We owe greatest thanks to our families, who patiently put up with us devoting
time to the book rather than them over the year of writing and editing this book.
Heiko is deeply thankful to his wife, Beatriz Raggio, for making these sacrifices
andsupportinghimthroughout.Nathalieisprofoundlythankfultoherhusbandand
sons, Santiago and Matthias Bock, for their love and support and for cheering for
allherprojects,includingthisone.Shealsothanksherparents,AdrianaandJesus;
thisandmanymoreachievementswouldnotbepossiblewithouttheiramazingand
continuoussupport.
SanJose,CA,USA HeikoLudwig
September2021 NathalieBaracaldo
Contents
1 IntroductiontoFederatedLearning...................................... 1
HeikoLudwigandNathalieBaracaldo
PartI FederatedLearningasaMachineLearningProblem
2 Tree-BasedModelsforFederatedLearningSystems................... 27
YuyaJeremyOng,NathalieBaracaldo,andYiZhou
3 SemanticVectorization:Text-andGraph-BasedModels.............. 53
ShalishaWitherspoon,DeanSteuer,andNirmitDesai
4 PersonalizationinFederatedLearning................................... 71
MayankAgarwal,MikhailYurochkin,andYuekaiSun
5 Personalized,RobustFederatedLearningwithFed+ .................. 99
PengqianYu,AchintyaKundu,LauraWynter,andShiauHongLim
6 Communication-EfficientDistributedOptimizationAlgorithms ..... 125
GauriJoshiandShiqiangWang
7 Communication-EfficientModelFusion................................. 145
MikhailYurochkinandYuekaiSun
8 FederatedLearningandFairness......................................... 177
AnnieAbay,YiZhou,NathalieBaracaldo,andHeikoLudwig
PartII SystemsandFrameworks
9 IntroductiontoFederatedLearningSystems............................ 195
SyedZawad,FengYan,andAliAnwar
10 LocalTrainingandScalabilityofFederatedLearningSystems....... 213
SyedZawad,FengYan,andAliAnwar
11 StragglerManagement..................................................... 235
SyedZawad,FengYan,andAliAnwar
vii
viii Contents
12 SystemsBiasinFederatedLearning ..................................... 259
SyedZawad,FengYan,andAliAnwar
PartIII PrivacyandSecurity
13 ProtectingAgainstDataLeakageinFederatedLearning:
WhatApproachShouldYouChoose?.................................... 281
NathalieBaracaldoandRunhuaXu
14 PrivateParameterAggregationforFederatedLearning............... 313
K.R.JayaramandAshishVerma
15 DataLeakageinFederatedLearning .................................... 337
XiaoJin,Pin-YuChen,andTianyiChen
16 SecurityandRobustnessinFederatedLearning........................ 363
Ambrish Rawat, Giulio Zizzo, Muhammad Zaid Hameed,
andLuisMuñoz-González
17 DealingwithByzantineThreatstoNeuralNetworks................... 391
YiZhou,NathalieBaracaldo,AliAnwar,andKamalaVarma
PartIV Beyond Horizontal Federated Learning: Partitioning
ModelsandDatainDiverseWays
18 Privacy-PreservingVerticalFederatedLearning ....................... 417
Runhua Xu, Nathalie Baracaldo, Yi Zhou, Annie Abay,
andAliAnwar
19 Split Learning: A Resource Efficient Model and Data
ParallelApproachforDistributedDeepLearning...................... 439
PraneethVepakommaandRameshRaskar
PartV Applications
20 FederatedLearningforCollaborativeFinancialCrimes
Detection..................................................................... 455
ToyotaroSuzumura,YiZhou,RyoKawahara,NathalieBaracaldo,
andHeikoLudwig
21 FederatedReinforcementLearningforPortfolioManagement....... 467
PengqianYu,LauraWynter,andShiauHongLim
22 ApplicationofFederatedLearninginMedicalImaging ............... 483
Ehsan Degan, Shafiq Abedin, David Beymer, Angshuman Deb,
NathanielBraman,BenediktGraf,andVandanaMukherjee
23 AdvancingHealthcareSolutionswithFederatedLearning............ 499
AmoghKamatTarcar
Contents ix
24 APrivacy-preservingProductRecommenderSystem ................. 509
TuanM.HoangTrong,MudhakarSrivatsa,andDineshVerma
25 ApplicationofFederatedLearninginTelecommunications
andEdgeComputing....................................................... 523
UtpalMangla
Chapter 1
Introduction to Federated Learning
HeikoLudwigandNathalieBaracaldo
Abstract Federated learning (FL) is an approach to machine learning in which
the training data is not managed centrally. Data is retained by data parties that
participate in the FL process and is not shared with any other entity. This makes
FLanincreasinglypopularsolutionformachinelearningtasksforwhichbringing
datatogetherinacentralizedrepositoryisproblematic,eitherforprivacy,regulatory,
orpracticalreasons.Inthischapter,weintroducethebasicconceptsofFL,provide
an overview of its application use cases, and discuss it from a machine learning,
distributedcomputing,andprivacyperspective.Wealsoprovideanintroductionto
divedeepintothemattercoveredinthesubsequentchapters.
1.1 Overview
Machine learning (ML) has become a crucial technique to develop cognitive and
analytic functionality difficult and ineffective to develop algorithmically. Applica-
tions in computer vision, speech recognition, and natural language understanding
progressedbyleapsandboundswiththeadventofDeepNeuralNetworks(DNNs)
and the computational hardware to train complex networks effectively. Also,
classicalMLtechniquessuchasdecisiontrees,linearregression,andsupportvector
models(SVMs)foundincreaseduse,inparticularrelatedtostructureddata.
The application of ML critically depends on the availability of high-quality
training data. Sometimes, though, privacy considerations prevent training data to
be brought to a central data repository to be curated and managed for the ML
process.Federatedlearning(FL)isanapproachfirstproposedbythisnamein[28]
totrainMLmodelsontrainingdataindisparatelocations,notrequiringthecentral
collectionofdata.
H.Ludwig((cid:2))·N.Baracaldo
IBMResearch–Almaden,SanJose,CA,USA
e-mail:[email protected];[email protected]
©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2022 1
H.Ludwig,N.Baracaldo(eds.),FederatedLearning,
https://doi.org/10.1007/978-3-030-96896-0_1