Federated Learning: Fundamentals and Advances

227 Pages·2022·5.849 MB·English

by  Yaochu Jin, Hangyu Zhu, Jinjin Xu, Yang Chen

Machine Learning: Foundations, Methodologies,
and Applications
Yaochu Jin · Hangyu Zhu ·
Jinjin Xu · Yang Chen
Federated
Learning
Fundamentals and Advances
Machine Learning: Foundations, Methodologies,
and Applications
SeriesEditors
KayChenTan,DepartmentofComputing,HongKongPolytechnicUniversity,
HongKong,China
DachengTao,UniversityofTechnology,Sydney,Australia
Bookspublishedinthisseriesfocusonthetheoryandcomputationalfoundations,
advanced methodologies and practical applications of machine learning, ideally
combiningmathematicallyrigoroustreatmentsofacontemporarytopicsinmachine
learningwithspecificillustrationsinrelevantalgorithmdesignsanddemonstrations
in real-world applications. The intended readership includes research students and
researchersincomputerscience,computerengineering,electricalengineering,data
science,andrelatedareasseekingaconvenientmediumtotracktheprogressesmade
inthefoundations,methodologies,andapplicationsofmachinelearning.
Topicsconsideredincludeallareasofmachinelearning,includingbutnotlimited
to:
(cid:129)
Decisiontree
(cid:129)
Artificialneuralnetworks
(cid:129)
Kernellearning
(cid:129)
Bayesianlearning
(cid:129)
Ensemblemethods
(cid:129)
Dimensionreductionandmetriclearning
(cid:129)
Reinforcementlearning
(cid:129)
Metalearningandlearningtolearn
(cid:129)
Imitationlearning
(cid:129)
Computationallearningtheory
(cid:129)
Probabilisticgraphicalmodels
(cid:129)
Transferlearning
(cid:129)
Multi-viewandmulti-tasklearning
(cid:129)
Graphneuralnetworks
(cid:129)
Generativeadversarialnetworks
(cid:129)
Federatedlearning
Thisseriesincludesmonographs,introductory,andadvancedtextbooks,andstate-
of-the-artcollections.Furthermore,itsupportsOpenAccesspublicationmode.
· · ·
Yaochu Jin Hangyu Zhu Jinjin Xu Yang Chen
Federated Learning
Fundamentals and Advances
YaochuJin HangyuZhu
FacultyofTechnology DepartmentofArtificialIntelligence
BielefeldUniversity andComputerScience
Bielefeld,Germany JiangnanUniversity
Wuxi,China
JinjinXu
IntelligentPerceptionandInteraction YangChen
ResearchDepartment SchoolofElectricalEngineering
OPPOResearchInstitute ChinaUniversityofMiningandTechnology
Shanghai,China Xuzhou,China
ISSN 2730-9908 ISSN 2730-9916 (electronic)
MachineLearning:Foundations,Methodologies,andApplications
ISBN 978-981-19-7082-5 ISBN 978-981-19-7083-2 (eBook)
https://doi.org/10.1007/978-981-19-7083-2
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SingaporePteLtd.2023
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
Iheardtheterminology“federatedlearning”forthefirsttimewhenIwaslistening
toatalkgivenbyDr.CatherineHuangfromIntelataworkshopoftheIEEESympo-
siumSeriesonComputationalIntelligenceinDecember2016inAthens,Greece.I
was fascinated by the idea of federated learning and immediately recognized the
paramount importance of preserving data privacy, since deep learning had been
increasinglyrelyingonthecollectionofalargeamountofdata,eitherfromindustrial
processesorfromhumaneverydaylife.
Federatedlearninghasnowbecomeapopularresearchareainmachinelearning
andreceivedincreasedinterestfrombothindustryandgovernment.Actuallyalready
inApril2016,theEuropeanUnionadoptedtheGeneralDataProtectionRegulation,
whichisthemoststrictlawondataprotectionandprivacyintheEuropeanUnionand
the European Economic Area and took effective in May 2018. Consequently, data
privacyandsecurityhasbecomeindispensableforpracticalapplicationsofmachine
learning and artificial intelligence, together with other importance requirements,
Includingexplainability,fairness,accountability,robustnessandreliability.
Thisbookpresentsacompactyetcomprehensiveandupdatedcoverageoffunda-
mentals and recent advances in federated learning. The book is self-contained
andsuitedforbothpostgraduatestudents,researchers,andindustrialpractitioners.
Chapter 1 starts with an introduction to the most popular neural network models
andgradient-basedlearningalgorithms,evolutionaryalgorithms,evolutionaryopti-
mization, and multi-objective evolutionary learning. Then, three main privacy-
preservingcomputingtechniques,includingsecuremulti-partycomputation,differ-
ential privacy, and homomorphic encryption are described. Finally, the basics of
federatedlearning,includingacategoryoffederatedlearningalgorithms,thevanilla
federated averaging algorithm, federated transfer learning, and main challenges
in federated learning over non independent and identically distributed data are
presented.Chapter2focusesoncommunicationefficientfederatedlearning,which
aimstoreducethecommunicationcostsinfederatedlearningwithoutdeteriorating
the learning performance. Two algorithms for reducing communication overhead
in federated learning are detailed. The first algorithm takes advantage of the fact
that shallow layers in a deep neural network are responsible for learning general
v
vi Preface
featuresacrossdifferentclasseswhiledeeplayerstakecareofclass-specificfeatures.
Consequently,thedeeplayerscanbeupdatedlessfrequentlythanthedeeplayers,
therebyreducingthenumberofparametersthatneedtobeuploadedanddownloaded.
The second algorithm adopts a completely different approach, which dramatically
decreases the communication load by converting the weights in real numbers into
ternaryvalues.Here,thekeyquestionishowtomaintainthelearningperformance
afterternarycompressionoftheweights,whichisachievedbytrainingareal-valued
co-efficientforeachlayer.Interestingly,weareabletoprovetheoreticallythatmodel
divergencecanevenbemitigatedbyintroducingternaryquantizationoftheweights,
inparticularwhenthedataisnotindependentandidenticallydistributed.Whilelayer-
wisesynchronousweightupdateandternarycompressiondescribedinChap.2can
effectively lower the communication cost, Chap. 3 addresses communication cost
bysimultaneouslymaximizingthelearningperformanceandminimizingthemodel
complexity (e.g., the number of parameters in the model) using a multi-objective
evolutionaryalgorithm.Togoastepfurther,areal-timemulti-objectiveevolutionary
federatedlearningalgorithmisgiven,whichsearchesforoptimalneuralarchitectures
bymaximizingtheperformance,minimizingthemodelcomplexity,andminimizing
thecomputationalperformance(indicatedbyfloatingpointoperationspersecond)at
thesametime.Tomakeitpossibleforreal-timeneuralarchitecturesearch,astrategy
thatsearchesforsubnetworksbysamplingasupernet,andreducingcomputational
costs by sampling clients is introduced. To enhance the security level of federated
learning,Chap.4elaboratestwosecurefederatedlearningalgorithmsbyintegrating
homomorphicencryptionanddifferentialprivacytechniqueswithfederatedlearning.
Thefirstalgorithmisbasedonahorizontalfederatedlearning,inwhichadistributed
encryptionalgorithmisappliedtotheweightstobeuploadedontopofternaryquan-
tization. By contrast,the second algorithm is meant for vertical federated learning
basedondecisiontrees,inwhichsecurenodesplitandconstructionaredeveloped
basedonhomomorphicencryptionandpredictedlabelsareaggregatedonthebasis
of partial differential privacy. This algorithm does not assume that all labels are
storedononeclient,makingitmorepracticalforreal-worldapplications.Thebook
isconcludedbyChap.5,providingasummaryofthepresentedalgorithms,andan
outlookoffutureresearch.
ThetwoalgorithmspresentedinChap.2weredevelopedbytwovisitingPh.D.
studentsIhostedatUniversityofSurrey,JinjinXuandYangCheng.Thetwomulti-
objectiveevolutionaryfederatedlearningalgorithmswereproposedbymyprevious
Ph.D. student, Hangyu Zhu. Finally, the two secure federated learning algorithms
weredesignedbyHangyuZhu,incollaborationwithDr.KaitaiLiang,andhisPh.D.
student,RuiWang,whowerewiththeDepartmentofComputerScience,University
ofSurrey,andarenowwiththeDepartmentofIntelligentSystems,DelftUniversity
ofTechnology,TheNetherlands.IwouldliketotakethisopportunitythanktoKaitai
andRuifortheircontributionstoChap.4.
Preface vii
Finally,IwouldliketoacknowledgethatmyresearchisfundedbyanAlexander
von Humboldt Professorship for Artificial Intelligence endowed by the German
FederalMinistryofEducationandResearch.
Bielefeld,Germany YaochuJin
August2022
Contents
1 Introduction ................................................... 1
1.1 ArtificialNeuralNetworksandDeepLearning .................. 1
1.1.1 ABriefHistoryofArtificialIntelligence ................ 1
1.1.2 Multi-layerPerceptrons ............................... 4
1.1.3 ConvolutionalNeuralNetworks ........................ 13
1.1.4 LongShort-TermMemory ............................ 20
1.1.5 DecisionTrees ....................................... 26
1.1.6 Gradient-BasedMethods .............................. 31
1.2 EvolutionaryOptimizationandLearning ....................... 36
1.2.1 OptimizationandLearning ............................ 36
1.2.2 GeneticAlgorithms .................................. 37
1.2.3 GeneticProgramming ................................ 42
1.2.4 EvolutionaryMulti-objectiveOptimization .............. 43
1.2.5 EvolutionaryMulti-objectiveLearning .................. 52
1.2.6 EvolutionaryNeuralArchitectureSearch ................ 54
1.3 Privacy-PreservingComputation .............................. 60
1.3.1 SecureMulti-partyComputation ....................... 61
1.3.2 DifferentialPrivacy .................................. 64
1.3.3 HomomorphicEncryption ............................. 66
1.4 FederatedLearning ......................................... 68
1.4.1 HorizontalandVerticalFederatedLearning .............. 68
1.4.2 FederatedAveraging .................................. 71
1.4.3 FederatedTransferLearning ........................... 73
1.4.4 FederatedLearningOverNon-IIDData ................. 76
1.5 Summary .................................................. 83
References ..................................................... 83
2 CommunicationEfficientFederatedLearning ..................... 93
2.1 CommunicationCostinFederatedLearning .................... 93
2.2 MainMethodologies ........................................ 94
ix
x Contents
2.2.1 Non-IID/IIDDataandDatasetShift .................... 94
2.2.2 Non-identicalClientDistributions ...................... 95
2.2.3 ViolationsofIndependence ............................ 95
2.3 TemporallyWeightedAveragingandLayer-WiseWeight
Update .................................................... 96
2.3.1 TemporallyWeightedAveraging ....................... 98
2.3.2 Layer-WiseAsynchronousWeightUpdate ............... 99
2.3.3 EmpiricalStudies .................................... 103
2.4 TrainedTernaryCompressionforFederatedLearning ............ 109
2.4.1 BinaryandTernaryCompression ....................... 110
2.4.2 TrainedTernaryCompression .......................... 113
2.4.3 TrainedTernaryCompressionforFederatedLearning ..... 114
2.4.4 TheoreticalAnalysis .................................. 119
2.4.5 EmpiricalStudies .................................... 126
2.5 Summary .................................................. 135
References ..................................................... 135
3 EvolutionaryMulti-objectiveFederatedLearning ................. 139
3.1 MotivationsandChallenges .................................. 139
3.2 OfflineEvolutionaryMulti-objectiveFederatedLearning ......... 140
3.2.1 SparseNetworkEncodingwithaRandomGraph ......... 141
3.2.2 EvolutionaryMulti-objectiveNeuralArchitecture
Search .............................................. 141
3.2.3 OverallFramework ................................... 144
3.2.4 EmpiricalResults .................................... 146
3.3 Real-Time Evolutionary Federated Neural Architecture
Search .................................................... 150
3.3.1 NetworkArchitectureEncodingBasedOnSupernet ....... 151
3.3.2 NetworkSamplingandClientSampling ................. 153
3.3.3 OverallFramework ................................... 154
3.3.4 EmpiricalStudies .................................... 157
3.4 Summary .................................................. 163
References ..................................................... 163
4 SecureFederatedLearning ...................................... 165
4.1 ThreatstoFederatedLearning ................................ 165
4.2 DistributedEncryptionforHorizontalFederatedLearning ........ 166
4.2.1 DistributedDataEncryption ........................... 167
4.2.2 FederatedEncryptionandDecryption ................... 168
4.2.3 TernaryQuantizationandApproximateAggregation ...... 173
4.2.4 OverallFramework ................................... 176
4.2.5 EmpiricalStudies .................................... 178
4.3 SecureVerticalFederatedLearning ........................... 185
4.3.1 VerticalFederatedLearningwithXGBoost .............. 187
4.3.2 SecureNodeSplitandConstruction .................... 192
4.3.3 PartialDifferentialPrivacy ............................ 196