Table Of ContentStudies in Computational Intelligence 1070
Yanan Sun
Gary G. Yen
Mengjie Zhang
Evolutionary
Deep Neural
Architecture Search:
Fundamentals,
Methods, and
Recent Advances
Studies in Computational Intelligence
Volume 1070
SeriesEditor
JanuszKacprzyk,PolishAcademyofSciences,Warsaw,Poland
The series “Studies in Computational Intelligence” (SCI) publishes new develop-
mentsandadvancesinthevariousareasofcomputationalintelligence—quicklyand
withahighquality.Theintentistocoverthetheory,applications,anddesignmethods
of computational intelligence, as embedded in the fields of engineering, computer
science, physics and life sciences, as well as the methodologies behind them. The
series contains monographs, lecture notes and edited volumes in computational
intelligence spanning the areas of neural networks, connectionist systems, genetic
algorithms,evolutionarycomputation,artificialintelligence,cellularautomata,self-
organizingsystems,softcomputing,fuzzysystems,andhybridintelligentsystems.
Ofparticularvaluetoboththecontributorsandthereadershiparetheshortpublica-
tiontimeframeandtheworld-widedistribution,whichenablebothwideandrapid
disseminationofresearchoutput.
This series also publishes Open Access books. A recent example is the book
Swan,Nivel,Kant,Hedges,Atkinson,Steunebrink:TheRoadtoGeneralIntelligence
https://link.springer.com/book/10.1007/978-3-031-08020-3.
IndexedbySCOPUS,DBLP,WTIFrankfurteG,zbMATH,SCImago.
AllbookspublishedintheseriesaresubmittedforconsiderationinWebofScience.
· ·
Yanan Sun Gary G. Yen Mengjie Zhang
Evolutionary Deep Neural
Architecture Search:
Fundamentals, Methods,
and Recent Advances
YananSun GaryG.Yen
DepartmentofArtificialIntelligence, SchoolofElectricalandComputer
CollegeofComputerScience Engineering,IntelligentSystems
SichuanUniversity andControlLaboratory
Chengdu,Sichuan,China OklahomaStateUniversity
Stillwater,OK,USA
MengjieZhang
SchoolofEngineeringandComputer
Science
VictoriaUniversityofWellington
Wellington,NewZealand
ISSN 1860-949X ISSN 1860-9503 (electronic)
StudiesinComputationalIntelligence
ISBN 978-3-031-16867-3 ISBN 978-3-031-16868-0 (eBook)
https://doi.org/10.1007/978-3-031-16868-0
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SwitzerlandAG2023
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsofreprinting,reuseofillustrations,
recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor
informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
The use of Evolutionary Computation (EC) methods to create optimal or nearly
optimalDeepNeuralNetwork(DNN)architecturesisreferredtoasevolutionarydeep
neural architecture design. The design process of architectures is often formalized
asanoptimizationproblem,whereECalgorithmsarecorrectlycreatedtotacklethe
optimizationproblem.
DNNshavehadremarkablesuccessinmanycomplicatedpracticalapplications
inrecentyears.ItiswellknownthattheperformanceofaDNNisonlypromising
whenthearchitectureisappropriate.Thearchitecture,ontheotherhand,istypically
createdbyhand,needingahighlevelofskillthatisinscarcesupplyinpractice.A
promisingdeeparchitectureisdifficulttocreateinpracticewithoutthiskindofskill,
whichisfrequentlyDNNexpertiseanddomainunderstandingoftheproblemtobe
solved.Reinforcementlearningtechniques,gradient-basedoptimizationalgorithms,
andECmethodsarethethreecommonmethodsutilizedtoconstructthearchitectures
of DNNs in the literature. This book mainly focuses on the EC methods for deep
neuralarchitecturedesign.
In this book, we will first introduce the fundamentals of commonly used EC
methods,includingGeneticAlgorithm(GA)[1],ParticleSwarmOptimization(PSO)
[2],DifferentialEvolution(DE)[3],andGeneticProgramming(GP)[4].Following
that,wewillgothroughtwodifferentformsofevolutionarydeepneuralarchitecture
designalgorithms.TheyarethedesignalgorithmsforunsupervisedDNNs,andthe
design algorithms for supervised DNNs. In addition, we will also discuss some
recent efforts to speed up the execution of such algorithms. These algorithms are
primarily based on the authors recent work, which has been published in journals
and international conferences devoted to EC and neural networks. We think that
bypresentingthemalltogetherinthisbook,readerswillbeabletoabsorbrelated
informationmorequicklyandconveniently.
Thisbookiscomposedoffiveparts,andeachpartisfulfilledwiththenecessary
contentindifferentchapters.IntheremainderofPart1,wewillprovideanoverview
v
vi Preface
of the fundamentals and backgrounds on EC and deep learning with a focus on
differenttypesofcommonlyusedDNNs.
Chengdu,China YananSun
Stillwater,USA GaryG.Yen
Wellington,NewZealand MengjieZhang
References
1. JohnHenryHollandetal.Adaptationinnaturalandartificialsystems:anintroductoryanalysis
withapplicationstobiology,control,andartificialintelligence.MITpress,1992.
2. James Kennedy and R Eberhart Particle Swarm Optimization. Ieee int. In Conf. on Neural
Networks,volume4,1995.
3. RainerStornandKennethPrice.DifferentialevolutionâASasimpleandefficientheuristicfor
globaloptimizationovercontinuousspaces,1997.URLhttps://doi.org/10.1023/A:100820282
1328.
4. WolfgangBanzhaf,PeterNordin,RobertEKeller,andFrankDFrancone.Geneticprogram-
ming:anintroduction,volume1.MorganKaufmannSanFrancisco,1998.
Contents
PartI FundamentalsandBackgrounds
1 EvolutionaryComputation ..................................... 3
1.1 GeneticAlgorithms(GAs) ................................. 3
1.2 ParticleSwarmOptimization(PSO) ......................... 4
1.3 DifferentialEvolution(DE) ................................ 5
1.4 GeneticProgramming(GP) ................................ 6
1.5 ChapterSummary ........................................ 7
References .................................................... 7
2 DeepNeuralNetworks ......................................... 9
2.1 DeepBeliefNetworks .................................... 12
2.2 StackedAuto-Encoders ................................... 13
2.2.1 SparseAuto-Encoders ............................. 14
2.2.2 WeightDecayAuto-Encoders ...................... 15
2.2.3 DenoisingAuto-Encoders(DAEs) ................... 15
2.2.4 ContractiveAuto-Encoders ......................... 16
2.2.5 ConvolutionalAuto-Encoders(CAEs) ............... 17
2.2.6 VariationalAuto-Encoders(VAEs) .................. 18
2.3 ConvolutionalNeuralNetworks(CNNs) ..................... 19
2.3.1 CNNSkeleton .................................... 20
2.3.2 Convolution ...................................... 20
2.3.3 Pooling .......................................... 21
2.3.4 ReflectPadding ................................... 22
2.3.5 BatchNormalization(BN) ......................... 23
2.3.6 ResNetBlocks(RBs)andDenseNetBlocks(DBs) ..... 23
2.4 BenchmarksforDeepNeuralNetworks ..................... 24
2.5 ChapterSummary ........................................ 27
References .................................................... 28
vii
viii Contents
PartII EvolutionaryDeepNeuralArchitectureSearch
forUnsupervisedDNNs
3 ArchitectureDesignforStackedAEsandDBNs .................. 39
3.1 Introduction ............................................. 39
3.2 RelatedWorkandMotivations ............................. 40
3.2.1 UnsupervisedDeepLearning ....................... 40
3.2.2 EvolutionaryAlgorithmsforEvolvingNeural
Networks ........................................ 41
3.3 AlgorithmDetails ........................................ 43
3.3.1 FrameworkofEUDNN ............................ 43
3.3.2 EvolvingConnectionWeightsandActivation
Functions ........................................ 45
3.3.3 Fine-TuningConnectionWeights ................... 47
3.3.4 Discussion ....................................... 48
3.4 ExperimentalDesign ..................................... 49
3.4.1 PerformanceMetric ............................... 50
3.4.2 PeerCompetitors ................................. 50
3.4.3 ParameterSettings ................................ 51
3.5 ExperimentalResultsandAnalysis ......................... 52
3.5.1 PerformanceofEUDNN ........................... 52
3.5.2 AnalysisonPre-trainingofEUDNN ................. 54
3.5.3 AnalysisonFine-TuningofEUDNN ................ 55
3.5.4 RepresentationVisualizations ....................... 56
3.6 ChapterSummary ........................................ 57
References .................................................... 58
4 ArchitectureDesignforConvolutionalAuto-Encoders ............ 61
4.1 Introduction ............................................. 61
4.2 MotivationofFCAE ...................................... 62
4.3 AlgorithmDetails ........................................ 64
4.3.1 AlgorithmOverview .............................. 64
4.3.2 EncodingStrategy ................................ 64
4.3.3 ParticleInitialization .............................. 65
4.3.4 FitnessEvaluation ................................ 66
4.3.5 VelocityCalculationandPositionUpdate ............ 67
4.3.6 DeepTrainingonGlobalBestParticle ............... 70
4.4 ExperimentalDesign ..................................... 70
4.4.1 PeerCompetitors ................................. 70
4.4.2 ParameterSettings ................................ 70
4.5 ExperimentalResultsandAnalysis ......................... 71
4.5.1 OverviewPerformance ............................ 71
4.5.2 EvolutionTrajectoryofPSOAO ..................... 72
4.5.3 PerformanceonDifferentNumbersofTraining
Examples ........................................ 73
4.5.4 Investigationonx-ReferenceVelocityCalculation ..... 74
Contents ix
4.6 ChapterSummary ........................................ 75
References .................................................... 75
5 ArchitectureDesignforVariationalAuto-Encoders ............... 79
5.1 Introduction ............................................. 79
5.2 AlgorithmDetails ........................................ 80
5.2.1 AlgorithmOverview .............................. 80
5.2.2 StrategyofGeneEncoding ......................... 81
5.2.3 InitializationofPopulation ......................... 85
5.2.4 Evaluation ....................................... 86
5.2.5 CrossoverOperatorandMutationOperator ........... 88
5.2.6 EnvironmentalSelection ........................... 90
5.3 ExperimentalDesign ..................................... 90
5.3.1 ParameterSetting ................................. 91
5.3.2 PeerCompetitors ................................. 91
5.3.3 PerformanceEvaluation ........................... 92
5.4 ExperimentalResultsandAnalysis ......................... 93
5.4.1 OverallPerformance .............................. 93
5.4.2 EvolutionTrajectoryofEvoVAE .................... 97
5.4.3 RunningTime .................................... 99
5.4.4 TheObtainedArchitecture ......................... 99
5.4.5 AblationExperiments ............................. 102
5.5 ChapterSummary ........................................ 103
References .................................................... 104
PartIII Evolutionary Deep Neural Architecture Search for
SupervisedDNNs
6 ArchitectureDesignforPlainCNNs ............................. 109
6.1 Introduction ............................................. 109
6.2 AlgorithmDetails ........................................ 110
6.2.1 AlgorithmOverview .............................. 110
6.2.2 StrategyofGeneEncoding ......................... 111
6.2.3 InitializationofPopulation ......................... 112
6.2.4 EvaluationofFitness .............................. 113
6.2.5 SlackBinaryTournamentSelection ................. 115
6.2.6 OffspringGeneration .............................. 116
6.2.7 EnvironmentalSelection ........................... 117
6.2.8 SelectandDecodeBestIndividual .................. 118
6.3 ExperimentalDesign ..................................... 118
6.3.1 PeerCompetitors ................................. 119
6.3.2 ParameterSettings ................................ 119
6.4 ExperimentalResultsandDiscussion ........................ 119
6.4.1 OverallResults ................................... 120
6.4.2 PerformanceofWeightInitialization ................. 122
6.4.3 Discussion ....................................... 123
