Table Of ContentStudies in Computational Intelligence 975
Yaochu Jin
Handing Wang
Chaoli Sun
Data-Driven
Evolutionary
Optimization
Integrating Evolutionary Computation,
Machine Learning and Data Science
Studies in Computational Intelligence
Volume 975
SeriesEditor
JanuszKacprzyk,PolishAcademyofSciences,Warsaw,Poland
The series “Studies in Computational Intelligence” (SCI) publishes new develop-
mentsandadvancesinthevariousareasofcomputationalintelligence—quicklyand
with a high quality. The intent is to cover the theory, applications, and design
methods of computational intelligence, as embedded in the fields of engineering,
computer science, physics and life sciences, as well as the methodologies behind
them. The series contains monographs, lecture notes and edited volumes in
computational intelligence spanning the areas of neural networks, connectionist
systems, genetic algorithms, evolutionary computation, artificial intelligence,
cellular automata, self-organizing systems, soft computing, fuzzy systems, and
hybrid intelligent systems. Of particular value to both the contributors and the
readership are the short publication timeframe and the world-wide distribution,
whichenablebothwideandrapiddisseminationofresearchoutput.
IndexedbySCOPUS,DBLP,WTIFrankfurteG,zbMATH,SCImago.
AllbookspublishedintheseriesaresubmittedforconsiderationinWebofScience.
Moreinformationaboutthisseriesathttp://www.springer.com/series/7092
· ·
Yaochu Jin Handing Wang Chaoli Sun
Data-Driven Evolutionary
Optimization
Integrating Evolutionary Computation,
Machine Learning and Data Science
YaochuJin HandingWang
DepartmentofComputerScience SchoolofArtificialIntelligence
UniversityofSurrey XidianUniversity
Guildford,UK Xi’an,China
ChaoliSun
SchoolofComputerScience
andTechnology
TaiyuanUniversityofScience
andTechnology
Taiyuan,China
ISSN1860-949X ISSN1860-9503 (electronic)
StudiesinComputationalIntelligence
ISBN978-3-030-74639-1 ISBN978-3-030-74640-7 (eBook)
https://doi.org/10.1007/978-3-030-74640-7
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SwitzerlandAG2021
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Foreword
One of the appeals of optimization is its applicability across countless fields of
interest; another is the opportunity it affords to interface with, and benefit from,
allieddisciplines.
Optimization methodologies have been transformed dramatically since I was
drawn to their allure 50 years ago. Notable advances, that I have had a particular
interestin,includethosethathavebeenmadeinglobaloptimization,multi-objective
optimization,robustoptimization,andblackboxanddata-drivenoptimization.
Thesedevelopmentshavegreatlyenhancedoptimization’scapabilitytosatisfacto-
rilyaddressreal-worldproblems,therebystrengtheningusers’confidence.Through
globaloptimization,wearereleasedfromthehazardsoffocusingonspecific,local
areasofthesearchspace.Multi-objectiveoptimizationsatisfiesthedilemmaofhowto
balancecompetingmultipleobjectives;itoffersincreasedtransparencybyproviding
afamilyoftrade-offsolutionstoenabledomainexpertstoselectthedesiredcompro-
misesolution.Inreal-worldapplications,itisalmostinevitablethatuncertaintieswill
arisefromeitherproblemformulationortheimplementationenvironmentorboth.
Forexample,thesemightoccurthroughdeficienciesinthefidelityofamodel,the
precisionofanobjectiveortheabilitytoperfectlyrealizetheproposedsolutionin
practice.Robustoptimizationmethodsenablepractitionerstomanagetheseuncer-
tainties.Evolutionarycomputinghasbeenacatalystthathaspromotedprogressin
thesedevelopments.
But what if, in a design optimization, there is no analytic representation for a
particular objective or objectives? Instead, an objective (or objectives) might rely
ontheoutcomeofcomputersimulations,suchasdynamicsystemsmodelling,finite
elementanalysisorthemodellingofcomputationalfluiddynamics.Or,theobjectives
mayevendependonresultsobtainedfromexperiments.Further,thecollecteddata
may be incomplete and noisy or vary in content. It will not come as a surprise,
then,tolearnthatevolutionarycomputinghasanimportantroleindealingwithsuch
data-drivenproblemsaswell.
Data-driven optimization forms the central focus of this textbook. Allying the
population-basedmetaheuristicsofevolutionarycomputingwiththeuseofsurrogate
modelsandmachinelearningalgorithms,theauthorsdemonstrateaformidablearray
ofapproachestoaddressarangeofdata-drivenoptimizationchallenges.
v
vi Foreword
AsIstartedoutonmyadventureswithoptimizationresearch(andtheseresulted
in a lifetime fascination), a textbook that I regularly turned to, and still do from
timetotime,wasEdgarandHimmelblau’sOptimizationofChemicalProcesses,not
becauseIwasachemicalengineer(Iwasnot)butbecauseoftheblendofitsclarity
ofexposition,itspracticalapproachandtheauthors’evidentexperienceofapplying
theirknowledgetoreal-worldapplications.Data-DrivenEvolutionaryOptimization
hasthesameappealtome;itiswrittenbyextremelyexperiencedresearcherswho
haveaclearviewoftheimportanceoftailoringmethodologiestoaddresspractical
problemsandheretheyprovideasoundgroundingintheassortedrequiredstrategies
andtheirunderpinningmethodologies.
Sheffield,UK PeterFleming
March2021
Preface
IstartedworkingonfitnessapproximationinevolutionaryoptimizationwhenImoved
back to Germany from the USA in 1999 to take up a research scientist position at
the Honda Research Institute Europe in Offenbach to complete my second Ph.D.
degree.Themotivationthenwastouseevolutionaryalgorithms,inparticularevolu-
tion strategies to optimize the aerodynamic performance of a turbine engine by
designingthegeometryofthestatororrotorblades.Toaccomplishthistask,time-
consuming computational fluid dynamics simulations must be performed, which
preventsonefromusingtensofthousandsoffitnessevaluations,astypicallydone
inevolutionaryoptimization.Toreducethetimeconsumptionforevolutionaryaero-
dynamic optimization, fitness approximation using machine learning models, also
calledmeta-modelsorsurrogates,cameintoplay.Iwasverymuchattractedbythis
researchtopic,sinceitprovidesaniceplatformtointegrateevolutionarycomputa-
tionwithneuralnetworks,twotopicsIwasinterestedin.Outofthisreason,Imade
much effort to promote this new area in the evolutionary computation community
andcontinuedworkingonthistopicwhenevertherewasanopportunityafterImoved
totheUniversityofSurreyin2010.
Over the pasttwenty years,fitness approximation in evolutionary optimization,
also known as surrogate-assisted evolutionary optimization, has grown and devel-
opedintoafascinatingresearchfield,whichisnowtermeddata-drivenevolutionary
optimization.Data-drivenevolutionaryoptimizationfocusesonaclassofreal-world
optimizationproblemsinwhichnoanalyticalmathematicalfunctionscanbeestab-
lishedfortheobjectivesorconstraints.Generallyspeaking,data-drivenoptimization
mayincludethefollowingsituations:first,simulation-basedoptimization,wherethe
qualityofasolutionisassessedbyaniterative,computationallyintensiveprocess,
ranging from numerically solving a large set of differential equations, to training
a deep neural network on a large data set, and second, physical experiment-based
optimization, where the objective or constraint values of a candidate solution can
be evaluated only by performing physical or human experiments. This is usually
because a high-quality computer simulation of the whole system is not possible,
either because it is computationally prohibitive, e.g. numerical simulations of the
aerodynamicsofthewholeaircraft,orbecauseitisintractablebecausetheprocessis
notyetfullyunderstood,e.g.theworkingmechanismofthehumandecision-making
vii
viii Preface
process.Finally,purelydata-drivenoptimization,whereonlydatacollectedinreal-
lifeisavailableforoptimizationandneitheruser-designedcomputersimulationsnor
physical experiments are allowed. For example, optimization of a complex indus-
trialprocessorsocialsystem.Inalltheabovecases,theamountofthecollecteddata
may be either small or big, and the data may be heterogeneous, noisy, erroneous,
incomplete,ill-distributedorincremental.
Clearly, data-driven evolutionary optimization involves three different but
complementary scientific disciplines, namely evolutionary computation, machine
learning and deep learning, and data science. To effectively and efficiently solve
a data-driven optimization problem, data must be properly pre-processed. Mean-
while, machine learning techniques become indispensable for handling big data,
datapaucityandvariousdegreesofuncertaintyinthedata.Finally,solvingtheopti-
mizationproblembecomesextremelydemandingwhentheoptimizationproblemis
high-dimensionalorlarge-scale,multi-objectiveandtime-varying.
Thisbookaimstoprovideresearchers,includingpostgraduateresearchstudents,
and industrial practitioners a comprehensive description of the state-of-the-art
methodsdevelopedfordata-drivenevolutionaryoptimization.Thebookisdivided
into12chapters.Fortheself-containednessofthebook,abriefintroductiontocare-
fullyselectedimportanttopicsandmethodsinoptimization,evolutionarycomputa-
tionandmachinelearningisprovidedinChaps.1–4.Chapter5providesthefunda-
mentals of data-driven optimization, including heuristics and acquisition function-
based surrogate management, followed by Chaps. 6–8, presenting ideas that use
multiple surrogates for single-objective optimization. Representative evolutionary
algorithms for solving multi- and many-objective optimization algorithms and
surrogate-assisteddata-drivenevolutionarymulti-andmany-objectiveoptimization
aredescribedinChaps.7and8,respectively.Approachestohigh-dimensionaldata-
drivenoptimizationareelaboratedinChap.9.Aplethoraoftechniquesfortransfer-
ringknowledgefromunlabelledtolabelleddata,fromcheapobjectivestoexpensive
ones,andfromcheapproblemstoexpensiveonesarepresentedinChap.10,withthe
helpofsemi-supervisedlearning,transferlearningandtransferoptimization.Since
data-drivenoptimizationisastronglyapplication-drivenresearcharea,offlinedata-
drivenevolutionaryoptimizationistreatedinChap.11,exemplifiedwithreal-world
optimizationproblemssuchasairfoildesignoptimization,crudeoildistillationopti-
mizationandtraumasystemoptimization.Finally,deepneuralarchitecturesearchas
adata-drivenexpensiveoptimizationproblemishighlightedinChap.12.Ofthe12
chapters,§ 3.5–3.6,§ 4.2,§ 5.2,§ 6.4–6.5,§ 7.2–7.3,§ 9.6–9.7,§ 11.1,§
11.3andChap. 12arewrittenbyHandingWang,and§ 3.7–3.8,§ 5.4.1,§ 5.5,§
6.2–6.3,§ 9.2–9.3andChap. 10byChaoliSun.HandingworkedasPostdoctoral
Associateduring2015–18andChaoliatfirstasAcademicVisitorduring2012–13
andthenasPostdoctoralAssociateduring2015–17inmygroupatSurrey.
To make it easier for the reader to understand and use the algorithms intro-
duced in the book, the source code for most data-driven evolutionary algorithms
presentedinChaps.5–12ismadeavailable(http://www.soft-computing.de/DDEO/
DDEO.html)andallbaselinemulti-objectiveevolutionaryalgorithmsintroducedin
Preface ix
thisbookareimplementedinPlatEMO,anopen-sourcesoftwaretoolforevolutionary
multi-objectiveoptimization(https://github.com/BIMK/PlatEMO).
This book would not have been possible without the support of many previous
colleagues, collaborators and Ph.D. students of mine. First of all, I would like to
thankProf.Dr.BernhardSendhoffandProf.Dr.MarkusOlhofer,withbothofwhom
IcloselyworkedattheHondaResearchInstituteEuropeduring1999–2010.After
I joined Surrey in 2010, Markus and I still maintained close collaboration on a
numberofresearchprojectsonevolutionaryoptimization.IwouldalsothankProf.
Kaisa Miettinen from the University of Jyväskylä, Finland, with whom I worked
closelyasFinlandDistinguishedProfessor during2015–17onevolutionarymulti-
objectiveoptimization.ThanksgotoProf.TianyouChaiandProf.JinliangDingfrom
NortheasternUniversity,China,withwhomIalsocollaboratewithonevolutionary
optimization as Changjiang Distinguished Professor. The following collaborators
andpreviousorcurrentPh.D.studentsofminehavecontributedtopartofthework
presented in this book: Prof. Yew-Soon Ong, Prof. Jürgen Branke, Prof. Qingfu
Zhang,Prof.XingyiZhang,Prof.AiminZhou,Prof.RanCheng,Prof.XiaoyanSun,
Dr.IngoPaenke,Dr.TinkleChugh,Mr.JohnDoherty,Dr.DanGuo,Dr.CuieYang,
Dr. Ye Tian, Dr. Cheng He, Dr. Dudy Lim, Dr. Mingh Nhgia Le, Dr. Jie Tian, Dr.
Haibo Yu, Dr.Guo Yu,Dr.Michael Hüsken, Ms.Huiting Li,Ms.Xilu Wang, Ms.
ShufenQin,Mr.HaoWang,Ms.GuoxiaFu,Mr.PengLiao,Mr.SebastianSchmitt,
Ms. Kailai Gao, Dr. Jussi Hakanen, Dr. Tatsuya Okabe, Dr. Yanan Sun, Dr. Jan
O. Jansen, Mr. Martin Heiderich, Dr. Yuanjun Huang and Dr. Tobias Rodemann.
I would also like to take this opportunity to thank Prof. Xin Yao, Prof. Gary Yen,
Prof. Kay Chen Tan, Prof. Mengjie Zhang, Prof. Richard Everson, Prof. Jonathon
Fieldsend, Prof. Dr. Stefan Kurz, Prof. Edgar Körner and Mr. Andreas Richter for
theirkindsupportoverthepasttwodecades.Finally,financialsupportfromEPSRC
(UK), TEKES (Finland), National Natural Science Foundation of China, Honda
Research Institute Europe, Honda R&D Europe and Bosch Germany is gratefully
acknowledged.
Guildford,UK YaochuJin
February2021
