Table Of ContentNatural Computing Series
Leonardo Vanneschi
Sara Silva
Lectures
on Intelligent
Systems
Natural Computing Series
Founding Editor
Grzegorz Rozenberg
Series Editors
Thomas Bäck , Natural Computing Group–LIACS, Leiden University, Leiden,
The Netherlands
Lila Kari, School of Computer Science, University of Waterloo, Waterloo,
ON, Canada
Susan Stepney, Department of Computer Science, University of York, York, UK
Scope
Natural Computing is one of the most exciting developments in computer science,
and there is a growing consensus that it will become a major field in this century.
This series includes monographs, textbooks, and state-of-the-art collections
covering the whole spectrum of Natural Computing and ranging from theory to
applications.
Leonardo Vanneschi Sara Silva
(cid:129)
Lectures on Intelligent
Systems
123
Leonardo Vanneschi SaraSilva
NOVAInformation Management School LASIGE, Departamento deInformática
Universidade NovadeLisboa Faculdade deCiências
Lisbon, Portugal Universidade deLisboa
Lisbon, Portugal
ISSN 1619-7127
NaturalComputing Series
ISBN978-3-031-17921-1 ISBN978-3-031-17922-8 (eBook)
https://doi.org/10.1007/978-3-031-17922-8
©SpringerNatureSwitzerlandAG2023
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors, and the editorsare safeto assume that the adviceand informationin this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
hereinorforanyerrorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregard
tojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
ToourchildrenDani,DavidandLara.
ToourparentsAltina,Eurico,GiovannaandMarco.
LeonardoandSara
Tomydad.Hewouldbeproud.
Sara
Acknowledgements
MyfirstacknowledgmentgoestoMarcoTomassini.AsmyformerPhDsupervisor,
he is the person who introduced me to the area of Computational Intelligence. He
has been a central figure for the development of my profession. This book would
surelynotexistwithouthim.
MysecondacknowledgmentgoestoMauroCastelli.Hewasanundergraduatestu-
dentwhenIfirstusedsomeoftheideasofthisbookinmyclasses.Thenhebecame
myMastersstudent,andlatermyPhDstudent.Finally,heisnowmycolleagueand
my great friend. His collaboration has been invaluable for at least the past twelve
years.Ithadatremendousimpactonmyworkandso,indirectly,alsoonthisbook.
Then,IwouldliketothankallthecollaboratorsthatIhavehadfromthebeginning
ofmycareeruntiltoday.Theyaremany,soIcannotmentionthemall,buteachone
ofthemhasbeen,intheirownway,specialandimportantformywork.
Lastbutnotleast,Iwouldliketoexpressmygratitudetomyfamilyfortheirinfinite
support and for making me grow up in an environment where value was given to
studyingandeducation.
Leonardo
First things first, thank you to my family for never having tried to convince me
that “scientist of artificial intelligence” was not what a young teenage girl should
want to be. In a country where there is still no such thing as a research career, it
took (and still takes) a lot of stubbornness on my part, but also a lot of support
frommanypeople,toachieveit.IacknowledgeeverysingleresearchgrantIhave
everreceived,everytravelgrantthatallowedmetoparticipateinconferences,every
researchcentrethattookmein,andeveryonewhobelievedinmeandrespectedmy
wishtotaketheroadlesstraveled.
Sara
ix
Contents
1 Introduction................................................... 1
1.1 MotivationforIntelligentSystems ............................ 3
1.2 IntelligentSystemsandBioinspiredAlgorithms ................. 5
1.3 ApplicationsofIntelligentSystems............................ 6
1.4 ObjectivesandLimitsofThisBook ........................... 7
1.5 OrganizationoftheBook .................................... 8
PartI ComputationalIntelligenceforOptimization
2 OptimizationProblemsandLocalSearch ......................... 13
2.1 IntroductiontoOptimization ................................. 13
2.2 ExamplesofOptimizationProblems........................... 16
2.3 NoFreeLunchTheorem..................................... 19
2.4 HillClimbing.............................................. 22
2.5 FitnessLandscapes ......................................... 28
2.6 SimulatedAnnealing........................................ 34
2.6.1 TheoryofSimulatedAnnealing ........................ 40
3 GeneticAlgorithms............................................. 45
3.1 SelectionAlgorithms ....................................... 47
3.1.1 FitnessProportionateSelection(or“RouletteWheel”) ..... 48
3.1.2 RankingSelection ................................... 50
3.1.3 TournamentSelection ................................ 50
3.2 GeneticOperators .......................................... 51
3.3 GeneralFunctioningofGeneticAlgorithms .................... 55
3.4 TheoryofGeneticAlgorithms................................ 61
3.5 AdvancedMethodsforGeneticAlgorithms..................... 77
3.5.1 PrematureConvergence............................... 77
3.5.2 PositionProblemofStandardCrossover ................. 84
3.5.3 UniquenessoftheFitnessFunction ..................... 90
3.6 HowtoOrganizeanExperimentalComparison.................. 94
xi
xii Contents
3.6.1 ComparisonAgainstIterations ......................... 94
3.6.2 ComparisonAgainstComputationalEffort ............... 98
3.7 GeneticAlgorithmsforContinuousOptimization................102
4 ParticleSwarmOptimization....................................105
4.1 TheAlgorithm.............................................105
4.2 ParameterSetting ..........................................109
4.3 Variants...................................................110
PartII MachineLearning
5 IntroductiontoMachineLearning ...............................115
5.1 FromDatatoModel ........................................122
5.1.1 EstimatingthePredictiveError.........................124
5.1.2 ChoosingMethodandParameters ......................125
5.1.3 InducingandTestingtheFinalModel ...................126
5.1.4 PreprocessingtheData................................127
5.2 DataSplitting..............................................128
5.3 Cross-Validation ...........................................129
5.4 MeasuresofPerformanceofaClassifier .......................131
5.5 FeatureProcessing .........................................136
5.6 Regularization .............................................140
5.6.1 RidgeRegression ....................................141
5.6.2 LassoRegression ....................................142
5.6.3 ElasticNet..........................................143
5.7 SomeSimpleMachineLearningMethods ......................143
5.7.1 K-NearestNeighbors .................................143
5.7.2 LinearRegression....................................145
5.7.3 LogisticRegression ..................................146
6 DecisionTreeLearning .........................................149
6.1 ID3Algorithm.............................................152
6.2 ContinuousAttributesandTargets.............................157
7 ArtificialNeuralNetworks ......................................161
7.1 Perceptron ................................................162
7.1.1 SingleNeuronPerceptron .............................162
7.1.2 MulticlassClassificationandMultineuron,Single-Layer
Perceptron..........................................172
7.1.3 NonlinearlySeparableProblems.AnExample............173
7.1.4 NonlinearlySeparableProblemsandMultilayerANNs.....176
7.2 AdaptiveLinearElement(ADALINE)andDeltaRule............178
7.3 Backpropagation ...........................................183
7.3.1 WeightsUpdate–OutputLayer ........................186
7.3.2 WeightsUpdate–HiddenLayer........................190
7.3.3 TheBackpropagationAlgorithm .......................195
