Machine Learning with Neural Networks: An Introduction for Scientists and Engineers

261 Pages·2021·9.259 MB·English

by  Bernhard Mehlig

MACHINE LEARNING WITH NEURAL NETWORKS
This modern and self-contained book offers a clear and accessible introduction
to the important topic of machine learning with neural networks. In addition to
describing the mathematical principles of the topic, and its historical evolution,
strong connections are drawn with underlying methods from statistical physics
and current applications within science and engineering. Closely based around
a well-established undergraduate course, this pedagogical text provides a solid
understandingofthekeyaspectsofmodernmachinelearningwithartificialneural
networks, for students in physics, mathematics, and engineering. Numerous exer-
cises expand and reinforce key concepts within the book and allow students to
hone their programming skills. Frequent references to current research develop a
detailedperspectiveonthestate-of-the-artinmachinelearningresearch.
BERNHARD MEHLIG is Professor in Complex Systems at the University of
Gothenburg,Sweden.Hisresearchisfocusedonstatisticalphysicsofcomplexsys-
temsandfluidmechanics,andhehaspublishedextensivelyintheseareas.In2010,
hewasawardedtheprestigiousGöranGustafssonPrizeinphysicsforhisresearch
instatisticalphysics.Hehastaughtacourseonmachinelearningformorethan15
yearsatChalmersUniversityofTechnologyandattheUniversityofGothenburg.

MACHINE LEARNING WITH NEURAL
NETWORKS
An Introduction for Scientists and Engineers
BERNHARD MEHLIG
UniversityofGothenburg,Sweden
UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom
OneLibertyPlaza,20thFloor,NewYork,NY10006,USA
477WilliamstownRoad,PortMelbourne,VIC3207,Australia
314–321,3rdFloor,Plot3,SplendorForum,JasolaDistrictCentre,NewDelhi–110025,India
103PenangRoad,#05–06/07,VisioncrestCommercial,Singapore238467
CambridgeUniversityPressispartoftheUniversityofCambridge.
ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof
education,learning,andresearchatthehighestinternationallevelsofexcellence.
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781108494939
DOI:10.1017/9781108860604
©BernhardMehlig2022
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2022
PrintedintheUnitedKingdombyTJBooksLimited,PadstowCornwall
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary.
LibraryofCongressCataloging-in-PublicationData
Names:Mehlig,Bernhard,1964-author.
Title:Machinelearningwithneuralnetworks:anintroductionforscientistsandengineers/Bernhard
Mehlig,UniversityofGothenburg,Sweden.
Description:Cambridge,UnitedKingdom;NewYork,NY:CambridgeUniversityPress,2021.|
Includesbibliographicalreferencesandindex.
Identifiers:LCCN2021027339|ISBN9781108494939(hardback)|ISBN9781108860604(ebook)
Subjects:LCSH:Neuralnetworks(Computerscience)|Machinelearning.|BISAC:SCIENCE/Physics/
Mathematical&Computational|SCIENCE/Physics/Mathematical&Computational
Classification:LCCQA76.87.M4472021|DDC006.3/2–dc23
LCrecordavailableathttps://lccn.loc.gov/2021027339
ISBN978-1-108-49493-9Hardback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyof
URLsforexternalorthird-partyinternetwebsitesreferredtointhispublication
anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain,
accurateorappropriate.
Thecoverimageshowsaninputpatterndesignedtomaximisetheoutputofneuronscorrespondingtoone
featuremapinagivenconvolutionlayerofadeepconvolutionalneuralnetwork.(SeeSection8.7and
referencescitedthere.)ImagebyHampusLinander.Reproducedwithpermission.
Contents
Acknowledgements pageix
1 Introduction 1
1.1 NeuralNetworks 5
1.2 McCulloch-PittsNeurons 7
1.3 ActivationFunctions 9
1.4 AsynchronousUpdates 11
1.5 Summary 11
1.6 FurtherReading 11
PartI HopfieldNetworks
2 DeterministicHopfieldNetworks 15
2.1 PatternRecognition 15
2.2 HopfieldNetworksandHebb’sRule 16
2.3 TheCross-TalkTerm 22
2.4 One-StepErrorProbability 23
2.5 EnergyFunction 27
2.6 Summary 30
2.7 Exercises 31
3 StochasticHopfieldNetworks 34
3.1 StochasticDynamics 34
3.2 OrderParameters 35
3.3 Mean-FieldTheory 37
3.4 CriticalStorageCapacity 41
3.5 BeyondMean-FieldTheory 47
3.6 CorrelatedandNon-RandomPatterns 49
v
vi Contents
3.7 Summary 50
3.8 FurtherReading 50
3.9 Exercises 50
4 TheBoltzmannDistribution 52
4.1 ConvergenceoftheStochasticDynamics 53
4.2 Monte-CarloSimulation 55
4.3 SimulatedAnnealing 57
4.4 BoltzmannMachines 60
4.5 RestrictedBoltzmannMachines 64
4.6 Summary 70
4.7 FurtherReading 70
4.8 Exercises 71
PartII SupervisedLearning
5 Perceptrons 75
5.1 AClassificationProblem 76
5.2 IterativeLearningAlgorithm 80
5.3 GradientDescentforLinearUnits 81
5.4 ClassificationCapacity 83
5.5 MultilayerPerceptrons 86
5.6 Summary 90
5.7 FurtherReading 90
5.8 Exercises 90
6 StochasticGradientDescent 96
6.1 ChainRuleandErrorBackpropagation 96
6.2 StochasticGradient-DescentAlgorithm 99
6.3 PreprocessingtheInputData 102
6.4 OverfittingandCrossValidation 106
6.5 AdaptationoftheLearningRate 108
6.6 Summary 110
6.7 FurtherReading 110
6.8 Exercises 111
7 DeepLearning 114
7.1 HowManyHiddenLayers? 114
7.2 VanishingandExplodingGradients 119
7.3 RectifiedLinearUnits 125
Contents vii
7.4 ResidualNetworks 126
7.5 OutputsandEnergyFunctions 129
7.6 Regularisation 132
7.7 Summary 138
7.8 FurtherReading 139
7.9 Exercises 139
8 ConvolutionalNetworks 141
8.1 ConvolutionLayers 142
8.2 PoolingLayers 144
8.3 LearningtoReadHandwrittenDigits 145
8.4 CopingwithDeformationsoftheInputDistribution 148
8.5 DeepLearningforObjectRecognition 149
8.6 Summary 152
8.7 FurtherReading 154
8.8 Exercises 154
9 SupervisedRecurrentNetworks 157
9.1 RecurrentBackpropagation 159
9.2 BackpropagationthroughTime 162
9.3 VanishingGradients 167
9.4 RecurrentNetworksforMachineTranslation 169
9.5 ReservoirComputing 171
9.6 Summary 173
9.7 FurtherReading 174
9.8 Exercises 174
PartIII LearningwithoutLabels
10 UnsupervisedLearning 179
10.1 Oja’sRule 179
10.2 CompetitiveLearning 183
10.3 Self-OrganisingMaps 185
10.4 K-MeansClustering 191
10.5 RadialBasisFunctions 193
10.6 Autoencoders 197
10.7 Summary 201
10.8 FurtherReading 201
10.9 Exercises 202
viii Contents
11 ReinforcementLearning 206
11.1 AssociativeReward-PenaltyAlgorithm 209
11.2 TemporalDifferenceLearning 212
11.3 Q-Learning 216
11.4 Summary 223
11.5 FurtherReading 223
11.6 Exercises 223
Bibliography 227
AuthorIndex 238
SubjectIndex 242
Acknowledgements
This textbook is based on lecture notes for the course Artificial Neural Networks
that I have given at Gothenburg University and at Chalmers Technical University
inGothenburg,Sweden.WhenIpreparedmylectures,mymainsourcewas Intro-
duction to the Theory of Neural Computation by Hertz, Krogh, and Palmer [1].
OthersourceswereNeuralNetworks:AComprehensiveFoundationbyHaykin[2];
Horner’s lecture notes from Heidelberg [3]; Deep Learning by Goodfellow, Ben-
gio, and Courville [4]; and the online book Neural Networks and Deep Learning
byNielsen[5].
I thank Martin Cˇejka for typesetting the first version of my handwritten lec-
ture notes, Erik Werner and Hampus Linander for their help in preparing Chapter
8, Kristian Gustafsson for his detailed feedback on Chapter 11, Nihat Ay for his
comments on Section 4.5, and Mats Granath for discussions about autoencoders.
I would also like to thank Juan Diego Arango, Oleksandr Balabanov, Anshuman
Dubey,JohanFries,PhillipGräfensteiner,NavidMousavi,MarinaRafajlovic,Jan
Schiffeler, Ludvig Storm, and Arvid Wenzel Wartenberg for implementing algo-
rithms described in this book. Many figures are based on their results. Oleksandr
Balabanov, Anshuman Dubey, Jan Meibohm, and in particular Johan Fries and
Marina Rafajlovic contributed exam questions that became exercises in this book.
Finally, I would like to express my gratitude to Stellan Östlund for his encourage-
ment and criticism. Last but not least, a large number of colleagues and students
–pastandpresent–pointedoutmisprintsanderrorsandsuggestedimprovements.
Ithankthemall.
ix