Table Of ContentOliver Nelles
Nonlinear
System
Identification
From Classical Approaches to Neural
Networks, Fuzzy Models, and Gaussian
Processes
Second Edition
Nonlinear System Identification
Oliver Nelles
Nonlinear System
Identification
From Classical Approaches to
Neural Networks, Fuzzy Models,
and Gaussian Processes
Second Edition
OliverNelles
UniversityofSiegen
Netphen,Germany
ISBN978-3-030-47438-6 ISBN978-3-030-47439-3 (eBook)
https://doi.org/10.1007/978-3-030-47439-3
1stedition:©Springer-VerlagBerlinHeidelberg2001
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerland
AG2020
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG.
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface to the Second Edition
Thesecondeditionofthisbookcontainsnovelchaptersonthefollowingtopics:
• Inputselectionwithlocalmodelnetworks,Chap.15.
• Gaussianprocessmodels,Chap.16.
• Designofexperimentsapplications,Chap.26.
• Inputselectionapplications,Chap.27.
• Localmodelnetworktoolbox,Chap.29.
This book focuses mainly on the key benefits that local model networks offer,
compared to alternative model architectures. These features are presented with
significantwidthanddepth.Additionally,extensivelytreatednewtopicsare:
• Axis-obliquepartitioningstrategies,inparticular,HILOMOT.
• Design of experiment strategies utilizing the special structure of local model
networks.
• Input selection strategies exploiting the possibility for distinct input spaces in
local model networks, namely the input space where the local models live, and
theinputspacewherethepartitioningtakesplace.
• Nonlinearfiniteimpulseresponse(NFIR)localmodelnetworksthattakeadvan-
tage of the input space separation mentioned above and the local estimation
approachforthelinearmodels.
• Bagging, boosting, regularization, and knowledge discovery in the context of
localmodelnetworks.
Furthermore, besides a couple of small corrections, the following issues are
addressedwhichwerenewlydiscoveredorgainedhighersignificancesincethefirst
editionofthisbook:
• Deep learning was discovered/invented since the first edition of this book. Due
toitsoverwhelmingsuccess,wholemonographshavebeenwrittenonthistopic.
Section11.2.9touchesthekeyideasandmajoringredientsofdeeplearning.
• Long short-term memory (LSTM) networks are extremely promising neural
networkarchitectureswithinternaldynamics.Theyareverysuccessfullyapplied
v
vi PrefacetotheSecondEdition
tomachinelearningproblemswithcomplexdynamics,andcurrentlytheirprop-
ertiesinanengineeringcontextforsystemidentificationareanopenquestionof
futureresearch.RefertoSect.23.5forashortexplanationofLSTMnetworksin
theterminologyandlanguageofthisbook.
• Excitationsignaldesignfornonlineardynamicprocessesisaveryimportantbut
neglectedfieldinresearchonsystemidentification.Anewapproachbasedrather
on an optimal signal generator than on a certain signal type is proposed and
analyzed.RefertoSects.19.8and26.4.
• Amore elaborate discussionof finite impulse response (FIR) models addresses
the important progress made in this field. A detailed comparison between FIR
andARXmodelsiscarriedoutmethodically.Duetothenewandverypromising
developmentsinregularizedFIRmodelestimation,theprosandconsof(N)ARX
vs.(N)FIRbecomesmuchmoreimportant.RefertoSects.18.6.1and18.6.2for
thelinearcaseandtoSect.22.7forthenonlinear,localmodelnetworkcase.
• L -norm regularization has significantly gained popularity as it is capable of
1
producingsparsemodels.Therefore,somebasicsandpossibilitiesarediscussed
inthisbook.RefertoSects.3.1.6and13.2.5.
Onthereal-worldapplications’side,twonewchaptersareincludedinthisedition
onthefollowingtopics:
• Designofexperimentsfortwostaticapplications(instructuralhealthmonitoring
and combustion engine modeling) with an active learning strategy exploiting
local model network characteristics. In addition, a new nonlinear dynamic
excitationsignalgeneratorisappliedtoacommonrailinjectionsystem.
• The power of input selection is demonstrated on a couple of processes of
different complexity. In particular, the feature of distinct input spaces for local
model networks is analyzed and exploited. Furthermore, metamodeling for the
efficiency modeling of fans depending on their geometrical characteristics is
discussed in detail. Data is generated by computational fluid dynamics (CFD)
simulations, which are orders of magnitude too slow for a fan efficiency
optimizationintheindustrycallingforfastmetamodels.
Finally, the reader is additionally pointed to more than 200 novel references,
mostly detailing the newly discussed topics and refreshing the state-of-the-art on
thematurematerial.
I would like to thank my dear colleagues Claus-Peter Fritzen and Thomas
Carolus for contributing to the applications part of this book via successful joint
projects.Furthermore,manythanksfornumerousfruitfuldiscussionstomyformer
and current research assistants Benjamin Hartmann, Tobias Ebert, Julian Belz,
Geritt Kampmann, Mark Schillinger, Tim Oliver Heinz, Tobias Münker, and Max
Schüssler.
Siegen,Germany OliverNelles
March2020
Preface to the First Edition
The goal of this book is to provide engineers and scientists in academia and
industry with a thorough understanding of the underlying principles of nonlinear
system identification. The reader will be able to apply the discussed models and
methods to real problems with the necessary confidence and the awareness of
potential difficulties that may arise in practice. This book is self-contained in
the sense that it requires merely basic knowledge of matrix algebra, signals and
systems,andstatistics.Therefore,italsoservesasanintroductiontolinearsystem
identificationandgivesapracticaloverviewonthemajoroptimizationmethodsused
in engineering. The emphasis of this book is on an intuitive understanding of the
subjectandthepracticalapplicationofthediscussedtechniques.Itisnotwrittenin
atheorem/proofstyle;ratherthemathematicsiskepttoaminimumandthepursued
ideasareillustratedbynumerousfigures,examples,andreal-worldapplications.
Fifteen years ago, nonlinear system identification was a field of several ad-
hoc approaches, each applicable only to a very restricted class of systems. With
the advent of neural networks, fuzzy models, and modern structure optimization
techniques a much wider class of systems can be handled. Although one major
characteristicofnonlinearsystemsisthatalmosteverynonlinearsystemisunique,
tools have been developed that allow the use of the same approach for a broad
varietyofsystems.Certainly,amoreproblem-specificproceduretypicallypromises
superiorperformance,butfromanindustrialpointofviewagoodtradeoffbetween
developmenteffortandperformanceisthedecisivecriterionforsuccess.Thisbook
presentsneuralnetworksandfuzzymodelstogetherwithmajorclassicalapproaches
inaunifiedframework.Thestrictdistinctionbetweenthemodelarchitecturesonthe
one hand (Part II) and the techniques for fitting these models to data on the other
hand(PartI)triestoovercometheconfusingmixturebetweenboththatisfrequently
encountered in the neuro and fuzzy literature. Nonlinear system identification is
currently a field of very active research; many new methods will be developed
and old methods will be refined. Nevertheless, I am confident that the underlying
principleswillcontinuetobevaluableinthefuture.
Thisbookoffersenoughmaterialforatwo-semestercourseonoptimizationand
nonlinear system identification. A higher level one-semester graduate course can
vii
viii PrefacetotheFirstEdition
focusonChap.7,thecompletePartII,andChaps.19,20,21,22,and23inPartIII.
For a more elementary one-semester course without prerequisites in optimization
and linear system identification Chaps.1, 2, 3, 4, and 18 might be covered, while
Chaps.10, 14, 20, and 23 can be skipped. Alternatively, a course might omit the
dynamicsystemsinPartIIIandinsteademphasizetheoptimizationtechniquesand
nonlinear static modeling treated in Parts I and II. The applications presented in
Part IV focus on a certain model architecture and will convince the user of the
practicalusefulnessofthediscussedmodelsandtechniques.Itisrecommendedthat
the reader should complement theses applications with personal experiences and
individualprojects.
Many people supported me while I was writing this book. First of all, I
would like to express my sincerest gratitude to my Ph.D. advisor, Professor Rolf
Isermann, Darmstadt University of Technology, for his constant help, advice, and
encouragement. He gave me the possibility of experiencing the great freedom and
pleasureofindependentresearch.Duringmywonderfulbutmuchtooshortstayasa
postdocattheUniversityofCaliforniainBerkeley,ProfessorMasayoshiTomizuka
gavemetheprivilegeoftakingutmostadvantageofallthatBerkeleyhastooffer.It
hasbeenanamazingatmosphereofinspiration.Thelast6yearsinDarmstadtand
Berkeley have been the most rewarding learning experience in my life. I am very
grateful to Professor Isermann and Professor Tomizuka for giving me the chance
to teach a graduate course on neural networks for nonlinear system identification.
Earlier versions of this book served as a basis for these courses, and the feedback
formthestudentscontributedmuchtoitsimprovement.
IhighlyappreciatethehelpofmycolleaguesinDarmstadtandBerkeley.Their
collaboration and kindness made this book possible. Big thanks go to Dr. Martin
Fischer,AlexanderFink,SusanneTöpfer,MichaelHafner,MatthiasSchüler,Martin
Schmidt,DominkFüssel,PeterBallé,ChristophHalfmann,HenningHolzmann,Dr.
Stefan Sinsel, Jochen Schaffnit, Dr. Ralf Schwarz, Norbert Müller, Dr. Thorsten
Ullrich, Oliver Hecker, Dr. Martin Brown, Carlo Cloet, Craig Smith, Ryan White,
andBrigitteHoppe.
Finally, I want to deeply thank my dear friends Martin and Alex and my
wonderful family for being there, whenever I needed them. I didn’t take it for
granted.Thisbookisdedicatedtoyou!
Kronberg,Germany OliverNelles
June2000
Contents
1 Introduction............................................................... 1
1.1 RelevanceofNonlinearSystemIdentification .................. 1
1.1.1 LinearorNonlinear?................................... 2
1.1.2 Prediction............................................... 2
1.1.3 Simulation.............................................. 4
1.1.4 Optimization ........................................... 4
1.1.5 Analysis ................................................ 5
1.1.6 Control ................................................. 5
1.1.7 FaultDetection......................................... 6
1.2 ViewsonNonlinearSystemIdentification....................... 6
1.3 TasksinNonlinearSystemIdentification........................ 7
1.3.1 ChoiceoftheModelInputs ........................... 9
1.3.2 ChoiceoftheExcitationSignals...................... 11
1.3.3 ChoiceoftheModelArchitecture..................... 11
1.3.4 ChoiceoftheDynamicsRepresentation.............. 12
1.3.5 ChoiceoftheModelOrder............................ 13
1.3.6 ChoiceoftheModelStructureandComplexity...... 13
1.3.7 ChoiceoftheModelParameters...................... 14
1.3.8 ModelValidation....................................... 14
1.3.9 TheRoleofFiddleParameters........................ 15
1.4 WhiteBox,BlackBox,andGrayBoxModels.................. 17
1.5 OutlineoftheBookandSomeReadingSuggestions ........... 18
1.6 Terminology....................................................... 20
PartI Optimization
2 IntroductiontoOptimization............................................ 23
2.1 OverviewofOptimizationTechniques........................... 25
2.2 Kangaroos......................................................... 25
ix
x Contents
2.3 LossFunctionsforSupervisedMethods......................... 28
2.3.1 MaximumLikelihoodMethod ........................ 30
2.3.2 MaximumAPosterioriandBayesMethod........... 32
2.4 LossFunctionsforUnsupervisedMethods...................... 34
3 LinearOptimization...................................................... 35
3.1 LeastSquares(LS)................................................ 37
3.1.1 CovarianceMatrixoftheParameterEstimate........ 45
3.1.2 Errorbars ............................................... 47
3.1.3 OrthogonalRegressors ................................ 50
3.1.4 Regularization/RidgeRegression ..................... 51
3.1.5 RidgeRegression:AlternativeFormulation.......... 57
3.1.6 L Regularization...................................... 60
1
3.1.7 NoiseAssumptions .................................... 61
3.1.8 WeightedLeastSquares(WLS)....................... 63
3.1.9 RobustRegression..................................... 65
3.1.10 LeastSquareswithEqualityConstraints ............. 66
3.1.11 SmoothingKernels .................................... 67
3.1.12 EffectiveNumberofParameters ...................... 70
3.1.13 L Boosting ............................................ 71
2
3.2 RecursiveLeastSquares(RLS) .................................. 74
3.2.1 ReducingtheComputationalComplexity ............ 75
3.2.2 TrackingTime-VariantProcesses..................... 77
3.2.3 RelationshipBetweentheRLSandtheKalman
Filter.................................................... 78
3.3 LinearOptimizationwithInequalityConstraints................ 79
3.4 SubsetSelection................................................... 80
3.4.1 MethodsforSubsetSelection ......................... 81
3.4.2 OrthogonalLeastSquares(OLS)forForward
Selection................................................ 85
3.4.3 RidgeRegressionorSubsetSelection?............... 89
3.5 Summary .......................................................... 90
3.6 Problems .......................................................... 91
4 NonlinearLocalOptimization .......................................... 93
4.1 BatchandSampleAdaptation.................................... 95
4.1.1 Mini-BatchAdaptation................................ 97
4.1.2 SampleAdaptation..................................... 97
4.2 InitialParameters ................................................. 99
4.3 DirectSearchAlgorithms......................................... 101
4.3.1 SimplexSearchMethod............................... 101
4.3.2 Hooke-JeevesMethod ................................. 104
4.4 GeneralGradient-BasedAlgorithms............................. 105
4.4.1 LineSearch ............................................ 106
4.4.2 FiniteDifferenceTechniques.......................... 108
4.4.3 SteepestDescent....................................... 109
