Output Feedback Reinforcement Learning Control for Linear Systems

304 Pages·2022·5.865 MB·English

by  Syed Ali Asad Rizvi, Zongli Lin

Control Engineering
Syed Ali Asad Rizvi
Zongli Lin
Output Feedback
Reinforcement
Learning Control
for Linear
Systems

Control Engineering
SeriesEditor
WilliamS.Levine,DepartmentofElectricalandComputerEngineering,University
ofMaryland,CollegePark,MD,USA
EditorialBoardMembers
RichardBraatz,RoomE19-551,MIT,Cambridge,MA,USA
Graham C Goodwin, School of Elect Eng and Com Sc, University of Newcastle,
Callaghan,Australia
Zongli Lin, Electrical and Computer Engineering, University of Virginia, Char-
lottesville,VA,USA
MarkW.Spong,ErikJonssonSchoolofEng&Comp,UniversityofTexasatDallas,
Richardson,USA
Maarten Steinbuch, Mechanical Engineering, Technische Universiteit Eindhoven,
EINDHOVEN,Noord-Brabant,TheNetherlands
MathukumalliVidyasagar,UniversityofTexasatDallas,Cecil&IdaGreenChair
inSystemsBiolo,Richardson,TX,USA
YutakaYamamoto,Dept.ofAppliedAnalysis,KyotoUniversity,Sakyo-ku,Kyoto,
Kyoto,Japan
Syed Ali Asad Rizvi • Zongli Lin
Output Feedback
Reinforcement Learning
Control for Linear Systems
SyedAliAsadRizvi ZongliLin
ElectricalandComputerEngineering ElectricalandComputerEngineering
TennesseeTechnologicalUniversity UniversityofVirginia
Cookeville,TN,USA Charlottesville,VA,USA
ISSN2373-7719 ISSN2373-7727 (electronic)
ControlEngineering
ISBN978-3-031-15857-5 ISBN978-3-031-15858-2 (eBook)
https://doi.org/10.1007/978-3-031-15858-2
MathematicsSubjectClassification:49N05,49N30,49N90,93C40,68T05
©SpringerNatureSwitzerlandAG2023
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
This book is published under the imprint Birkhäuser, www.birkhauser-science.com by the registered
companySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Weliveinatechnology-drivenworldthatisstronglydependentontheoperationof
complexsystemsfoundinautomobiles,aerospace,industrialautomation,manufac-
turingsystems,communication,computerssystems,defense,medicine,andfinance.
Smooth and efficient operation of such systems is ensured by means of automatic
controlanddecision-makingmethods.Controltheoryprovidesmathematicaltools
for designing control systems to enable desired system behavior and ensure stable
operationofthesesystems.Withtheincreasingcomplexityofsystems,thedemand
for extracting better and better performance from these control systems has now
becomearequirement.Whilestabilityisessential,itisonlyaminimumrequirement
for the operation of acontrol system.Itisdesirable that acontrol systemoperates
inaprescribedoptimalmannerthatmeetscertaincostandperformanceobjectives.
Traditional control theory has been pushed to its limits, and new techniques are
being sought that can keep up with the accelerating evolution pace of today’s
complexsystemsandtheirperformancespecifications.
Optimal control theory is one of the founding pillars of modern control that
enables the synthesis of optimal controllers. An optimal control problem is the
problem of designing a control law that fulfills a control task while minimizing a
prescribedcostfunctionthatreflectsthebalancebetweentheperformanceachieved
and the available resources consumed. The foundations of optimal control theory
werelaidinthe1950sbasedonthePontryagin’sminimumprincipleandBellman’s
theory of dynamic programming. Formally, the solution of the optimal control
problem is obtained by solving the Hamilton-Jacobi-Bellman (HJB) equation.
Although mathematically elegant, the HJB equation is intractable because it is a
partialdifferentialequation.Forthespecialcaseoflinearsystemswithaquadratic
cost function, solving the HJB equation amounts to finding the solution of an
algebraic Riccati equation (ARE). Iterative computational schemes have been
developed to solve AREs. All these optimal control design algorithms are offline
innatureandrequireacompleteknowledgeofthesystemdynamics.
Optimal control theory is fundamentally a model-based control paradigm that
employs a system model to obtain the optimal control law. The HJB equation and
theAREincorporatethecompleteknowledgeofthedynamicsystem,andtherefore,
v
vi Preface
the optimality of the solution is subject to the accuracy of the system model. The
requirement of an accurate system model is hard to satisfy in practice because
of the presence of unavoidable uncertainties. Furthermore, these uncertainties add
up as the complexity of system increases. More often, the modeling and system
identificationprocessisundesirableowingtothecostandeffortassociatedwiththis
process. As a result, the dynamics of the underlying system is often unknown for
thecontroldesign.Evenwhenthesystemdynamicsisavailableintheearlydesign
phase, the system parameters are subject to change over the operating life span of
thesystemduetovariationsintheprocessitself.Thesemodelvariationsmayarise
asaresultofaging,faults,orsubsequentupgradesofthesystemcomponents.Thus,
itisdesirabletodevelopoptimalcontrolmethodsthatdonotrelyontheknowledge
ofthesystemdynamics.
Machinelearningisanareaofartificialintelligenceandinvolvesthedevelopment
ofcomputationalintelligencealgorithmsthatlearnbyanalyzingthesystembehavior
insteadofbeingexplicitlyprogrammedtoperformcertaintasks.Recentadvancesin
machinelearninghaveopenedanewavenueforthedesignofintelligentcontrollers.
Machine learning techniques come into the picture when accurate system models
are not known but examples of system behavior are available or a measure of the
goodnessofthebehaviorcanbeassigned.Machinelearningalgorithmsareformally
categorized into three types based on the kind of supervision available, namely
supervisedlearning,unsupervisedlearning,andreinforcementlearning.Supervised
learningisundoubtedlythemostcommontypeofmachinelearningwhichfindsuse
in applications where examples of the desired behavior are available in the form
of input-output training data. However, the nature of control problems requires
selecting control actions whose consequences emerge over time. As a result, the
optimal control inputs that achieve the desired output are not known in advance.
In these scenarios, reinforcement learning can be used to learn the desired control
inputs by providing the controller with a suitable evaluation of its performance
duringthecourseoflearning.
Today, more and more engineering designs are finding inspiration from our
mother nature. Living organisms tend to learn, adapt, and optimize their behavior
overtimebyinteractingwiththeirenvironment.Thisisthemainmotivationbehind
reinforcementlearningtechniques,whicharecomputationalintelligencealgorithms
based on the principle of action and reward that naturally incorporate a feedback
mechanismtocaptureoptimalbehavior.Theideasofadaptabilityandoptimalityare
alsopresentinthecontrolcommunityintheformofadaptivecontrolanddynamic
programming (optimal control). However, reinforcement learning has provided a
newdirectionoflearningoptimaladaptivecontrolbyobservingarewardstimulus.
Recently,theideaofusingreinforcementlearningtosolveoptimalcontrolproblems
hasattractedalotofattentioninthecontrolcommunity.
ThetreatmentofRLproblemsincontrolsettingscallsforamoremathematically
rigorousformulationsothatconnectionswiththefundamentalcontrolconceptssuch
as stability, controllability, and observability can be made. We start to see to such
sophisticationsrightfromthebeginningwhenwearerequiredtoselectarigorously
formulatedobjectivefunctionthatsatisfiescertaincontrolassumptions.
Preface vii
While significant progress has been made in recent years, the reinforcement
learning (RL) control paradigm continues to undergo developments more rapidly
thaneverbefore.Reinforcementlearningfordynamicsystemsrequirestotakeinto
accountthesystemdynamics.Controlofdynamicsystemsgivesprimeimportance
to the closed-loop stability. Control algorithms are required to guarantee closed-
loopstability,whichisabareminimumrequirementforafeedbacksystem.Control
systemswithoutperformance,robustness,andsafetymarginsarenotacceptableby
theindustry.CurrentdevelopmentsintheRLcontrolliteraturearedirectedtowards
makingRLalgorithmsapplicableinrealworldscenarios.
ThisbookfocusesontherecentdevelopmentsinthedesignofRLcontrollersfor
generallinearsystemsrepresentedbystatespacemodels,eitherincontinuous-time
orindiscrete-time.ItisdedicatedtothedesignofoutputfeedbackRLcontrollers.
WhiletheearlydevelopmentsinRLalgorithmshavebeenprimarilyattributedtothe
computersciencecommunitywhereitisofpracticalinteresttoposetheproblemin
thestochasticsetting,thescopeofthisresearchmonographistowardsenhancingthe
output feedback capability of the mainstream RL algorithms developed within the
control engineering community, which are primarily in the deterministic setting.
This specialized choice of topics sets this research monograph apart from the
leading mainstream books that develop the conceptual foundations of the theory
of approximate dynamic programming [8], reinforcement learning [9], and their
connection with feedback control [54, 68]. The book presents control algorithms
that are aimed to deal with the challenges associated with learning under limited
sensing capability. Fundamental to these algorithms are the issues of exploration
bias and stability guarantee. Model-free solutions to the classical optimal control
problemssuchasstabilizationandtrackingarepresentedforbothdiscrete-timeand
continuous-time dynamic systems. Output feedback control for linear systems is
currentlymoreformalizedinthelatestliteraturecomparedtotheongoingextensions
fornonlinearsystems;therefore,thediscussionsforthemostpartarededicatedto
linearsystems.
The output feedback formulation presented in this book differs from the tra-
ditional design method that involves a separate observer design. As such, the
emphasis of the monograph is not on optimal state estimation, as is involved in
an observer and control design process that leads to the optimal output feedback
control. Instead, our motivation is to learn the optimal control parameters directly
basedonastateparameterizationapproach.Thisinturncircumventstheseparation
principleofperformingtwo-steplearningforoptimalstateestimationandoptimal
controldesign.
The results covered in the book extend beyond the classical problems of
stabilizationandtracking.Avarietyofpracticalchallengesarestudiedrangingfrom
the disturbance rejection, control constraints, and communication delays. Model-
free H∞ controllers are developed using output feedback based on the principles
of game theory. The ideas of low gain feedback control are employed to develop
RL controllers that achieve global stability under control constraints. New results
on the design of model-free optimal controllers for systems subject to both state
andinputdelaysarepresentedbasedonanextendedstateaugmentationapproach,
viii Preface
whichrequiresneithertheknowledgeofthelengthsofthedelaysnortheknowledge
ofthenumberofthedelays.
Theorganizationofthebookisasfollows.InChap.1,weintroducethereaders
to optimal control theory and reinforcement learning. Fundamental concepts and
resultsarereviewed.Thischapteralsoprovidesabriefsurveyoftherecentdevelop-
mentsinthefieldofRLcontrol.ChallengesassociatedwiththeRLcontrollersare
highlighted.
Chapter 2 presents model-free output feedback RL algorithms to solve the
optimal stabilization problem. Both continuous-time systems and discrete-time
systemsareconsidered.Areviewofexistingmainstreamapproachesandalgorithms
is presented to highlight some of the challenges in guaranteeing closed-loop
stabilityandoptimality.Q-learningandintegralreinforcementlearningalgorithms
are developed based on the parameterization of the system state. The issues of
discountingfactorandexplorationbiasarediscussedindetail,andimprovedoutput
feedbackRLmethodsarepresentedtoovercomethesedifficulties.
Chapter 3 brings attention to the disturbance rejection control problem formu-
lated as an H∞ control problem. The framework of game theory is employed to
develop both continuous-time and discrete-time algorithms. A literature review is
provided to elaborate the difficulties in some of the recent RL-based disturbance
rejection algorithms. Q-learning and integral reinforcement learning algorithms
are developed based on the input-output parameterization of the system state, and
convergencetotheoptimalsolutionisestablished.
Chapter 4 presents model-free algorithms for global asymptotic stabilization
of linear systems subject to actuator saturation. Existing reinforcement learning
approachestosolvingtheconstrainedcontrolproblemarefirstdiscussed.Theidea
ofgain-scheduledlowgainfeedbackispresentedtodevelopcontrollawsthatavoid
saturationandachieveglobalasymptoticstabilization.Todesignthesecontrollaws,
weemploytheparameterizedARE-basedlowgaindesigntechnique.Reinforcement
learningalgorithmsbasedonthisapproacharethenpresentedtofindthesolutionof
theparameterizedAREwithoutrequiringanyknowledgeofthesystemdynamics.
The presented scheme has the advantage that the resulting control laws have a
linearstructureandglobalasymptoticstabilityisensuredwithoutcausingactuator
saturation.Bothcontinuous-timesystemsanddiscrete-timesystemsareconsidered.
The last two chapters build upon the fundamental results established in the
earlierchapterstosolvesomeimportantproblemsincontroltheorypractice,control
of systems in the presence of time delays, the optimal tracking problem, and
the multi-agent synchronization problem. In particular, Chap.5 focuses on the
controlproblemsinvolvingtimedelays.AreviewofsomeexistingRLapproaches
to addressing the time-delay control problem of linear systems is first provided.
The design of model-free RL controllers is presented based on an extended state
augmentation approach. It is shown that discrete-time delay systems with input
and/or state delays can be brought into a delay-free form. Q-learning is then
employedtolearntheoptimalcontrolparametersfortheextendeddynamicsystem.
Systemswitharbitrarilylargedelayscanbedealtwithusingthepresentedapproach.
Furthermore,thismethodrequiresneitherthenumberandnorthelengthsofdelays.
Preface ix
Chapter 6 develops model-free output feedback RL algorithms to solve the
optimaltrackingproblemalongwiththeirextensiontothesynchronizationproblem
of multi-agent systems. Existing RL approaches to solving a multi-agent synchro-
nization problem based on output feedback are first described. A distributed two
degreeoffreedomapproachispresentedthatseparatesthelearningoftheoptimal
outputfeedbackandthefeedforwardterms.Theapproachovercomestheprevailing
discountingfactorissueandensuresasymptoticoptimaloutputtracking.
This monograph was typeset by the authors using LATEX. All simulation and
numericalcomputationwerecarriedoutinMATLAB.
Cookeville,TN,USA SyedAliAsadRizvi
Charlottesville,VA,USA ZongliLin