Reinforcement Learning and Approximate Dynamic Programming for Feedback Control

633 Pages·2012·44.479 MB·English

by  Frank L. Lewis, Derong Liu

REINFORCEMENT
LEARNINGA ND
APPROXIMTAE DYNAMIC
PROGRAMMINGF OR
FEEDBACKC ONTROL
IEEEP ress
445H oesL ane
PiscataNwJa y08,8 54
IEEE Press EdiBtooarrid2a 0l1 2
JohnA ndersoEnd, iitnoC rh ief
RamesAhb hari BernhardM .H aemmerliS aeiNda havandi
GeorgWe. A rnold DaviJda cobson Tariq Samad
FlaviCoa navero MaryL anzerotti GeorgZeo brist
DmitrGyo ldgof Om PM.a lik
KennetMho oreD,i reocfIt EoErBE o oakn Idnf ormaSteirovn(BIS)i ces
REINFORCEMENT
LEARNINAGN D
APPROXIMTAED YNAMIC
PROGRAMMINFGO R
FEEDBACKC ONTROL
Editbeyd
FranLk. L ewis
UTAA utomaatnRidoo bno Rteiscesa Irncsht itute
ForWto rTtXh ,
DeronLgi u
UniveorfIls ilintyo is
ChicaIgLo ,
+IEEE
IEEEP RESS
�WILEY
A JOHNWI LEY & SONSI,N C.P,U BLICATION
CoveIrl lustraCtoiuornt:eo sfy FranLke wLi.sa ndD eronLgi u
CoveDre sigJno:h nW ile&y SonsI,n c.
Copyrig©h 2t0 13b yT heI nstitoufEt lee ctraincdaE ll ectronics Engineers, Inc.
PublishbeyJd o hnW ile&y SonsI,n cH.o,b okeNne,w JerseAyl.lr ighrtess erved
Published simulitnaC naenoaudsal y
No parotf t hipsu blicamtaiyob ne r eproducsetdo,ri enda r etriesvyaslt eomr,t ransmiitnta endyf ormo r
bya nym eans, elecmtercohnainci,cp ahlo,t ocopyriencgo,r ding, socra nontihnegr,ew xicseepa,ts
permitutnedde Sre cti1o0n7 o r 108 o1f9 7t6Uh nei teSdt atCeosp yrigAhcttw ,i thoeuitt herp rtihoer
writtpeenr missoifot nh eP ublishoerar u,t horiztahtrioounpg ahy menotft hea ppropriate pfeere- copy
to tChoep yrigChlte aranCceen teIrn,c 2.2,2 R osewooDdr ivDea,n verMsA, 01923(,9 787)5 0-8400,
fax( 9787)5 0-447o0r, on thaet w wwewb. copyrightR.ecqoume.st tots h eP ublisfhoerpr e rmission
shoulbdea ddrestsoetd h eP ermissiDoenpsa rtmeJnoth,nW ile&y SonsI,n cI.,I IRi veSrt reeHotb,o ken,
NJ0 7030(,2 017)4 8-601f1a,x( 2017)4 8-600o8ro, n lianteh ttp://www..cwoiml/egyo/permission.
Limiotf LiabilitylDiosfcW laarirmaenrtW yh:i lteh ep ublisahnedra uthhoarv eus etdh ebiers etffo rtsi n
prepartihnigbs o okt,h ey mankore e presentoartw iaornrsa ntwiietsrh e spetcott h ea ccuraocry
completeonefts hsec ontenotfts h ibso oka nds pecifidciaslcllya ainmyi mpliweadr rantoife s
merchantaboirfl iittnyfe osrs p aar ticpuulrapro sNeo. w arranmtayy b ec reatoerde x tendbeyd sales
representoartw irvietsts eanl emsa teriTahlesa. d vicaen ds tratecgoinetsa ihneerde miany n otb e suitable
fory ousri tuations.h oYuolcudo nsuwlitt ahp rofessiwohnearlae p propriNaetiet.ht ehrep ublisnhoerr
authoshra lblel iabfloera nyl ososf p rofoirat n yo thecro mmercial damages,b uitnn octll iumdiittneogd
speciianlc,i denctoanls,e quentioatlh,ed ra omra ges.
Forg enerianlf ormatoinoo unr o theprr oducatnsds erviocre sf toerc hniscuaplp oprlte,a csoen taocutr
CustomCearr Dee partmewnitt htihne UniStteadta ets( 8007)6 2-297o4u,t sitdhee UniStteadta ets( 317)
572-39o9r3f ax( 3175)7 2-4002.
Wileayl spou blisihtebsso okisn a varieotfey l ectrfoonrimca tSso.m ec ontetnhta atp peairnsp rint may
notb ea vailaibnel lee ctrfoornmiact sF.o rm orei nformataiboonuW ti lepyr oducvtiss,oi utr web asti te
www.wiley.com.
LibroafCry o ngrCeastsa loging-Diant-aP:u blication
Reinforcelmeeanrtn ianndga pproximdaytnea mipcr ogrammifnogfr e edbaccokn tr/oedlitebdyF rank
L.L ewisD,e ronLgi u.
p. cm.
ISBN9 78-1-118-10(4h2a0r-d0b ack)
I. Reinforcelmeeanrtn i2n.g . Feedbcaocnkt rsoyls tem1s..L ewisF,r ank L.
II.L iu,D eron1g9,6 3-
Q325.6.R46240 12
003!.5-dc23
2012019014
Printientd h eU niteSdt atoefsA merica
10 9 8 7 6 5 I4 3 2
CONTENTS
PREFACE xix
CONTRIBUTORS xxiii
PART I FEEDBACK CONTROL USINGR L AND ADP
1. ReinforceLmeeanrtn ianngd A pproximaDtyen amic
Programmi{RnLgA DP)-FoundatioCnosm,m on
Misconceptainodnt sh,eC hallenAgheesa d 3
PauJl Werbos
1.1 Introduction 3
1.2 Whati sR LADP? 4
1.2.1D efinitoifoR nL ADPa ndt hTea sikt A ddresses 4
1.2.2B asiTco ols-BellEmqauna tiaonndV, a luaen dP olicy
Functions 9
1.2.3O ptimizaOtvieorTni meW ithoVuatl uFeu nctions 13
1.3 SomeB asiCch allenignIe msp lementAiDnPg 14
1.3.1A ccountfionUrgn seeVna riables 15
1.3.2O fflineC ontroDlelesri gVne rsRuesa l-TiLmeea rning 17
1.3.3" Model-BaVseerds"u" sM odeFlr eeD"e signs 18
1.3.4H owt oA pproximtahtVeea luFeu nctiBoent ter 19
1.3.5H owt oC hoosue ( tB)as eodn a V aluFeu nction 22
1.3.6H owt oB uilCdo operaMtuilvtei agSeynstt ewmist h
RLADP 25
References 26
2. StabAldea ptiNveeu raClo ntrooflP artiaOlblsye rvable
DynamiScy stems 31
J NatKen ighatn dC harlWe sA nderson
2.1 Introduction 31
2.2 Background 32
2.3 StabiBliiatsy 35
2.4 ExamplAep plication 38
2.4.1T heS imulaStyesdt em 38
2.4.2A n UncertLaiinne Palra nMto del 40
v
vi CONTENTS
2.4.3T heC loseLdo opC ontrSoyls tem 41
2.4.4D eterminRiNnNg W eighUtp datbeysR einforcement
Learning 44
2.4.5R esults 46
2.4.6C onclusions 50
References 50
3. OptimaClo ntrooflU nknowNno nlineDairs crete-STyismtee ms
Usintgh eI teratGilvoeb aliDzueadlH eurisPtriocg ramming
Algorithm 52
DerongL iua ndD ingW ang
3.1 BackgroMuantde rial 53
3.2 Neuro-OptCiomnatlr Soclh emBea seodn t heIt eraAtDiPv e
Algorithm 55
3.2.1I dentificoaftt ihoeUn n knowNno nlinSeyasrt em 55
3.2.2D erivatoifto hneI t eraAtDiPvA el gorithm 59
3.2.3C onvergeAnncael ysoifts h eI teraAtDiPvA el gorithm 59
3.2.4D esigPnr ocedoufrt eh eI teraAtDiPvA el gorithm 64
3.2.5N N Implementoaftt ihoeIn t eraAtDiPvA el goritUhsmi ng
GDHPT echnique 64
3.3 Generalization 67
3.4 SimulatSitound ies 68
3.5 Summary 74
References 74
4. Learnianngd O ptimizatiinHo ine rarchAidcaaplt iCvrei tic
Design 78
HaibHoe ,Z henN i,a ndD ongbiZnh ao
4.1 Introduction 78
4.2 HierarchAiDcPaA lr chitecwtiutMrhue l tiple-Goal
Representation 80
4.2.1S ysteLme veSlt ructure 80
4.2.2A rchitecDteusriega nn dI mplementation 81
4.2.3L earnianngdA daptatiinHo ine rarchAiDcPa l 83
4.3 CaseS tudTyh:e B all-and-SByesatme m 87
4.3.1P roblFeomr mulation 88
4.3.2E xperimCeonntfi guraatnidoP na rametSeertsu p 89
4.3.3S imulatRieosnu latnsdA nalysis 90
4.4 ConclusiaonndFs u ture Work 94
References 95
5. SingNleet worAkd aptive CNreittwiocrsk s-Development,
Analysainsd,A pplications 98
lieD ingA,li H eydarain,d5. N Balakrishnan
5.1 Introduction 98
5.2 ApproximDaytnea miPCr ograming 100
CONTENTS vii
5.3 SNAC 102
5.3.1S tatGee neratfiooNrne uraNle tworTkr aining 103
5.3.2N euraNle tworTkr aining 103
5.3.3C onvergeCnocned ition 104
5.4 ]-SNAC 104
5.4.1N euraNle tworTkr aining 105
5.4.2N umericAanla lysis 105
5.5 FinitNeA-CS 108
5.5.1N euraNle tworTkr aining 109
5.5.C2o nvergeTnhceeo rems 111
5.5.3N umericAanla lysis 112
5.6 Conclusions 116
References 116
6. LinearSloyl vabOlpet imaClo ntrol 119
K. Dvijothaamn dE. Todorov
6.1 Introduction 119
6.1.1N otation 121
6.1.2M arkoDve cisiPorno cesses 122
6.2 LineaSrollyv abOlpet imal CoPnrtorbolle ms 123
6.2.1P robabiSlhiitfAytn : A lternVaiteewo fC ontrol 123
6.2.2L ineaSrollyv abMlaer koDve cisiPorno cesses
(LMDPs) 124
6.2.3A n AlternVaiteewo fL MDPs 124
6.2.4O thePrr oblFeomr mulations 126
6.2.5A pplications 126
6.2.6L ineaSrollyv abCloen trolDliefdf usi(oLnDss ) 127
6.2.7R elationBsehtiwpe eDni scraentdeC ontinuous-Time
Problems 128
6.2.8H istorPiecraslp ective 129
6.3 ExtenstiooR ni sk-SensCiotnitvraeon ldG ameT heory 130
6.3.1G ameT heoreCtoinct rCoolm:p etitGiavmee s 130
6.3.2R enyDii vergence 130
6.3.3L ineaSrollyv abMlaer koGva mes 130
6.3.4L ineaSrollyv abDlief fereGnatmieasl 133
6.3.5R elationAsmhoinpgst heD iffereFnotr mulations 134
6.4 ProperatnideA sl gorithms 134
6.4.1S ampliAnpgp roximatainodPn ast h-Integral Con1t3r4o l
6.4.2R esiduMailn imizavtiiFaou nn ctiAopnp roximation 135
6.4.3N aturPaoll iGcrya dient 136
6.4.4C ompositioonfaO lpittiym Caoln trLoalw s 136
6.4.5S tochaMsatxiicm umP rinciple 137
6.4.6I nverOspet imCaoln trol 138
6.5 ConclusiaonndFs u tuWroer k 139
References 139
viii CONTENTS
7. ApproximatOipntgi maClo ntrwoilt Vha luGer adieLneta rning 142
MichaFeali rbaDnakn,i Plm khomv,a ndE duardAol onso
7.1 Introduction 142
7.2 ValuGer adieLneta rnianngdB PTT Algorithms 144
7.2.1P relimiDneafirnyi tions 144
7.2.2V GL(A)A lgorithm 145
7.2.3B PTTA lgorithm 147
7.3 A ConvergePnrcoeof fo Vr GL(1f)o Cro ntrwoilt Fhu nction
Approximation 148
7.3.1U sinag G reedPyo liwciyt ahC ritFiucn ction 149
7.3.2T heE quivaleonfVcGeL (1)t oB PTT 151
7.3.3C onvergeCnocned itions 152
7.3.4N oteosn t heS"2 tM atrix 152
7.4 VertiLcaanld eErx periment 154
7.4.1P roblDeemfi nition 154
7.4.2E fficieEnvta luatoifto hneG reedPyo licy 155
7.4.3O bservatoinot nhseP urpoosfeS"2 t 157
7.4.4E xperimeRnetsaull ftosVr e rtical PLraonbdleerm 158
7.5 Conclusions 159
References 160
8. A Constrained BackprAopppargoaatctihooF n u nction
ApproximatainodAn p proximaDtyen amiPcr ogramming 162
SilvFiear rarKie.i tRhu dda.n dG ianluDciMa u m
8.1 Background 163
8.2 ConstraBiancekdp ropag(aCtPiRoOnP A)p proach 163
8.2.1N euraNle tworAkr chitecatnudPr reo cedural
Memories 165
8.2.2D erivatoifLo TnM EqualCiotnys traainndAt dsj oinEerdr or
Gradient 165
8.2.3E xamplIen:c remental AFpupnrcotxiiomna tion 168
8.3 SolutoifoP na rtDiiaflf ereEnqtuiaatli ionnN so nstationary
Environments 170
8.3.1C PROPS olutoifoB no undaVrayl uPer oblems 170
8.3.2E xamplPeD:E S olutoinoa nU niCti rcle 171
8.3.3C PROPS oluttiooP na raboPlDiEcs 174
8.4 PreservPirnigoK rn owledignEe x ploraAtdoarpyt iCvrei tic
Designs 174
8.4.1D erivatoifLo TnM ConstrafionFrte se dbaCcokn trol 175
8.4.2C onstraAidnaepdt iCvrei tDiecs ign 177
8.5 Summary 179
AppendiAxl:g ebrAaNiNc ContrMoalt rices 180
References 180
CONTENTS ix
9. TowarDde sigonfN onlineAaDrP LearniCnogn trolwlietrhs
PerformaAnscseu rance 182
JennSiieL, e iY angC,h aoL u,K ostaS.s T sakalainsd,
ArmandAo.R odriguez
9.1 Introduction 183
9.2 Direct HeuDryinsatmiiPccr ogramming 184
9.3 A Control ThVeoireewot nti hceD irect HDP 186
9.3.1P roblSeemt up 187
9.3.2F requenDcoym aiAnn alysoifDs i reHcDtP 189
9.3.3I nsigfhrto Cmo mpariDnigr eHcDtP t oL QR 192
9.4 DireHcDtP Desigwni tIhm provPeedr formance
CaseI -DesigGnu idebdy a P riorLiQ R Information 193
9.4.1D ireHcDtP DesigGnu idebdy a P rioLriQ R
Information 193
9.4.2P erformaonftc heDe ireHcDtP BeyonLdi nearizatio1n9 5
9.5 DireHcDtP Desigwni tIhm proved PerfCoarsm2ea- nDcier ect
HDPf oCro orindated DCaomnptironofglL ow-Frequency
Oscillation 198
9.6 Summary 201
References 202
10.R einforceLmeeanrtn iCnogn trwoilt Thi me-DependAegnetn t
Dynamics 203
KentoKni rkpatarnidJc okh nVa lasek
10.1I ntroduction 203
10.2Q -Learning 205
10.2.Q1- LearnAilnggo rithm 205
10.2..s2- Greedy 207
10.2.F3u nctiAopnp roximation 208
10.3S ampleDda tQa- Learning 209
10.3.S1a mpleDda tQa- Learning Algorithm 209
10.3.E2x ample 210
10.4S ysteDmy namiAcpsp roximation 213
10.4.F1i rst-ODrydnearm iLcesa rning 214
10.4.M2u ltiagSeynstt eTmh ougEhxtp eriment 216
10.5C losiRnegm arks 218
References 219
11.O nlinOep timaClo ntrooflN onaffinNeo nlineDairs crete-Time
Systemwsi thoUusti nVga luaen dP oliIctye rations 221
HassaZna rgarzaQdienhm iYna nga,n dS. Jagannathan
11.1I ntroduction 221
11.2B ackground 224
11.3R einforceLmeeanrtn iBnags eCdo ntrol 225
11.3.A1ffi ne-LiDkyen amiCS 225
11.3.O2n linRee inforceLmeeanrtn iCnogn troDlelSeirg n 229
X CONTENTS
11.3.T3h eA ctioNnN Design 229
11.3.T4h eC ritNiNc D esign 230
11.3.W5e ighUtp datiLnagw sf otrh eN Ns 231
11.3.M6a inT heoreRteiscu lts 232
11.4T ime-BaAsdeadp tiDvyen amic ProgrammiOnpgt-iBmaasle d
Control 234
11.4.O1n linNeN- BaseIdd entifier 235
11.4.N2e uraNle twork-BOapsteidm Caoln troDlelSeirg n 237
11.4.C3o sFtu nctiAopnp roximaftoiOrop nt imRaelg ulator
Design 238
11.4.E4s timatoifto hneO ptimFaele dbaCcokn trSoilg nal 240
11.4.C5o nvergePnrcoeo f 242
11.4.R6o bustness 244
11.5S imulatRieosnu lt 247
11.5.R1e inforcement-LearCnoinntgr-ooBflaa N soendl inear
System 247
11.5.T2h eD rawbaocfkH DP PoliIctye raAtpiporno ach 250
11.5.O3L A-BasOepdt imCaoln trAoplp liteodH CCIE ngine 251
References 255
12.A n Actor-Critic-IAdrecnhtiitfieecrft ourAr dea ptive
ApproximaOtpet imaClo ntrol 258
S. BhasiRn.,K amaJapurMk alr;o hnsoKn C,. Va mvoudakis,
F.I.L ewiasn,d W E. Dixon
12.1I ntroduction 259
12.2A ctor-Critic-AIrdcehnittieficeftrouH rr] eBA pproximation 260
12.3A ctor-CrDietSiicg n 263
12.4I dentiDfieesri gn 264
12.5C onvergeanncdSe t abiAlniatly ysis 270
12.6S imulation 274
12.7C onclusion 275
References 278
13.R obusAtd aptiDvyen amiPcr ogramming 281
Yul ianagn dZ hong-Pliinagn g
13.1I ntroduction 281
13.2O ptimalVietrys Ruosb ustness 283
13.2.S1y stewmist Mha tcheDdi sturbance Input 283
13.2.A2d dinOgn eI ntegrator 284
13.2.S3y steimnsL ower-Triangular Form 286
13.3R obust-ADDePs igfno Dri sturbAatntceen uation 288
13.3.H1o rizonLteaalr ning 288
13.3.V2e rtical Learning 290
13.3.3 Robust-ADPf oADrli gsotruirtbhAamtn tceen uation 291
13.4R obust-AfDoPPr a rtial-FSeteadtbeaC cokn trol 292