Reinforcement Learning and Dynamic Programming Using Function Approximators (Automation and Control Engineering)

275 Pages·2010·8.34 MB·English

by  Lucian Busoniu

Reinforcement Learning
and Dynamic Programming
Using Function Approximators
K11117_FM.indd 1 3/22/10 4:10:24 AM
AUTOMATION AND CONTROL ENGINEERING
A Series of Reference Books and Textbooks
Series Editors
FRANK L. LEWIS, Ph.D., SHUZHI SAM GE, Ph.D.,
Fellow IEEE, Fellow IFAC Fellow IEEE
Professor Professor
Automation and Robotics Research Institute Interactive Digital Media Institute
The University of Texas at Arlington The National University of Singapore
Reinforcement Learning and Dynamic Programming Using Function
Approximators, Lucian Bus¸oniu, Robert Babuška, Bart De Schutter,
and Damien Ernst
Modeling and Control of Vibration in Mechanical Systems, Chunling Du
and Lihua Xie
Analysis and Synthesis of Fuzzy Control Systems: A Model-Based Approach,
Gang Feng
Lyapunov-Based Control of Robotic Systems, Aman Behal, Warren Dixon,
Darren M. Dawson, and Bin Xian
System Modeling and Control with Resource-Oriented Petri Nets, Naiqi Wu
and MengChu Zhou
Sliding Mode Control in Electro-Mechanical Systems, Second Edition,
Vadim Utkin, Jürgen Guldner, and Jingxin Shi
Optimal Control: Weakly Coupled Systems and Applications, Zoran Gajic´,
Myo-Taeg Lim, Dobrila Skataric´, Wu-Chung Su, and Vojislav Kecman
Intelligent Systems: Modeling, Optimization, and Control, Yung C. Shin
and Chengying Xu
Optimal and Robust Estimation: With an Introduction to Stochastic Control
Theory, Second Edition, Frank L. Lewis, Lihua Xie, and Dan Popa
Feedback Control of Dynamic Bipedal Robot Locomotion, Eric R. Westervelt,
Jessy W. Grizzle, Christine Chevallereau, Jun Ho Choi, and Benjamin Morris
Intelligent Freight Transportation, edited by Petros A. Ioannou
Modeling and Control of Complex Systems, edited by Petros A. Ioannou
and Andreas Pitsillides
Wireless Ad Hoc and Sensor Networks: Protocols, Performance, and Control,
Jagannathan Sarangapani
Stochastic Hybrid Systems, edited by Christos G. Cassandras
and John Lygeros
Hard Disk Drive: Mechatronics and Control, Abdullah Al Mamun,
Guo Xiao Guo, and Chao Bi
Autonomous Mobile Robots: Sensing, Control, Decision Making
and Applications, edited by Shuzhi Sam Ge and Frank L. Lewis
K11117_FM.indd 2 3/22/10 4:10:24 AM
Automation and Control
Engineering Series
Reinforcement Learning
and Dynamic Programming
Using Function Approximators
Lucian Busoniu
Delft University of Technology
Delft, The Netherlands
Robert Babuska
Delft University of Technology
Delft, The Netherlands
Bart De Schutter
Delft University of Technology
Delft, The Netherlands
Damien Ernst
University of Liège
Liège, Belgium
Boca Raton London New York
CRC Press is an imprint of the
Taylor & Francis Group, an informa business
K11117_FM.indd 3 3/22/10 4:10:24 AM
MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does
not warrant the accuracy of the text or exercises in this book. This book’s use or discussion of MAT-
LAB® software or related products does not constitute endorsement or sponsorship by The MathWorks
of a particular pedagogical approach or particular use of the MATLAB® software.
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2010 by Taylor and Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed in the United States of America on acid-free paper
10 9 8 7 6 5 4 3 2 1
International Standard Book Number: 978-1-4398-2108-4 (Hardback)
This book contains information obtained from authentic and highly regarded sources. Reasonable
efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and
publishers have attempted to trace the copyright holders of all material reproduced in this publication
and apologize to copyright holders if permission to publish in this form has not been obtained. If any
copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information stor-
age or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copy-
right.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222
Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that pro-
vides licenses and registration for a variety of users. For organizations that have been granted a pho-
tocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
K11117_FM.indd 4 3/22/10 4:10:24 AM
Preface
Control systems are making a tremendous impact on our society. Though invisible
tomostusers,theyareessentialfortheoperationofnearlyalldevices–frombasic
homeappliancestoaircraftandnuclearpowerplants.Apartfromtechnicalsystems,
theprinciplesofcontrolareroutinelyappliedandexploitedinavarietyofdisciplines
suchaseconomics,medicine,socialsciences,andartificialintelligence.
Acommondenominatorinthediverseapplicationsofcontrolistheneedtoin-
fluenceormodifythebehaviorofdynamicsystemstoattainprespecifiedgoals.One
approachtoachievethisistoassignanumericalperformanceindextoeachstatetra-
jectoryofthesystem.Thecontrolproblemisthensolvedbysearchingforacontrol
policythatdrivesthesystemalongtrajectoriescorrespondingtothebestvalueofthe
performance index. This approach essentially reduces the problem of finding good
controlpoliciestothesearchforsolutionsofamathematicaloptimizationproblem.
Early work in the field of optimal control dates back to the 1940s with the pi-
oneering research of Pontryagin and Bellman. Dynamic programming (DP), intro-
duced by Bellman, is still among the state-of-the-art tools commonly used to solve
optimalcontrolproblemswhenasystemmodelisavailable.Thealternativeideaof
findingasolutionintheabsenceofamodelwasexploredasearlyasthe1960s.In
the1980s,arevivalofinterestinthismodel-freeparadigmledtothedevelopmentof
thefieldofreinforcementlearning(RL).ThecentralthemeinRLresearchisthede-
signofalgorithmsthatlearncontrolpoliciessolelyfromtheknowledgeoftransition
samplesortrajectories,whicharecollectedbeforehandorbyonlineinteractionwith
thesystem.MostapproachesdevelopedtotackletheRLproblemarecloselyrelated
toDPalgorithms.
AcoreobstacleinDPandRListhatsolutionscannotberepresentedexactlyfor
problemswithlargediscretestate-actionspacesorcontinuousspaces.Instead,com-
pactrepresentationsrelyingonfunctionapproximatorsmustbeused.Thischallenge
was already recognized while the first DP techniques were being developed. How-
ever, it has only been in recent years – and largely in correlation with the advance
of RL – that approximation-based methods have grown in diversity, maturity, and
efficiency,enablingRLandDPtoscaleuptorealisticproblems.
This book provides an accessible in-depth treatment of reinforcement learning
and dynamic programming methods using function approximators. We start with a
concise introduction to classical DP and RL, in order to build the foundation for
the remainder of the book. Next, we present an extensive review of state-of-the-art
approachestoDPandRLwithapproximation.Theoreticalguaranteesareprovided
on the solutions obtained, and numerical examples and comparisons are used to il-
lustrate the properties of the individual methods. The remaining three chapters are
v
vi
dedicatedtoadetailedpresentationofrepresentativealgorithmsfromthethreema-
jor classes of techniques: value iteration, policy iteration, and policy search. The
propertiesandtheperformanceofthesealgorithmsarehighlightedinsimulationand
experimentalstudiesonarangeofcontrolapplications.
We believe that this balanced combination of practical algorithms, theoretical
analysis, and comprehensive examples makes our book suitable not only for re-
searchers,teachers,andgraduatestudentsinthefieldsofoptimalandadaptivecon-
trol, machine learning and artificial intelligence, but also for practitioners seeking
novelstrategiesforsolvingchallengingreal-lifecontrolproblems.
This book can be read in several ways. Readers unfamiliar with the field are
advised to start with Chapter 1 for a gentle introduction, and continue with Chap-
ter 2 (which discusses classical DP and RL) and Chapter 3 (which considers
approximation-based methods). Those who are familiar with the basic concepts of
RLandDPmayconsultthelistofnotationsgivenattheendofthebook,andthen
startdirectlywithChapter3.Thisfirstpartofthebookissufficienttogetanoverview
ofthefield.Thereafter,readerscanpickanycombinationofChapters4to6,depend-
ing on their interests: approximate value iteration (Chapter 4), approximate policy
iterationandonlinelearning(Chapter5),orapproximatepolicysearch(Chapter6).
Supplementary information relevant to this book, including a complete archive
ofthecomputercodeusedintheexperimentalstudies,isavailableattheWebsite:
http://www.dcsc.tudelft.nl/rlbook/
Comments, suggestions, or questions concerning the book orthe Web siteare wel-
come. Interested readers are encouraged to get in touch with the authors using the
contactinformationontheWebsite.
Theauthorshavebeeninspiredovertheyearsbymanyscientistswhoundoubt-
edly left their mark on this book; in particular by Louis Wehenkel, Pierre Geurts,
Guy-Bart Stan, Re´mi Munos, Martin Riedmiller, and Michail Lagoudakis. Pierre
Geurts also provided the computer program for building ensembles of regression
trees, used in several examples in the book. This work would not have been pos-
sible without our colleagues, students, and the excellent professional environments
at the Delft Center for Systems and Control of the Delft University of Technology,
theNetherlands,theMontefioreInstituteoftheUniversityofLie`ge,Belgium,andat
Supe´lecRennes,France.AmongourcolleaguesinDelft,JustinRicedeservesspecial
mentionforcarefullyproofreadingthemanuscript.Toallthesepeopleweextendour
sincerethanks.
WethankSamGeforgivingustheopportunitytopublishourbookwithTaylor
&FrancisCRCPress,andtheeditorialandproductionteamatTaylor&Francisfor
their valuable help. We gratefully acknowledge the financial support of the BSIK-
ICISproject“InteractiveCollaborativeInformationSystems”(grantno.BSIK03024)
and the Dutch funding organizations NWO and STW. Damien Ernst is a Research
Associate of the FRS-FNRS, the financial support of which he acknowledges. We
appreciatethekindpermissionofferedbytheIEEEtoreproducematerialfromour
previousworksoverwhichtheyholdcopyright.
vii
Finally, we thank our families for their continual understanding, patience, and
support.
LucianBus¸oniu
RobertBabusˇka
BartDeSchutter
DamienErnst
November2009
MATLAB(cid:13)R isatrademarkofTheMathWorks,Inc.
Forproductinformation,pleasecontact:
TheMathWorks,Inc.
3AppleHillDrive
Natick,MA01760-2098USA
Tel:[phone]
Fax:[phone]
E-mail:[email protected]
Web:www.mathworks.com
Abouttheauthors
LucianBus¸oniuisapostdoctoralfellowattheDelftCenterforSystemsandControl
of Delft University of Technology, in the Netherlands. He received his PhD degree
(cumlaude)in2009fromtheDelftUniversityofTechnology,andhisMScdegreein
2003fromtheTechnicalUniversityofCluj-Napoca,Romania.Hiscurrentresearch
interests include reinforcement learning and dynamic programming with function
approximation,intelligentandlearningtechniquesforcontrolproblems,andmulti-
agentlearning.
Robert Babusˇka is a full professor at the Delft Center for Systems and Control
of Delft University of Technology in the Netherlands. He received his PhD degree
(cumlaude) in Control in 1997 from the Delft University of Technology, and his
MSc degree (with honors) in Electrical Engineering in 1990 from Czech Technical
University,Prague.Hisresearchinterestsincludefuzzysystemsmodelingandiden-
tification, data-driven construction and adaptation of neuro-fuzzy systems, model-
basedfuzzycontrolandlearningcontrol.Heisactiveinapplyingthesetechniquesin
robotics,mechatronics,andaerospace.
BartDeSchutterisafullprofessorattheDelftCenterforSystemsandControland
at the Marine & Transport Technology department of Delft University of Technol-
ogy in the Netherlands. He received the PhD degree in Applied Sciences (summa
cumlaudewithcongratulationsoftheexaminationjury)in1996fromK.U.Leuven,
Belgium.Hiscurrentresearchinterestsincludemulti-agentsystems,hybridsystems
control,discrete-eventsystems,andcontrolofintelligenttransportationsystems.
Damien Ernst received the MSc and PhD degrees from the University of Lie`ge in
1998 and 2003, respectively. He is currently a Research Associate of the Belgian
FRS-FNRSandheisaffiliatedwiththeSystemsandModelingResearchUnitofthe
UniversityofLie`ge.DamienErnstspenttheperiod2003–2006withtheUniversity
ofLie`geasaPostdoctoralResearcheroftheFRS-FNRSandheldduringthisperiod
positionsasvisitingresearcheratCMU,MITandETH.Hespenttheacademicyear
2006–2007workingatSupe´lec(France)asprofessor.Hismainresearchinterestsare
inthefieldsofpowersystemdynamics,optimalcontrol,reinforcementlearning,and
designofdynamictreatmentregimes.
ix
Contents
1 Introduction 1
1.1 Thedynamicprogrammingandreinforcementlearningproblem . . 2
1.2 Approximationindynamicprogrammingandreinforcementlearning 5
1.3 Aboutthisbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Anintroductiontodynamicprogrammingandreinforcementlearning 11
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Markovdecisionprocesses . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 Deterministicsetting . . . . . . . . . . . . . . . . . . . . . 14
2.2.2 Stochasticsetting . . . . . . . . . . . . . . . . . . . . . . . 19
2.3 Valueiteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1 Model-basedvalueiteration . . . . . . . . . . . . . . . . . 23
2.3.2 Model-freevalueiterationandtheneedforexploration . . . 28
2.4 Policyiteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4.1 Model-basedpolicyiteration . . . . . . . . . . . . . . . . . 31
2.4.2 Model-freepolicyiteration . . . . . . . . . . . . . . . . . . 37
2.5 Policysearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.6 Summaryanddiscussion . . . . . . . . . . . . . . . . . . . . . . . 41
3 Dynamicprogrammingandreinforcementlearninginlargeandcontin-
uousspaces 43
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.2 Theneedforapproximationinlargeandcontinuousspaces . . . . . 47
3.3 Approximationarchitectures . . . . . . . . . . . . . . . . . . . . . 49
3.3.1 Parametricapproximation . . . . . . . . . . . . . . . . . . 49
3.3.2 Nonparametricapproximation . . . . . . . . . . . . . . . . 51
3.3.3 Comparisonofparametricandnonparametricapproximation 53
3.3.4 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4 Approximatevalueiteration . . . . . . . . . . . . . . . . . . . . . 54
3.4.1 Model-basedvalueiterationwithparametricapproximation 55
3.4.2 Model-freevalueiterationwithparametricapproximation . 58
3.4.3 Valueiterationwithnonparametricapproximation. . . . . . 62
3.4.4 Convergenceandtheroleofnonexpansiveapproximation . 63
3.4.5 Example:ApproximateQ-iterationforaDCmotor . . . . . 66
3.5 Approximatepolicyiteration . . . . . . . . . . . . . . . . . . . . . 71
3.5.1 Value iteration-like algorithms for approximate policy
evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
xi
xii
3.5.2 Model-free policy evaluation with linearly parameterized
approximation . . . . . . . . . . . . . . . . . . . . . . . . 74
3.5.3 Policyevaluationwithnonparametricapproximation . . . . 84
3.5.4 Model-basedapproximatepolicyevaluationwithrollouts . . 84
3.5.5 Policyimprovementandapproximatepolicyiteration . . . . 85
3.5.6 Theoreticalguarantees . . . . . . . . . . . . . . . . . . . . 88
3.5.7 Example:Least-squarespolicyiterationforaDCmotor . . 90
3.6 Findingvaluefunctionapproximatorsautomatically . . . . . . . . 95
3.6.1 Basisfunctionoptimization . . . . . . . . . . . . . . . . . 96
3.6.2 Basisfunctionconstruction . . . . . . . . . . . . . . . . . . 98
3.6.3 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
3.7 Approximatepolicysearch . . . . . . . . . . . . . . . . . . . . . . 101
3.7.1 Policygradientandactor-criticalgorithms . . . . . . . . . . 102
3.7.2 Gradient-freepolicysearch . . . . . . . . . . . . . . . . . . 107
3.7.3 Example:Gradient-freepolicysearchforaDCmotor . . . . 109
3.8 Comparisonofapproximatevalueiteration,policyiteration,andpol-
icysearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
3.9 Summaryanddiscussion . . . . . . . . . . . . . . . . . . . . . . . 114
4 Approximatevalueiterationwithafuzzyrepresentation 117
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.2 FuzzyQ-iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.2.1 ApproximationandprojectionmappingsoffuzzyQ-iteration 119
4.2.2 SynchronousandasynchronousfuzzyQ-iteration . . . . . . 123
4.3 AnalysisoffuzzyQ-iteration . . . . . . . . . . . . . . . . . . . . . 127
4.3.1 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.3.2 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.3.3 Computationalcomplexity . . . . . . . . . . . . . . . . . . 140
4.4 Optimizingthemembershipfunctions . . . . . . . . . . . . . . . . 141
4.4.1 Ageneralapproachtomembershipfunctionoptimization . . 141
4.4.2 Cross-entropyoptimization. . . . . . . . . . . . . . . . . . 143
4.4.3 Fuzzy Q-iteration with cross-entropy optimization of the
membershipfunctions . . . . . . . . . . . . . . . . . . . . 144
4.5 Experimentalstudy . . . . . . . . . . . . . . . . . . . . . . . . . . 145
4.5.1 DCmotor:Convergenceandconsistencystudy . . . . . . . 146
4.5.2 Two-link manipulator: Effects of action interpolation, and
comparisonwithfittedQ-iteration . . . . . . . . . . . . . . 152
4.5.3 Invertedpendulum:Real-timecontrol . . . . . . . . . . . . 157
4.5.4 Caronthehill:Effectsofmembershipfunctionoptimization 160
4.6 Summaryanddiscussion . . . . . . . . . . . . . . . . . . . . . . . 164
5 Approximatepolicyiterationforonlinelearningandcontinuous-action
control 167
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5.2 Arecapitulationofleast-squarespolicyiteration . . . . . . . . . . 168