Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles

306 Pages·2013·20.634 MB·English

by  Draguna Vrabie, Kyriakos G. Vamvoudakis, Frank. L Lewis

Control Engineering Series 81
RaO
Optimal Adaptive Control and
enp Optimal Adaptive
indt
Differential Games by Reinforcement f Dim
o
Learning Principles rciffal Control and
ee
A
mr
ed
ena
ntp
i
tat Differential Games
This book gives an exposition of recently developed approximate dynamic programming Draguna Vrabie is a Senior Ll iv
(ADP) techniques for decision and control in human engineered systems. ADP is a Research Scientist at United eGe
rbeioinlofogricceaml aenndt amnaimchainl esy lseteamrnisn. gIt tiesc chonniqnueec ttehda tf rios mm oa ttivhaetoedre tbicya lle paorninint go fm veiecwh awniitshm bso itnh TEeacsht nHoalrotgfoierds , RCeosnenaercchti cCuet.n ter, aram C
adaptive control and optimal control methods. The book shows how ADP can be used to Kyriakos G. Vamvoudakis n o by Reinforcement
design a family of adaptive optimal control algorithms that converge in real-time to optimal is a Faculty Project Research inen
Scientist at the Center for st
clocitoofe nnaratturrtootuollr messo yaasltudtiectai mopcnotsivsn, etbbr ouyclt o msndytoesro atnelslmoeutrr ssiyn. iagTenr ladddda oitotapipo tatniimlmaolan aalgl dp ctaeohprneftot irvsroeymls lectaeornsmnc ate rrto.er aOl lejtenwrcs ott ohl edreaiie srosntit.n hoGcenter lmnihneaeern tahidnllo,y ,rdt erisana d lf otithtirimo etn hecaeu lh rdoroeepwnst iitmg tona l CaDCnooednmp ttCrp oooulf,mt eEDrpl eyEuncnttaargmitc.i,oa iUcnl a na(l-niCSvdeyC srDsteiCtmy) ,os ,f g Pri by rol Learning Principles
n
controllers must be designed offline using full knowledge of the systems dynamics. California, Santa Barbara.
c
It is also shown how to use ADP methods to solve multi-player differential games online. Frank L. Lewis is the i
Moncrief-O'Donnell Endowed p
Differential games have been shown to be important in H-infinity robust control for
disturbance rejection, and in coordinating activities among multiple agents in networked Chair at the Automation & le
Robotics Research Institute,
teams. The focus of this book is on continuous-time systems, whose dynamical models can University of Texas at Arlington. s
be derived directly from physical principles based on Hamiltonian or Lagrangian dynamics.
aVV
nd Lewamvourabie, Draguna Vrabie, Kyriakoasn dG .F Vraanmkv Lo.u Ldeawkiiss
The Institution of Engineering and Technology isdak
www.theiet.org is
978-1-84919-489-1
Optimal Adaptive Control.indd 1 23/10/2012 16:47:04
IET CONTROL ENGINEERING SERIES 81
Optimal Adaptive
Control and
Differential Games
by Reinforcement
Learning Principles
Other volumes in this series:
Volume 8 A history of control engineering, 1800–1930 S. Bennett
Volume 18 Applied control theory, 2nd edition J.R. Leigh
Volume 20 Design of modern control systems D.J. Bell, P.A. Cook and N. Munro (Editors)
Volume 28 Robots and automated manufacture J. Billingsley (Editor)
Volume 33 Temperature measurement and control J.R. Leigh
Volume 34 Singular perturbation methodology in control systems D.S. Naidu
Volume 35 Implementation of self-tuning controllers K. Warwick (Editor)
Volume 37 Industrial digital control systems, 2nd edition K. Warwick and D. Rees (Editors)
Volume 39 Continuous time controller design R. Balasubramanian
Volume 40 Deterministic control of uncertain systems A.S.I. Zinober (Editor)
Volume 41 Computer control of real-time processes S. Bennett and G.S. Virk (Editors)
Volume 42 Digital signal processing: principles, devices and applications N.B. Jones and
J.D.McK. Watson (Editors)
Volume 44 Knowledge-based systems for industrial control J. McGhee, M.J. Grimble and
A. Mowforth (Editors)
Volume 47 A history of control engineering, 1930–1956 S. Bennett
Volume 49 Polynomial methods in optimal control and fi ltering K.J. Hunt (Editor)
Volume 50 Programming industrial control systems using IEC 1131-3 R.W. Lewis
Volume 51 Advanced robotics and intelligent machines J.O. Gray and D.G. Caldwell (Editors)
Volume 52 Adaptive prediction and predictive control P.P. Kanjilal
Volume 53 Neural network applications in control G.W. Irwin, K. Warwick and K.J. Hunt (Editors)
Volume 54 Control engineering solutions: a practical approach P. Albertos, R. Strietzel and
N. Mort (Editors)
Volume 55 Genetic algorithms in engineering systems A.M.S. Zalzala and P.J. Fleming (Editors)
Volume 56 Symbolic methods in control system analysis and design N. Munro (Editor)
Volume 57 Flight control systems R.W. Pratt (Editor)
Volume 58 Power-plant control and instrumentation D. Lindsley
Volume 59 Modelling control systems using IEC 61499 R. Lewis
Volume 60 People in control: human factors in control room design J. Noyes and M. Bransby
(Editors)
Volume 61 Nonlinear predictive control: theory and practice B. Kouvaritakis and M. Cannon
(Editors)
Volume 62 Active sound and vibration control M.O. Tokhi and S.M. Veres
Volume 63 Stepping motors: a guide to theory and practice, 4th edition P.P. Acarnley
Volume 64 Control theory, 2nd edition J.R. Leigh
Volume 65 Modelling and parameter estimation of dynamic systems J.R. Raol, G. Girija and J. Singh
Volume 66 Variable structure systems: from principles to implementation A. Sabanovic,
L. Fridman and S. Spurgeon (Editors)
Volume 67 Motion vision: design of compact motion sensing solution for autonomous
systems J. Kolodko and L. Vlacic
Volume 68 Flexible robot manipulators: modelling, simulation and control M.O. Tokhi and
A.K.M. Azad (Editors)
Volume 69 Advances in unmanned marine vehicles G. Roberts and R. Sutton (Editors)
Volume 70 Intelligent control systems using computational intelligence techniques A. Ruano
(Editor)
Volume 71 Advances in cognitive systems S. Nefti and J. Gray (Editors)
Volume 72 Control theory: a guided tour, 3rd edition James Ron Leigh
Volume 73 Adaptive sampling with mobile WSN K. Sreenath, M.F. Mysorewala, D.O. Popa and
F.L. Lewis
Volume 74 Eigenstructure control algorithms: applications to aircraft/rotorcraft handling
qualities design S. Srinathkumar
Volume 75 Advanced control for constrained processes and systems F. Garelli, R.J. Mantz and
H. De Battista
Volume 76 Developments in control theory towards glocal control L. Qiu, J. Chen, T. Iwasaki and
H. Fujioka (Editors)
Volume 77 Further advances in unmanned marine vehicles G.N. Roberts and R. Sutton (Editors)
Volume 78 Frequency-domain control design for high-performance systems J. O’Brien
Optimal Adaptive
Control and
Differential Games
by Reinforcement
Learning Principles
Draguna Vrabie, Kyriakos G. Vamvoudakis
and Frank L. Lewis
The Institution of Engineering and Technology
Published by The Institution of Engineering and Technology, London, United Kingdom
The Institution of Engineering and Technology is registered as a Charity in
England & Wales (no. 211014) and Scotland (no. SC038698).
© 2013 The Institution of Engineering and Technology
First published 2013
This publication is copyright under the Berne Convention and the Universal Copyright
Convention. All rights reserved. Apart from any fair dealing for the purposes of research
or private study, or criticism or review, as permitted under the Copyright, Designs and
Patents Act 1988, this publication may be reproduced, stored or transmitted, in any
form or by any means, only with the prior permission in writing of the publishers, or in
the case of reprographic reproduction in accordance with the terms of licences issued
by the Copyright Licensing Agency. Enquiries concerning reproduction outside those
terms should be sent to the publisher at the undermentioned address:
The Institution of Engineering and Technology
Michael Faraday House
Six Hills Way, Stevenage
Herts, SG1 2AY, United Kingdom
www.theiet.org
While the authors and publisher believe that the information and guidance given
in this work are correct, all parties must rely upon their own skill and judgement when
making use of them. Neither the authors nor the publisher assumes any liability to
anyone for any loss or damage caused by any error or omission in the work, whether
such an error or omission is the result of negligence or any other cause. Any and all
such liability is disclaimed.
The moral rights of the authors to be identifi ed as authors of this work have been
asserted by them in accordance with the Copyright, Designs and Patents Act 1988.
British Library Cataloguing in Publication Data
A catalogue record for this product is available from the British Library
ISBN 978-1-84919-489-1 (hardback)
ISBN 978-1-84919-490-7 (PDF)
Typeset in India by MPS Limited
Printed in the UK by CPI Group (UK) Ltd, Croydon, CR0 4YY
To Adrian Miron
Draguna Vrabie
To my parents, George and Evgenia. Your love and support was and
will continue to be my biggest inspiration and source of strength
Kyriakos Vamvoudakis
To Galina, Roma, Chris – who have made every day exciting
Frank Lewis

Contents
Preface xii
Acknowledgements xv
1 Introduction to optimal control, adaptive control and
reinforcement learning 1
1.1 Optimal control 2
1.1.1 Linear quadratic regulator 2
1.1.2 Linear quadratic zero-sum games 3
1.2 Adaptive control 4
1.3 Reinforcement learning 7
1.4 Optimal adaptive control 8
2 Reinforcement learning and optimal control of discrete-time
systems: Using natural decision methods to design optimal
adaptive controllers 9
2.1 Markov decision processes 11
2.1.1 Optimal sequential decision problems 12
2.1.2 A backward recursion for the value 14
2.1.3 Dynamic programming 15
2.1.4 Bellman equation and Bellman optimality equation 15
2.2 Policy evaluation and policy improvement 19
2.2.1 Policy iteration 21
2.2.2 Iterative policy iteration 21
2.2.3 Value iteration 22
2.2.4 Generalized policy iteration 25
2.2.5 Q function 26
2.3 Methods for implementing policy iteration and value iteration 29
2.4 Temporal difference learning 30
2.5 Optimal adaptive control for discrete-time systems 32
2.5.1 Policy iteration and value iteration for discrete-time
dynamical systems 34
2.5.2 Value function approximation 35
2.5.3 Optimal adaptive control algorithms for
discrete-time systems 36
2.5.4 Introduction of a second ‘Actor’ neural network 38
2.5.5 Online solution of Lyapunov and Riccati equations 42
viii Optimal adaptive control and differential games by RL principles
2.5.6 Actor–critic implementation of discrete-time
optimal adaptive control 43
2.5.7 Q learning for optimal adaptive control 43
2.6 Reinforcement learning for continuous-time
systems 46
PART I Optimal adaptive control using reinforcement learning
structures 49
3 Optimal adaptive control using integral reinforcement
learning for linear systems 51
3.1 Continuous-time adaptive critic solution for the
linear quadratic regulator 53
3.1.1 Policy iteration algorithm using integral
reinforcement 54
3.1.2 Proof of convergence 55
3.2 Online implementation of IRL adaptive optimal control 58
3.2.1 Adaptive online implementation of IRL
algorithm 58
3.2.2 Structure of the adaptive IRL algorithm 61
3.3 Online IRL load-frequency controller design for a power
system 64
3.4 Conclusion 69
4 Integral reinforcement learning (IRL) for non-linear
continuous-time systems 71
4.1 Non-linear continuous-time optimal control 72
4.2 Integral reinforcement learning policy iterations 74
4.2.1 Integral reinforcement learning policy iteration
algorithm 76
4.2.2 Convergence of IRL policy iteration 78
4.3 Implementation of IRL policy iterations using value function
approximation 79
4.3.1 Value function approximation and temporal
difference error 79
4.3.2 Convergence of approximate value function to
solution of the Bellman equation 81
4.3.3 Convergence of approximate IRL policy iteration
to solution of the HJB equation 85
4.4 Online IRL actor–critic algorithm for optimal adaptive
control 85
4.4.1 Actor–critic structure for online implementation
of adaptive optimal control algorithm 85
4.4.2 Relation of adaptive IRL control structure to learning
mechanisms in the mammal brain 88
Contents ix
4.5 Simulation results 89
4.5.1 Non-linear system example 1 89
4.5.2 Non-linear system example 2 90
4.6 Conclusion 92
5 Generalized policy iteration for continuous-time systems 93
5.1 Policy iteration algorithm for optimal control 94
5.1.1 Policy iteration for continuous-time systems 94
5.1.2 Integral reinforcement learning for continuous-time
systems 95
5.2 Generalized policy iteration for continuous-time systems 96
5.2.1 Preliminaries: Mathematical operators for
policy iteration 97
5.2.2 Contraction maps for policy iteration 98
5.2.3 A new formulation of continuous-time policy
iteration: Generalized policy iteration 100
5.2.4 Continuous-time generalized policy iteration 101
5.3 Implementation of generalized policy iteration algorithm 103
5.4 Simulation results 104
5.4.1 Example 1: Linear system 104
5.4.2 Example 2: Non-linear system 105
5.5 Conclusion 107
6 Value iteration for continuous-time systems 109
6.1 Continuous-time heuristic dynamic programming for the
LQR problem 110
6.1.1 Continuous-time HDP formulation using integral
reinforcement learning 111
6.1.2 Online tuning value iteration algorithm for partially
unknown systems 113
6.2 Mathematical formulation of the HDP algorithm 114
6.3 Simulation results for online CT-HDP design 117
6.3.1 System model and motivation 117
6.3.2 Simulation setup and results 118
6.3.3 Comments on the convergence of CT-HDP
algorithm 120
6.4 Conclusion 122
PART II Adaptive control structures based on reinforcement
learning 123
7 Optimal adaptive control using synchronous online learning 125
7.1 Optimal control and policy iteration 127
7.2 Value function approximation and critic neural network 129
7.3 Tuning and convergence of critic NN 132