Intelligent Controller Selection for Aggressive Quadrotor Manoeuvring

88 Pages·2016·2.3 MB·English

g
n
i
r
e
e
n
i
g
n
E
e
c
a
p
s
o
r
e
A
f
o
y
t
l
u
c
a
F
Intelligent Controller Selection for
Aggressive Quadrotor Manoeuvring
A reinforcement learning approach
D. Molenkamp
April 21, 2016

Intelligent Controller Selection for
Aggressive Quadrotor Manoeuvring
A reinforcement learning approach
Master of Science Thesis
For obtaining the degree of Master of Science in Aerospace Engineering
at Delft University of Technology
D. Molenkamp
April 21, 2016
Faculty of Aerospace Engineering Delft University of Technology
·
Delft University of Technology
Copyright c D. Molenkamp
(cid:13)
All rights reserved.
Delft University Of Technology
Department Of
Control and Simulation
The undersigned hereby certify that they have read and recommend to the Faculty of
Aerospace Engineering for acceptance a thesis entitled “Intelligent Controller Selection
for Aggressive Quadrotor Manoeuvring” by D. Molenkamp in partial fulfillment of
the requirements for the degree of Master of Science.
Dated: April 21, 2016
Readers:
Dr. Q. P. Chu
Dr. ir. E. van Kampen
Dr. ir. C. C. de Visser
Ir. R. Noomen

Preface
This thesis is the final product of my research performed at the Control and Simulation Divi-
sion of Aerospace Engineering at Delft University of Technology, and is in partial fulfilment
of the Master of Science in Aerospace Engineering degree.
The subject of this thesis is intelligent controller selection for aggressive quadrotor manoeu-
vring and the main contribution is the paper that can be found in Part I. This paper is a
stand-alone document that contains the most important findings of my graduation project.
A big part of research into Fault Detection and Isolation (FDI) is eventually not included
in this thesis report, since the research direction had shifted away from this field after the
literature study.
I would like to express my gratitude to my daily supervisor Erik-Jan van Kampen for his
continued support and feedback throughout my research. My thanks also go to my fellow
students in the graduation room for their advice, discussions during group lunch and all the
coffeebreaks. FinallyIwouldliketothankmyfamilyandfriendsforgivingmethemotivation
that kept me going.
Djim Molenkamp
April 21, 2016
Delft
vi
D.Molenkamp IntelligentControllerSelectionforAggressiveQuadrotorManoeuvring
Contents
1 Introduction 1
1-1 Research question, aims and objectives . . . . . . . . . . . . . . . . . . . . . . . 3
1-2 Outline of this report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
I Paper 5
II Additional Results and Simulation 23
2 Additional Results 25
2-1 Policy evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2-2 Performance Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3 Simulation Model 35
3-1 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3-1-1 Controller block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3-1-2 Quadrotor Model block . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3-1-3 Sensor Model block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3-2 Hardware: AR.Drone 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
III Literature Study 47
4 Reconfigurable Control Techniques 49
4-1 Passive Fault Tolerant Control . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4-1-1 Robust Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4-1-2 Reliable Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4-2 On-line Automatic Redesign. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
IntelligentControllerSelectionforAggressiveQuadrotorManoeuvring D.Molenkamp
viii Contents
4-2-1 Model Predictive Control (MPC) . . . . . . . . . . . . . . . . . . . . . . 50
4-2-2 Dynamic Inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4-2-3 Model Reference Adaptive Control (MRAC) . . . . . . . . . . . . . . . . 52
4-2-4 Model Following (MF) . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4-2-5 Pseudo-Inverse (PI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4-2-6 Linear Quadratic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4-3 Pre-computed Control Laws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4-3-1 Generalized Internal Model Control . . . . . . . . . . . . . . . . . . . . . 58
4-3-2 Gain Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4-3-3 Multiple Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4-4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5 Reinforcement Learning 63
5-1 Agent-Environment Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5-2 Markov Decision Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5-2-1 Continuous state MDP . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5-3 Policy, Reward and Value Function . . . . . . . . . . . . . . . . . . . . . . . . . 65
5-4 Solving a MDP problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5-4-1 Value Iteration method . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5-4-2 Policy Iteration method . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5-5 Reinforcement Learning Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5-5-1 Temporal Difference Method . . . . . . . . . . . . . . . . . . . . . . . . 68
5-5-2 SARSA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5-5-3 Q-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5-5-4 Eligibility Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5-5-5 The Exploration/Exploitation Trade-off . . . . . . . . . . . . . . . . . . 69
5-6 Example: Windy Grid World . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5-7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Bibliography 73
D.Molenkamp IntelligentControllerSelectionforAggressiveQuadrotorManoeuvring