Combined Use of Reinforcement Learning and Simulated Annealing

119 Pages·2003·1.34 MB·English

U M
NIVERSITY OF ISKOLC
D M E
EPARTMENT OF ECHANICAL NGINEERING

Combined Use of Reinforcement
Learning and Simulated Annealing:
Algorithms and Applications

P S
ÉTER TEFÁN

PH.D. THESIS

Supervisors:
LÁSZLÓ DUDÁS, PH.D.
LÁSZLÓ MONOSTORI, D.SC.

JÓZSEF HATVANY PH.D. PROGRAM IN INFORMATION SCIENCE

Program director:
TIBOR TÓTH, D.SC.

Miskolc-Budapest, 2003
Table of contents

Table of contents...............................................................................................................2
List of Acronyms..............................................................................................................5
List of Symbols.................................................................................................................7
Acknowledgements.........................................................................................................10
Chapter 1.........................................................................................................................11
The Scope of the Research..............................................................................................11
Chapter 2.........................................................................................................................14
Fundamentals of Reinforcement Learning......................................................................14
2.1 Machine learning...................................................................................................14
2.2 The reinforcement learning problem.....................................................................15
2.3 Time-horizon models, Markov property, Bellman optimality equation...............17
2.4 Solutions to the RL problem.................................................................................20
2.4.1 Dynamic programming..................................................................................20
2.4.2 Temporal-difference learning........................................................................23
2.5 Q-learning.............................................................................................................27
2.6 Maintenance of state information.........................................................................29
2.7 Policies, exploration vs. exploitation....................................................................30
2.8 Discussion.............................................................................................................31
Chapter 3.........................................................................................................................33
Annealing Schedules Using the Boltzmann Distribution...............................................33
3.1 Introduction...........................................................................................................33
3.2 The convergence property of the Boltzmann formula..........................................35
3.3 The accuracy of the approach...............................................................................35
3.4 The continuous case..............................................................................................37
3.5 Illustration of the temperature bounds theorem....................................................38
3.6 Annealing schedules.............................................................................................38
3.6.1 General annealing method.............................................................................39
3.6.2 Linear annealing.............................................................................................40
3.6.3 Quadratic, inversely quadratic and exponential schedules............................41
3.7 Variable preferences.............................................................................................41
3.8 Computational validation of the annealing model................................................43
3.9 The use of Boltzmann distribution in other fields................................................47
3.10 Discussion...........................................................................................................48
Chapter 4.........................................................................................................................49
Internet Protocol Packet Routing Algorithms.................................................................49
2
4.1 Introduction...........................................................................................................49
4.2 Principles of dynamic routing algorithms.............................................................51
4.2.1 Classification and metrics..............................................................................51
4.2.2 Design goals...................................................................................................52
4.2.3 Distance-vector algorithms............................................................................53
4.2.4 Link-state algorithms .....................................................................................55
4.3 Dynamic routing protocols...................................................................................56
4.3.1 Routing Information Protocol........................................................................56
4.3.2 Interior Gateway Routing Protocol................................................................57
4.3.3 Open Shortest Path First................................................................................57
4.3.4 Border Gateway Protocol...............................................................................58
4.4 Shortest-path computation....................................................................................58
4.4.1 Dijkstra’s algorithm.......................................................................................59
4.4.2 Floyd-Warshall algorithm..............................................................................61
4.5 Reinforcement learning based routing..................................................................62
4.6 Proposed extensions..............................................................................................65
4.7 Implementation, experimental results...................................................................67
4.7.1 General principles..........................................................................................67
4.7.2 Time to live fields..........................................................................................69
4.7.3 Loop detection................................................................................................69
4.7.4 Experimental results.......................................................................................70
4.8 Implementation proposal.......................................................................................71
4.8.1 IP-layer implementation.................................................................................71
4.8.2 UDP implementation.....................................................................................75
4.9 Discussion.............................................................................................................76
Chapter 5.........................................................................................................................77
Flow-shop Scheduling in Virtual Manufacturing Environment .....................................77
5.1 Introduction...........................................................................................................77
5.2 The concept of virtual manufacturing, distributed models...................................78
5.3 Job scheduling.......................................................................................................79
5.4 Flow-shop scheduling...........................................................................................80
5.4.1 The definition of flow-shop scheduling.........................................................80
5.4.2 Johnson’s algorithm.......................................................................................81
5.4.3 Palmer’s method............................................................................................82
5.4.4 Dannenbring’s algorithm...............................................................................83
5.4.5 The quality of the solution.............................................................................83
5.5 The structure of the proposed dynamic scheduler................................................83
5.5.1 General principles..........................................................................................84
5.5.2 Evaluation/reward function............................................................................85
5.5.3 Job sequence setup.........................................................................................87
5.5.4 Update rules...................................................................................................87
5.5.5 Dynamic scheduler.........................................................................................88
3
5.6 Validation of the model.........................................................................................89
5.6.1 Static analysis on different models ................................................................90
5.6.2 Dynamic behavior..........................................................................................92
5.7 Implementation of precedence constrains.............................................................93
5.8 Discussion.............................................................................................................94
Chapter 6.........................................................................................................................95
Summary and Future Work.............................................................................................95
References.......................................................................................................................97
List of Publications .......................................................................................................101
Appendix A...................................................................................................................103
Proofs to Temperature Bounds Theorems....................................................................103
Appendix B...................................................................................................................109
Routing Table Example from a CISCO 12000 Router.................................................109
Appendix C...................................................................................................................110
IP Packet Format Extensions of the Boltzmann-exploration Q-routing Algorithm .....110
Appendix D...................................................................................................................114
The Flow-shop Schedule-evaluation Function.............................................................114
Appendix E...................................................................................................................119
Structure of the CD-ROM.............................................................................................119

4
List of Acronyms

ACS Ant Colony System
AI Artificial Intelligence
AS Autonomous System
BA Boltzmann Annealing
BGP Border Gateway Protocol
BOE Bellman Optimality Equations
CIM Computer Integrated Manufacturing
CPU Central Processing Unit
DM Distributed Manufacturing
DNC Distributed Numerical Control
DP Dynamic Programming
EGRP Exterior Gateway Routing Protocol
FMC Flexible Manufacturing Cell
FMS Flexible Manufacturing System
FWA Floyd-Warshall Algorithm
GA Genetic Algorithm
ICMP Internet Control Message Protocol
IGRP Interior Gateway Routing Protocol
IMS Intelligent Manufacturing System
IP Internet Protocol
IPv4 Internet Protocol version 4
IPv6 Internet Protocol version 6
MDP Markov Decision Process
ML Machine Learning
MS Manufacturing System
NIC Network Interconnection Card
NP Non-Polynomial (time algorithm)
OS Operating System
OSPF Open Shortest-Path First
RIP Routing Information Protocol
RIS Real Information System
RL Reinforcement Learning
RP Routing Protocol
RPS Real Production System
RRR Restricted Rank-based Randomized (policy)
SA Simulated Annealing
5
SMQ “Send me Q”
TCP Transmission Control Protocol
TD Temporal Difference (learning)
TH Time Horizon
TSP Traveling Salesman’s Problem
TTL Time to Live (field)
TU Time Units
UDP User Datagram Protocol
VE Virtual Enterprise
VIS Virtual Information System
VMS Virtual Manufacturing System
VPS Virtual Production System
6
List of Symbols

[a,b],[c,d] real intervals
(cid:209) partial derivatives of a quantity according to the weight vector elements
w
A ,B ,C machining times on machines A, B, and C respectively
i i i
a jth action in the ith state of the agent
i
j
a learning rate for “mean value type” action-state values
b vector of boundary conditions in flow-shop scheduling
b learning rate for “standard deviation type” action-state values
b “inverse temperature” (Chapter 3)
C set of constraints (Chapter 5)
C number of precedence constraints
c,c constant parameters (Chapter 3)
i
c precedence constraints (Chapter 5)
i
c vector of parameters (Chapter 3)
c cost of delivery between nodes i and j (Chapter 4)
ij
g discount factor
D compound matrix
d distance label on node i
i
d distance label on node i (matrix representation)
ij
dim(a) dimension of vector a
d temporal difference factor
E {x} expected value of x following policy p
p
E “potential energy”
i
e (s) eligibility value of state s
t
e the natural number » 2.71...
e base vector in dimension i (Chapter 5)
i
e small positive error
f (x) transfer function
f '(x), f"(x) reward function
g(x) reward function
g auxiliary machining-delay vector (Chapter 5)
h'(x),h"(x) reward function
7
I Palmer index for job j
j
h visibility of a trail from node i to node j
ij
j the jth job under processing
j
k the number of actions sharing the same, maximal action-state value
k minimum of the difference between each action-state value pair
L(A) link state description of node A
l decay rate for eligibility traces
M matrix of machining times
m(i) number of actions in the ith state (Chapter 2)
n number of states
n small positive redundancy variable
o ,o abstract machining times
1 2
Pa probability of getting into state s' from state s via action a
ss'
p probability of selecting action a
i i
pˆ scaled probability value
p(x) continuous probability distribution function
p,r permutation vectors (Chapter 5)
pred predecessor node
i
pred predecessor node (matrix representation)
ij
p(s ,a ) policy of the agent
i i
j
p* optimal policy
p' non-optimal policy
Q,Q action-state values
i
Q matrix of action-state values
Q* optimal action-state value
Qˆ scaled action-state value
Q the largest action-state value
max
Q the smallest action-state value
min
Q(x) continuous function of the action-state value
q vector of action-state values
R total reinforcement on time horizon T
T
Ra reward of getting into state s' from state s via action a
ss'
r reinforcement in time step t
t
s the ith state of an agent
i
x integration auxiliary variable
8
s standard deviation
T time horizon (Chapter 2)
T temperature (Chapter 3)
T total machining time (Chapter 5)
Tˆ scaled temperature
T ,Tmax the largest temperature
max i
T ,T min the smallest temperature
min i
T(t) time-temperature function (Chapter 3)
T(A) routing table of node A (Chapter 4)
t time step
t machining time of job j on machine i (Chapter 5)
ij
t starting time of the annealing process
start
t ending time of the annealing process
end
t vector of temperature values
tmax vector of maximal temperature values
tmin vector of minimal temperature values
t strength of a trail from node i to node j
ij
Q fix-point operator applied to action-state values
V ,V state values
i
V* optimal state values
D w vector of weight components’ changes
X ,Y off-machining times on machines B and C
i i

9
Acknowledgements

The dissertation summarizes the results of my four years research in the field of
machine learning. During this time I learnt many-many new concepts which governed
my work, and formed the way how I look to the world and how I approach to problems
and to the solutions. Here I would like to express my gratitude to many-many people.
First of all I would like to render my acknowledgements to my supervisors
László Dudás and László Monostori who introduced the field of artificial intelligence
and machine learning to me, and who gave me continuous scientific support during the
whole of my research. They taught me how to be able to see the same thing from the
engineer’s as well as from the scientist’s point of view. Many thanks to both of them!
I’m also grateful to András Márkus, József Váncza for their inspiration, and for
the interesting discussions conducted not only in the field of AI, but other interesting
and important topics as well. Special thanks to Anikó Ekárt who helped me to formulate
my mathematical results.
I’m indebted to my former and present colleagues Elizabeta Zudor, Botond
Kádár and Zsolt János Viharos, Gábor Ivánszky who could always give me new ideas
whenever I got into a deadlock. I would also like to thank Tibor Tóth, Ferenc Erdélyi
and Dénes Vadász for revealing me the broadness of the information science. I’m also
indebted to the community of the Department of Information Science at the University
of Miskolc.
Last but not least, I’m especially grateful to my family, Zsuzsa and my parents,
Mária Stefán and György Stefán for their patience in the “hard days” and for the moral
support.

10