Optimization of Stochastic Discrete Systems and Control on Complex Networks: Computational Networks

420 Pages·2015·7.195 MB·English

by  Dmitrii Lozovanu, Stefan Pickl (auth.)

Advances in Computational Management Science 12
Dmitrii Lozovanu
Stefan Pickl
Optimization of
Stochastic Discrete
Systems and
Control on Complex
Networks
Computational Networks
Advances in Computational Management
Science
Volume 12
Series editors
Hans M. Amman, Eindhoven, The Netherlands
Berc Rustem, London, UK
More information about this series at http://www.springer.com/series/5594
Dmitrii Lozovanu Stefan Pickl
(cid:129)
Optimization of Stochastic
Discrete Systems
and Control on
Complex Networks
Computational Networks
123
Dmitrii Lozovanu Stefan Pickl
Instituteof Mathematics andInformatics Institutfür Theoretische Informatik
Academy ofSciences ofMoldova Mathematik undOperationsResearch
Chisinau UniversitätderBundeswehr München
Moldova Munchen
Germany
ISSN 1388-4301
ISBN 978-3-319-11832-1 ISBN 978-3-319-11833-8 (eBook)
DOI 10.1007/978-3-319-11833-8
LibraryofCongressControlNumber:2014951665
SpringerChamHeidelbergNewYorkDordrechtLondon
©SpringerInternationalPublishingSwitzerland2015
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor
informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodology now known or hereafter developed. Exempted from this legal reservation are brief
excerpts in connection with reviews or scholarly analysis or material supplied specifically for the
purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe
work. Duplication of this publication or parts thereof is permitted only under the provisions of
theCopyrightLawofthePublisher’slocation,initscurrentversion,andpermissionforusemustalways
beobtainedfromSpringer.PermissionsforusemaybeobtainedthroughRightsLinkattheCopyright
ClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Complex Networks are the heart of Operations
ResearchandOperationsResearchisinthecore
ofComplexNetworks.
PanosPardalos
Thisbookcomprisestherecentresultsonstudyingsto-
chasticdynamicprogrammingmodelsandsolvingopti-
malcontrolproblemsonnetworks.Themainattentionis
addressed to stochastic versions of classical discrete
control problems, Markov decision processes, and a
game-theoretical approach to these stochastic dynamic
models. The authors included in the book their own
results concerning the determination of the optimal
solution of discrete optimal control problems on net-
worksandsolvingthegamevariantsofMarkovdecision
problemsinthesenseofcomputationalnetworks.
Investigations on elaborations of efficient methods and algorithms for modern
stochasticdynamicprogrammingproblemsrepresentarelevanttopicinOperations
Research.Withrespecttocomputationalnetworks,stochasticdynamicoptimization
modelsarewidelyusedforstudyingandsolvingmanypracticaldecisionproblems
from diverse areas such as ecology, economics, engineering, communications
systems, etc. Therefore, in the book an attempt is made to systematize the most
important existing methods for the mentioned class of problems as well as to
describe new algorithms for solving different classes of the stochastic dynamic
programming problems.
The book consists offour chapters.
The first chapter studies the finite state space of Markov processes and gives a
reviewofexistingmethodsandalgorithmsfordeterminingthemaincharacteristicsin
Markovchains.Newapproachesbasedondynamicprogrammingandcombinatorial
methods for determining the state-time probabilities, the matrix of limiting proba-
bilities, and differential matrices in a Markov chain are proposed and formulated.
Based on a classical numerical methods in this chapter new polynomial-time
algorithms for determining the limiting probabilities and differential matrices are
described.
v
vi Preface
Along with well-known numerical methods for determining the basic charac-
teristics in finite Markov processes, new calculation procedures and algorithms for
their finding in the case of stationary and nonstationary discrete processes are
presented.
Asymptotic behavior of the average and expected total rewards (costs) in
Markov processes with rewards (costs) are analyzed and the corresponding
asymptotic formulae are derived. Computational complexity aspects of the prob-
lems of determining the average and the discounted expected total rewards (costs)
are analyzed and thecorresponding procedures forcalculating these characteristics
are described.
Chapter 2 is dedicated to infinite horizon stochastic discrete optimal control
modelsandMarkovdecisionproblemswith average andexpectedtotal discounted
optimization criteria. Necessary and sufficient conditions for determining the
optimal stationary strategies in such problems are formulated and algorithms for
determining their solutions are developed. The stochastic control problems are
formulated on networks and the corresponding algorithms for determining their
optimalsolutionsusingalinearprogrammingapproachareproposed.Furthermore,
a relationship between the class of stationary Markov decision problems and the
classofstochasticcontrolproblemsonnetworksisanalyzedandprocedureshowto
reduceoneclassofproblemstoanotheraredescribed.Themostimportantresultsof
Chap. 2 are related to a linear programming approach for unichain and multichain
Markov decision problems with average and expected total discounted costs opti-
mizationcriteria. Based onsuchanapproach new algorithms forsolving problems
in general form are derived. Afterwards, efficient iterative procedures for deter-
mining the optimal solutions of Markov decision problems and stochastic control
problemsonnetworksaredescribed.Additionally,anewclassofMarkovdecision
problems and stochastic control problems with stopping states are introduced and
thecorrespondingmathematicaltoolforsolvingthisclassofproblemsisdeveloped.
Some extensions and generalizations of the stochastic decision models are con-
sidered and their possible applications for solving classical optimization problems
are suggested.
In Chap. 3 a special game-theoretical approach to Markov decision processes
and stochastic discrete optimal control problems is developed. An essentially new
class of stochastic positional games is formulated and studied applying the game-
theoretical concept to Markov decision problems with average and expected total
discountedcostsoptimizationcriteria.Toformulatethisclassofgamesweassume
that Markov decision processes may be controlled by several actors (players).
The set of states of the system in such processes is divided into several disjoint
subsetswhichrepresentthecorrespondingpositionsetsoftheplayers.Eachplayer
has to determine which action should be taken in each state of his position set in
ordertominimize(ormaximize)hisownaveragecostpertransitionortheexpected
total discounted cost. For the stochastic discrete optimal control problems with
infinitetimehorizonthisapproachisdevelopedinasimilarwayandanewclassof
stochastic positional games on networks is obtained.
Preface vii
ThemainresultswedescribeinthischapterarerelatedtotheexistenceofNash
equilibriaforaconsideredclassofstochasticpositionalgamesandanelaborationof
the algorithms for determining the optimal stationary strategies of the players. We
formulate and prove Nash equilibria conditions for the stochastic positional games
withaverageanddiscountedpayofffunctionsoftheplayersanddevelopalgorithms
for determining the optimal strategies for different classes of games. These results
are specified for antagonistic stochastic positional games and algorithms for
determining the optimal strategies of the players are gained. In the following, we
show that the obtained results generalize the well-known results for deterministic
positionalgamesandnewconditionsfordeterminingthesolutionsoftheproblems
can be derived. Moreover, we show that the considered class of stochastic posi-
tional games canbeusedforstudyingcyclicgames andShapleystochasticgames.
New polynomial-time algorithms for deterministic antagonistic positional games
are described. The algorithms for determining the optimal strategies of the players
indeterministiccasesaredevelopedforamoregeneralclassofpositionalgameson
networks. Additionally, the multi-criteria decision problems with Pareto and
Stackelberg optimization principles are formulated and some approaches for
determining the solutions of these problems are suggested.
Chapter 4 is devoted to finite horizon stochastic control problems and Markov
decision processes. In this chapter dynamic programming algorithms for deter-
mining the optimal solutions of the problems using the network representation of
the finite discrete processes are developed. We show that the solutions for the
considered stochastic horizon decision problems can be efficiently found using a
backward dynamic induction principle. The algorithms are described in general
form for stationary and nonstationary cases of the problems. Moreover, the algo-
rithms are developed for the case with varying time of states’ transitions of the
dynamical systems. These algorithms are a contribution to the important field of
computational network theory.
January 2015 Dmitrii Lozovanu
Stefan Pickl
Contents
1 Discrete Stochastic Processes, Numerical Methods
for Markov Chains and Polynomial Time Algorithms. . . . . . . . . . 1
1.1 The Basic Definitions and Some Preliminary Results. . . . . . . 2
1.1.1 Discrete Markov Processes and Determining
the State-Time Probabilities of the System. . . . . . . . 2
1.1.2 Definition of the Limit Matrix and the Classification
of the States in Markov Chains. . . . . . . . . . . . . . . . 6
1.1.3 A Decomposition Algorithm for Determining
the Limit Matrix ... . . . . . . . . . . . . . . . . . . . . . . . 8
1.1.4 The z-Transform and the Asymptotic Behavior
of State-Time Probabilities. . . . . . . . . . . . . . . . . . . 11
1.2 An Algorithm for Determining the Limit Matrix Based
on the z-Transform and Classical Numerical Methods . . . . . . 15
1.2.1 The Main Approach and the General Scheme
of the Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2.2 The Calculation of the Coefficients
of the Characteristic Polynomial . . . . . . . . . . . . . . . 19
1.2.3 Determining the z-Transform Function. . . . . . . . . . . 21
1.2.4 An Algorithm for Calculating the Limit Matrix. . . . . 22
1.3 An Algorithm for Determining the Differential Matrices
in Markov Chains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.3.1 Determining the Differential Matrices
Based on the z-Transform . . . . . . . . . . . . . . . . . . . 28
1.3.2 Linear Recurrent Equations and its Main
Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.3.3 The Main Results and an Algorithm . . . . . . . . . . . . 30
1.3.4 Comments on the Computational
Complexity of the Algorithm . . . . . . . . . . . . . . . . . 32
ix
x Contents
1.4 An Algorithm for Determining the Limit and the
Differential Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
1.4.1 Some Auxiliary Results Concerning the
Representation of the z-Transform. . . . . . . . . . . . . . 33
1.4.2 Expansion of the z-Transform with Respect
to Nonzero Characteristic Values . . . . . . . . . . . . . . 34
1.4.3 The Main Conclusion and the Description
of the Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 39
1.4.4 Numerical Examples . . . . . . . . . . . . . . . . . . . . . . . 41
1.5 A Dynamic Programming Approach for Discrete
Markov Processes and Combinatorial Algorithms
for Determining the Limit Matrix . . . . . . . . . . . . . . . . . . . . 51
1.5.1 Calculation of the State-Time Probability
of the System with the Restriction
on the Number of Transitions . . . . . . . . . . . . . . . . 52
1.5.2 Determining the Limiting State Probabilities
in Markov Chains Based on Dynamic
Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
1.5.3 An Algorithm for the Calculation
of the Limit Matrix in Markov Chains
with Running Time Oðn3Þ . . . . . . . . . . . . . . . . . . . 66
1.5.4 An Algorithm for Determining the Limit
Probabilities Based on the Ergodicity
Condition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
1.5.5 Determining the First Hitting Limiting
Probability of the State in a Markov Chain. . . . . . . . 77
1.6 Determining the State-Time Probabilities of the
System for Non-stationary Markov Processes . . . . . . . . . . . . 79
1.7 Markov Processes with Rewards and Transition Costs. . . . . . 84
1.7.1 Definition and Calculation of the Expected
Total Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
1.7.2 An Asymptotic Behavior Analysis
of the Expected Total Cost Based
on the z-Transform . . . . . . . . . . . . . . . . . . . . . . . . 87
1.7.3 Determining the Expected Total Cost
for Non-stationary Markov Processes. . . . . . . . . . . . 91
1.7.4 Definition and Calculation of the Variance
of the Total Cost. . . . . . . . . . . . . . . . . . . . . . . . . . 92
1.8 Markov Processes with Discounted Costs. . . . . . . . . . . . . . . 93
1.9 Semi-Markov Processes with Transition Costs . . . . . . . . . . . 96
1.10 Determining the Expected Total Cost for Markov
Processes with Stopping States. . . . . . . . . . . . . . . . . . . . . . 99