Markov Processes for Stochastic Modeling

505 Pages·2008·2.11 MB·English

by  Oliver Ibe

Markov Processes
for Stochastic Modeling
ELSEVIER
science &
technology books
Companion Web Site:
http://www.elsevierdirect.com/companions/9780123744517
Markov Processes for Stochastic Modeling,
by Oliver C. Ibe
Resources for Professors:
• Links to web sites carefully chosen to supplement the content of the textbook
• Also available with purchase of Markov Processes for Stochastic Modeling, password protected
and activated upon registration, online Instructors Solutions Manual
• Online Student Solutions Manual is now available through separate purchase.
TOOLS ALL YOUR TEACHING NEEDS
FOR
textbooks.elsevier.com
ACADEMIC
PRESS
To adopt this book for course use, visit http://textbooks.elsevier.com
Markov Processes
for Stochastic Modeling
Oliver C. Ibe
UniversityofMassachusetts
Lowell,Massachusetts
AMSTERDAM•BOSTON•HEIDELBERG•LONDON
NEWYORK•OXFORD•PARIS•SANDIEGO
SANFRANCISCO•SINGAPORE•SYDNEY•TOKYO
AcademicPressisanimprintofElsevier
ElsevierAcademicPress
30CorporateDrive,Suite400,Burlington,MA01803,USA
525BStreet,Suite1900,SanDiego,California92101-4495,USA
84Theobald’sRoad,LondonWC1X8RR,UK
(cid:3)
Thisbookisprintedonacid-freepaper. ∞
Copyright©2009,ElsevierInc.Allrightsreserved.
Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,
electronicormechanical,includingphotocopy,recording,oranyinformationstorageand
retrievalsystem,withoutpermissioninwritingfromthepublisher.
PermissionsmaybesoughtdirectlyfromElsevier’sScience&TechnologyRights
DepartmentinOxford,UK:phone:[phone],fax:[phone],
E-mail:[e-mail]queston-lineviathe
Elsevierhomepage(http://elsevier.com),byselecting“CustomerSupport”andthen
“ObtainingPermissions.”
LibraryofCongressCataloging-in-PublicationData
Ibe,OliverC.(OliverChukwudi),1947-
Markovprocessforstochasticmodeling/OliverC.Ibe.
p.cm.
ISBN978-0-12-374451-7(hardcover:alk.paper)1.Markovprocesses.2.Stochastic
processes.I.Title.II.Series:Stochasticmodeling.
QA274.7.I242009
519.2’33–dc22
2008021501
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
ISBN13:978-0-12-3744451-7
ForallinformationonallElsevierAcademicPresspublications
visitourWebsiteatwww.elsevierdirect.com
PrintedintheUnitedStatesofAmerica
08 09 10 9 8 7 6 5 4 3 2 1
Contents
Preface xiii
Acknowledgments xiv
1. Basic Concepts 1
1.1. ReviewofProbability 1
1.1.1. ConditionalProbability 4
1.1.2. Independence 4
1.1.3. TotalProbabilityandtheBayes’Theorem 5
1.2. RandomVariables 6
1.2.1. DistributionFunctions 6
1.2.2. DiscreteRandomVariables 6
1.2.3. ContinuousRandomVariables 7
1.2.4. Expectations 8
1.2.5. ExpectationofNonnegativeRandomVariables 8
1.2.6. MomentsofRandomVariablesandtheVariance 8
1.3. TransformMethods 9
1.3.1. Thes-Transform 9
1.3.2. Thez-Transform 10
1.4. BivariateRandomVariables 12
1.4.1. DiscreteBivariateRandomVariables 13
1.4.2. ContinuousBivariateRandomVariables 13
1.4.3. CovarianceandCorrelationCoefficient 14
1.5. ManyRandomVariables 14
1.6. Fubini’sTheorem 15
1.7. SumsofIndependentRandomVariables 16
1.8. SomeProbabilityDistributions 18
1.8.1. TheBernoulliDistribution 18
1.8.2. TheBinomialDistribution 19
1.8.3. TheGeometricDistribution 20
1.8.4. ThePascalDistribution 20
1.8.5. ThePoissonDistribution 21
1.8.6. TheExponentialDistribution 21
1.8.7. TheErlangDistribution 22
1.8.8. NormalDistribution 23
1.9. IntroductiontoStochasticProcesses 24
v
vi Contents
1.10. ClassificationofStochasticProcesses 25
1.11. CharacterizingaStochasticProcess 25
1.11.1. MeanandAutocorrelationFunctionofaStochastic
Process 26
1.12. StationaryStochasticProcesses 27
1.12.1. Strict-SenseStationaryProcesses 27
1.12.2. Wide-SenseStationaryProcesses 28
1.13. ErgodicStochasticProcesses 28
1.14. SomeModelsofStochasticProcesses 29
1.14.1. Martingales 29
1.14.2. CountingProcesses 32
1.14.3. IndependentIncrementProcesses 32
1.14.4. StationaryIncrementProcess 33
1.14.5. PoissonProcesses 33
1.15. Problems 38
2. Introduction to Markov Processes 45
2.1. Introduction 45
2.2. StructureofMarkovProcesses 46
2.3. StrongMarkovProperty 48
2.4. ApplicationsofDiscrete-TimeMarkovProcesses 49
2.4.1. BranchingProcesses 49
2.4.2. SocialMobility 50
2.4.3. MarkovDecisionProcesses 50
2.5. ApplicationsofContinuous-TimeMarkovProcesses 50
2.5.1. QueueingSystems 50
2.5.2. Continuous-TimeMarkovDecisionProcesses 51
2.5.3. StochasticStorageSystems 51
2.6. ApplicationsofContinuous-StateMarkovProcesses 52
2.6.1. ApplicationofDiffusionProcessestoFinancialOptions 52
2.6.2. ApplicationsofBrownianMotion 52
3. Discrete-Time Markov Chains 55
3.1. Introduction 55
3.2. StateTransitionProbabilityMatrix 56
3.2.1. Then-StepStateTransitionProbability 56
3.3. StateTransitionDiagrams 58
3.4. ClassificationofStates 59
3.5. Limiting-StateProbabilities 61
3.5.1. DoublyStochasticMatrix 65
Contents vii
3.6. SojournTime 66
3.7. TransientAnalysisofDiscrete-TimeMarkovChains 67
3.8. FirstPassageandRecurrenceTimes 69
3.9. OccupancyTimes 72
3.10. AbsorbingMarkovChainsandtheFundamentalMatrix 73
3.10.1. TimetoAbsorption 74
3.10.2. AbsorptionProbabilities 77
3.11. ReversibleMarkovChains 78
3.12. Problems 79
4. Continuous-Time Markov Chains 83
4.1. Introduction 83
4.2. TransientAnalysis 86
4.3. BirthandDeathProcesses 90
4.3.1. LocalBalanceEquations 94
4.3.2. TransientAnalysisofBirthandDeathProcesses 95
4.4. FirstPassageTime 96
4.5. TheUniformizationMethod 98
4.6. ReversibleContinuous-TimeMarkovChains 99
4.7. Problems 99
5. Markovian Queueing Systems 105
5.1. Introduction 105
5.2. DescriptionofaQueueingSystem 105
5.3. TheKendallNotation 108
5.4. TheLittle’sFormula 109
5.5. ThePASTAProperty 110
5.6. TheM/M/1QueueingSystem 110
5.6.1. StochasticBalance 114
5.6.2. TotalTimeandWaitingTimeDistributionsoftheM/M/1
QueueingSystem 114
5.7. ExamplesofOtherM/MQueueingSystems 117
5.7.1. TheM/M/cQueue:Thec-ServerSystem 118
5.7.2. TheM/M/1/KQueue:TheSingle-ServerFinite-Capacity
System 121
5.7.3. TheM/M/c/cQueue:Thec-ServerLossSystem 126
5.7.4. TheM/M/1//KQueue:TheSingle-ServerFinite-Customer
PopulationSystem 128
5.8. M/G/1Queue 130
5.8.1. WaitingTimeDistributionoftheM/G/1Queue 132
viii Contents
5.8.2. TheM/E /1Queue 135
k
5.8.3. TheM/D/1Queue 137
5.8.4. TheM/M/1QueueRevisited 138
5.8.5. TheM/H /1Queue 138
K
5.9. G/M/1Queue 140
5.9.1. TheE /M/1Queue 144
k
5.9.2. TheD/M/1Queue 145
5.9.3. TheH /M/1Queue 146
2
5.10. ApplicationsofMarkovianQueues 147
5.11. Problems 148
6. Markov Renewal Processes 153
6.1. Introduction 153
6.2. TheRenewalEquation 154
6.2.1. AlternativeApproach 156
6.3. TheElementaryRenewalTheorem 158
6.4. RandomIncidenceandResidualTime 159
6.5. MarkovRenewalProcess 161
6.5.1. TheMarkovRenewalFunction 162
6.6. Semi-MarkovProcesses 164
6.6.1. Discrete-TimeSemi-MarkovProcesses 165
6.6.2. Continuous-TimeSemi-MarkovProcesses 170
6.7. MarkovJumpProcesses 175
6.7.1. TheHomogeneousMarkovJumpProcess 178
6.8. Problems 181
7. Markovian Arrival Processes 185
7.1. Introduction 185
7.2. OverviewofMatrix-AnalyticMethods 186
7.3. MarkovianArrivalProcess 191
7.3.1. PropertiesofMAP 194
7.4. BatchMarkovianArrivalProcess 196
7.4.1. PropertiesofBMAP 199
7.5. Markov-ModulatedPoissonProcess 200
7.5.1. TheInterruptedPoissonProcess 201
7.5.2. TheSwitchedPoissonProcess 203
7.5.3. PropertiesofMMPP 203
7.5.4. TheMMPP(2)/M/1Queue 205
7.6. Markov-ModulatedBernoulliProcess 209
7.6.1. TheMMBP(2) 210
Contents ix
7.7. SampleApplicationsofMAPandItsDerivatives 212
7.8. Problems 213
8. Random Walk 215
8.1. Introduction 215
8.2. TheTwo-DimensionalRandomWalk 217
8.3. RandomWalkasaMarkovChain 218
8.4. SymmetricRandomWalkasaMartingale 219
8.5. RandomWalkwithBarriers 219
8.6. Gambler’sRuin 220
8.6.1. RuinProbability 220
8.6.2. DurationofaGame 222
8.7. FirstReturnTimes 224
8.8. FirstPassageTimes 227
8.9. MaximumofaRandomWalk 229
8.10. RandomWalkonaGraph 231
8.10.1. RandomWalkonaWeightedGraph 236
8.11. MarkovRandomWalk 237
8.11.1. MarkovRandomWalkinSemisupervisedMachine
Learning 239
8.12. RandomWalkwithCorrelation 240
8.13. Continuous-TimeRandomWalk 246
8.13.1. TheMasterEquation 248
8.14. SampleApplicationsofRandomWalk 251
8.14.1. TheBallotProblem 251
8.14.2. WebSearch 254
8.14.3. MobilityModelsinMobileNetworks 255
8.14.4. InsuranceRisk 257
8.14.5. ContentofaDam 257
8.14.6. CashManagement 258
8.15. Problems 258
9. Brownian Motion and Diffusion Processes 263
9.1. Introduction 263
9.2. BrownianMotion 263
9.2.1. BrownianMotionwithDrift 265
9.2.2. BrownianMotionasaMarkovProcess 265
9.2.3. BrownianMotionasaMartingale 266
9.2.4. FirstPassageTimeofaBrownianMotion 266
9.2.5. MaximumofaBrownianMotion 268