Lecture notes for Stochastic Processes 2011/2012 by Anders Tolver

144 Pages·2017·0.93 MB·English

university of copenhagen
Lecture notes for Stochastic Processes 2011/2012
Tolver, Anders
Publication date:
2011
Document version
Peer reviewed version
Citation for published version (APA):
Tolver, A. (2011). Lecture notes for Stochastic Processes 2011/2012. Department of Natural Sciences,
University of Copenhagen.
Download date: 05. Feb. 2023
Lecture notes for Stochastic Processes 2011/2012
by Anders Tolver
Intensity Invariant Kolmogorov’s
matrix distribution diff. equation
Q πQ = 0 P(cid:48)(t) = QP(t)
Sample path Transition probabilities
4 1.0
0.8
ate3 ability0.6
St2 Prob0.4
0.2
1 0.0
0.0 0.5 1.0 1.5 2.0 2.5 0.0 0.2 0.4 0.6 0.8 1.0
Time (cid:60)(cid:60)2(cid:79)(cid:79) (cid:88)(cid:88) Time
4
1/2 5
4(cid:108)(cid:108)(cid:124)(cid:124)(cid:120)(cid:120)(cid:114)(cid:114)(cid:114)(cid:114)(cid:114)(cid:114)(cid:114)1(cid:114)(cid:114)2(cid:114)(cid:114)(cid:114)(cid:114)11/2(cid:76)(cid:76)(cid:76)(cid:76)3(cid:76)(cid:76)1(cid:76)(cid:76)(cid:76)(cid:76)(cid:76)(cid:76)(cid:76)(cid:38)(cid:38)(cid:44)(cid:44)(cid:24)(cid:24)3
2
Transition diagram
2
Contents
1 Introduction 9
1.1 About these lecture notes . . . . . . . . . . . . . . . . . . . . 10
1.2 Transition diagrams . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Overview of exercises . . . . . . . . . . . . . . . . . . . . . . . 13
2 Discrete-time Markov chains 15
2.0 Results for discrete-time Markov chains . . . . . . . . . . . . . 16
2.0.1 Definition of Markov chains . . . . . . . . . . . . . . . 16
2.0.2 Classification of states . . . . . . . . . . . . . . . . . . 19
2.0.3 Invariant distributions and absorption . . . . . . . . . 22
2.1 Markov chains with two states . . . . . . . . . . . . . . . . . . 28
2.1.1 General two state Markov chain . . . . . . . . . . . . . 28
2.1.2 Recurrence times . . . . . . . . . . . . . . . . . . . . . 29
2.1.3 Two state absorbing Markov chain . . . . . . . . . . . 29
2.1.4 Transition probabilities for the two-state chain . . . . . 30
2.2 Markov chains with three states . . . . . . . . . . . . . . . . . 31
2.2.1 Classification of states . . . . . . . . . . . . . . . . . . 32
2.2.2 General three state Markov chain . . . . . . . . . . . . 33
2.2.3 The one-way Markov chain . . . . . . . . . . . . . . . . 33
2.2.4 Markov property under aggregation of states . . . . . . 34
2.3 Markov chains with finite state space . . . . . . . . . . . . . . 36
2.3.1 Find the communication classes . . . . . . . . . . . . . 36
2.3.2 A numeric example . . . . . . . . . . . . . . . . . . . . 37
2.3.3 Two component repair system . . . . . . . . . . . . . . 38
2.3.4 Random walk reflected at two barriers . . . . . . . . . 40
2.3.5 Yahtzee . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.3.6 Markov chain with two regimes . . . . . . . . . . . . . 43
2.3.7 Periodicity of a Markov chain . . . . . . . . . . . . . . 45
3
4 CONTENTS
2.3.8 More about absorption probabilities . . . . . . . . . . . 47
2.4 Markov chains on countable state spaces . . . . . . . . . . . . 47
2.4.1 Queueing system . . . . . . . . . . . . . . . . . . . . . 48
2.4.2 Random walk on Z . . . . . . . . . . . . . . . . . . . . 50
2.4.3 Random walk on Z2 . . . . . . . . . . . . . . . . . . . 50
2.4.4 Random walk on Zd . . . . . . . . . . . . . . . . . . . 51
2.4.5 Brancing processes . . . . . . . . . . . . . . . . . . . . 51
2.4.6 Positive recurrence and null-recurrence . . . . . . . . . 53
3 Continuous-time Markov chains 55
3.0 Results for continuous-time Markov chains . . . . . . . . . . . 56
3.0.1 Definition of Markov chains . . . . . . . . . . . . . . . 56
3.0.2 Properties of the transition probabilities . . . . . . . . 59
3.0.3 Invariant distributions and absorption . . . . . . . . . 63
3.0.4 Birth-and-death processes . . . . . . . . . . . . . . . . 69
3.1 Markov chains with two states . . . . . . . . . . . . . . . . . . 73
3.1.1 Two-state absorbing Markov chain . . . . . . . . . . . 73
3.1.2 Two-state Markov chain with equal intensities . . . . . 74
3.1.3 Transition probabilities for a two-state chain . . . . . . 75
3.2 Markov chains with three states . . . . . . . . . . . . . . . . . 77
3.2.1 Model for interest rates . . . . . . . . . . . . . . . . . . 77
3.2.2 Model with two states of health and death . . . . . . . 79
3.2.3 Model for disabilities, recoveries, and death . . . . . . . 80
3.2.4 Model for single life with 2 causes of death . . . . . . . 82
3.2.5 Model with one zero in the transition matrix . . . . . . 83
3.2.6 A numerical example . . . . . . . . . . . . . . . . . . . 84
3.3 Markov chains with finite state space . . . . . . . . . . . . . . 86
3.3.1 Model for two lifes . . . . . . . . . . . . . . . . . . . . 86
3.3.2 Single life with r causes of death . . . . . . . . . . . . 87
3.3.3 Forward differential equations for four state chain . . . 90
3.3.4 Time to absorption . . . . . . . . . . . . . . . . . . . . 91
3.4 Markov chains on countable state spaces . . . . . . . . . . . . 93
3.4.1 Pure death process with constant intensity . . . . . . . 93
3.4.2 Linear birth-and-death process . . . . . . . . . . . . . . 95
3.4.3 Queueing systems . . . . . . . . . . . . . . . . . . . . . 97
3.4.4 Positive recurrence and null-recurrence . . . . . . . . . 101
3.4.5 More examples of birth-and-death processes . . . . . . 102
3.5 The Poisson process . . . . . . . . . . . . . . . . . . . . . . . . 104
CONTENTS 5
3.5.1 Basic properties of the Poisson process . . . . . . . . . 104
3.5.2 Advanced exercise involving the Poisson process . . . . 105
4 Additional exercises 107
4.1 Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4.1.1 Assignment 1 from 2010/2011 . . . . . . . . . . . . . . 109
4.1.2 Assignment 2 from 2010/2011 . . . . . . . . . . . . . . 112
4.1.3 Assignment 1 from 2011/2012 . . . . . . . . . . . . . . 116
4.1.4 Assignment 2 from 2011/2012 . . . . . . . . . . . . . . 116
4.2 Mathematical proofs of selected results . . . . . . . . . . . . . 117
4.2.1 Strong Markov property . . . . . . . . . . . . . . . . . 117
4.2.2 Recurrence criterion 1 . . . . . . . . . . . . . . . . . . 117
4.2.3 Number of visits to state j . . . . . . . . . . . . . . . . 118
4.2.4 Recurrence is a class property . . . . . . . . . . . . . . 120
4.2.5 Recurrence criterion 2 . . . . . . . . . . . . . . . . . . 120
4.2.6 Invariant distribution - finite state space . . . . . . . . 120
4.2.7 Invariant distribution . . . . . . . . . . . . . . . . . . . 121
4.2.8 Absorption probabilities - finite state space . . . . . . . 124
4.2.9 Absorption probabilities - countable state space . . . . 124
4.2.10 Backward equations . . . . . . . . . . . . . . . . . . . 124
4.2.11 Uniqueness of invariant distribution (cont) . . . . . . . 124
5 Auxiliary results 127
5.1 Elementary conditional probabilities . . . . . . . . . . . . . . . 128
5.2 Some important probability distributions . . . . . . . . . . . . 128
5.2.1 The binomial distribution . . . . . . . . . . . . . . . . 128
5.2.2 The Poisson distribution . . . . . . . . . . . . . . . . . 129
5.2.3 The geometric distribution . . . . . . . . . . . . . . . . 129
5.2.4 The negative binomial distribution . . . . . . . . . . . 129
5.2.5 The exponential distribution . . . . . . . . . . . . . . . 130
5.2.6 The gamma distribution . . . . . . . . . . . . . . . . . 130
5.3 Useful formulaes for sums and series . . . . . . . . . . . . . . . 131
5.4 Some results for matrices . . . . . . . . . . . . . . . . . . . . . 131
5.4.1 Determinants of a square matrix . . . . . . . . . . . . . 131
5.4.2 Diagonalisation of matrices . . . . . . . . . . . . . . . . 132
5.4.3 Exponential matrices . . . . . . . . . . . . . . . . . . . 133
5.5 First order differential equations . . . . . . . . . . . . . . . . . 134
5.6 Second order linear recurrence equations . . . . . . . . . . . . 135
6 CONTENTS
5.7 R stuff for dealing with Markov chains . . . . . . . . . . . . . 136
5.7.1 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.7.2 Computing exponential matrices . . . . . . . . . . . . . 138
5.7.3 Simulation of Markov chains . . . . . . . . . . . . . . . 139
CONTENTS 7
Preface
The present material has been developed for the course Stochastic Processes
at Department of Mathematical Sciences, University of Copenhagen during
the teaching years 2010/2011 and 2011/2012. The topics covered are mainly
Markov chains in discrete and continuous time on finite or countable state
spaces.
The back bone of this work is the collection of exercises in Chapters 2 and
3. Hopefully, all of the theoretical results required to solve the exercises are
contained in the first Section of Chapters 2 and 3. The manuscript was never
intended to provide complete mathematical proofs of the main results since
these may be found elsewhere in the literature. It is my intention to spend
part of the lectures on (sketches of) proofs in order to illustrate how to work
with Markov chains in a formally correct way. Some exercises in Chapter
4.2 are formulated as step-by-step instructions on how to construct formal
proofs of selected theoretical results. It is definitely advisable (and probably
necessary) to consult other textbooks on Markov chains in order to be able
to solve all of the exercises given here. However, it is my strong belief that
you should postpone the more theoretical exercises until you feel familiar
with most of the exercises in Chapter 2 and 3. For further reading I can
recommend selected Chapters of the books by Asmussen (2003), Br´emaud
(1999) and Lawler (2006). My own introduction to the topic was the lecture
notes (in Danish) by Jacobsen & Keiding (1985).
A number of the exercises presented here are greatly inspired by examples
in Ragner Nordberg’s lecture notes on Basic Life Insurance Mathematics
(Version: September 2002). The paragraphs presenting theoretical results
on Markov chains are greatly inspired by various lecture notes by Jacobsen
& Keiding (1985), M., by Nielsen, S. F., and by Jensen, S. T.
PartofthismaterialhasbeenusedforthecourseStochasticProcesses2010/2011
at University of Copenhagen and I thank Massimiliano Tamborrino, Ketil
Biering Tvermosegaard, and the students for many useful comments to this
revised version.
Copenhagen, November 2011
Anders Tolver
8 CONTENTS
Chapter 1
Introduction
9