Table Of ContentAPM 541: Stochastic Modelling in Biology
Continuous-time Markov Chains
Jay Taylor
Fall 2013
JayTaylor (ASU) APM541 Fall2013 1/77
;
Basics
Continuous-time Markov Processes
Beginning with these slides, we switch focus from discrete-time stochastic processes to
those that are continuous-in-time. Although continuous-time processes are more
complicated in some respects, they are needed in order to be able to effectively model
two kinds of biological processes:
processes with continuously-changing state, such as the diffusion of a particle
through the cytoplasm of a cell;
processes with discrete changes that occur at irregular times, such as births and
deaths in a population that breeds at different rates throughout the year.
Before introducing the general theory, we will start by describing a particularly
important example of a continuous-time Markov process called the Poisson process.
These can be thought of as building blocks for the larger class of continuous-time
Markov chains, but are also important models in their own right.
JayTaylor (ASU) APM541 Fall2013 2/77
Basics
Example: The Poisson Process
Let η ,η ,··· be a sequence of independent exponentially-distributed random variables
1 2
with rate λ and define a new sequence of random variables N =(N ;t ≥0), with
t
continuous time parameter t, by
N =sup{n≥0:η +···+η <t}.
t 1 n
The process N defined in this manner is said to be a Poisson process with rate λ and
is an example of a counting process. It is often interpreted in the following manner.
Think of η as the random waiting time between successive events and let
i
T =η +···+η
n 1 n
be the time of the n’th event. Then N is the number of events that have occurred up
t
to time t.
JayTaylor (ASU) APM541 Fall2013 3/77
Basics
To calculate the distribution of N , we need to recall that the distribution of a sum of n
t
i.i.d. exponential random variables with parameter λ is the Gamma distribution with
parameters n and λ. Then, since
{N =n}={T ≤t <T +η }
t n n n+1
it follows that
P(N =n) = P(T ≤t <T +η )
t n n n+1
Z t
= P(T =s)P(η >t−s)ds
n n+1
0
Z t λn
= sn−1e−λse−λ(t−s)ds
Γ(n)
0
λn Z t
= e−λt sn−1ds
(n−1)!
0
(λt)n
= e−λt .
n!
Since this holds for every n≥0, it follows that N is Poisson-distributed with parameter
t
λt. Of course, this is also the source of the name of this process.
JayTaylor (ASU) APM541 Fall2013 4/77
Basics
With some additional work, it can be shown that the process N also has the following
properties:
1 N has independent increments: for all 0≤t0 <t1 <···<tk, the random
variables N , N −N , N −N ,··· ,N −N are independent.
t0 t1 t0 t2 t1 tk tk−1
2 For each t >0,
P(N =n+1|N =n) = λδt+o(δt)
t+δt t
P(N =n|N =n) = 1−λδt+o(δt)
t+δt t
P(N =i|N =n) = o(δt), for every i (cid:54)=n,n+1.
t+δt t
Both of these properties are related to the fact that the Poisson process is itself a
Markov process. The first property asserts that the numbers of events occurring in
disjoint time intervals are independent; in fact, this is stronger than the Markov
property. The second property asserts that the probability of having more than one
jump in any short time interval is negligibly small. It also asserts that the rate of jumps
from n to n+1 is equal to λ.
JayTaylor (ASU) APM541 Fall2013 5/77
Basics
We now turn to the general definitions.
Definition
A stochastic process X =(X ;t ≥0) with values in a set E is said to be a
t
continuous-time Markov process if for every sequence of times 0≤t <···<t <t
1 n n+1
and every set of values x ,··· ,x ∈E, we have
1 n
P(X ∈A|X =x ,··· ,X =x )=P(X ∈A|X =x ),
tn+1 t1 1 tn n tn+1 tn n
whenever A is a subset of E such that {X ∈A} is an event. In this case, the
tn+1
function defined by
p(s,t;x,A)=P(X ∈A|X =x), t ≥s ≥0
t s
is called the transition function of X. If this function depends on s and t only through
the difference t−s, i.e., if we can write
p(t−s,x,A)=P(X ∈A|X =x)
t s
for every t ≥s ≥0, then we say that X is time-homogeneous.
JayTaylor (ASU) APM541 Fall2013 6/77
Basics
Sample Paths
Remarks: A continuous-time stochastic process X =(X :t ≥0) can be thought of in
t
two different ways.
Ontheonehand,X issimplyacollectionofrandomvariablesdefinedonthesame
probability space, Ω.
On the other hand, we can also think of X as a path- or function-valued random
variable. Inotherwords,givenanoutcomeω∈Ω,wewillviewX(ω)asafunction
from [0,∞) into E defined by
X(ω)(t)≡X (ω)∈E.
t
ThepathtracedoutbyX(ω)ast variesfrom0to∞issaidtobeasample pathofX.
JayTaylor (ASU) APM541 Fall2013 7/77
Basics
In these notes, we will mainly be concerned with Markov processes that take values in
countable state spaces. As in our discussion of discrete-time Markov chains, we will
assume without loss of generality that the state space is either E ={1,2,··· ,n} or
E ={1,2,···}.
Definition
A stochastic process X =(X ;t ≥0) with values in a countable set E ={1,2,···} is
t
said to be a time-homogeneous continuous-time Markov chain (CTMC) if for every
sequence of times 0≤t <···<t <t and every set of values x ,··· ,x ∈E,
1 n n+1 1 n+1
there is a function
p:[0,∞)×E ×E →[0,1]
such that
P(X =x |X =x ,··· ,X =x )=p(t −t ,x ,x ).
tn+1 n+1 t1 1 tn n n+1 n n n+1
JayTaylor (ASU) APM541 Fall2013 8/77
Basics
Transition Matrices
An important difference between the treatment of discrete-time and continuous-time
Markov chains is that in the latter case there is no one canonical transition matrix that
is used to characterize the entire process. Instead, if X =(X :t ≥0) is a
t
continuous-time Markov chain, we can define an entire family of transition matrices
indexed by time. Specifically, for each t ≥0 and each pair of elements i,j ∈E, let
p (t)≡p(t,i,j)=P(X =j|X =i)
ij t 0
and let P(t) be the matrix with entries p (t). In particular, if E ={1,2,··· ,n} is
ij
finite, then P(t) is the n×n matrix
0 p (t) p (t) ··· p (t) 1
11 12 1n
B p21(t) p22(t) ··· p2n(t) C
P(t)=B . . . C.
B . . . C
@ . . . A
p (t) p (t) ··· p (t)
n1 n2 nn
JayTaylor (ASU) APM541 Fall2013 9/77
Basics
It can be shown that the family of transition matrices (P(t):t ≥0) satisfies the
following properties:
For each t ≥0, P(t) is a stochastic matrix:
X
p (t)=1.
ij
j∈E
P(0) is the identity matrix: p (0)=δ .
ij ij
For every s,t ≥0, the matrices P(s) and P(t) commute and
P(t+s)=P(t)P(s).
The third property is called the semigroup property and the family of matrices is said
to be a transition semigroup. When written in terms of coordinates, this property is
X
p (s+t)= p (s)p (t)
ij ik kj
k∈E
which we recognize as the continuous-time analogue of the Chapman-Kolmogorov
equations.
JayTaylor (ASU) APM541 Fall2013 10/77
Description:Continuous-time Markov Processes. Beginning with these slides, we switch focus
from discrete-time stochastic processes to those that are continuous-in-time.
