Markov processes and applications: Algorithms, networks, genome and finance

324 Pages·2009·1.32 MB·English

by  Etienne Pardoux

Markov Processes
and Applications
Algorithms, Networks,
Genome and Finance
E´tienne Pardoux
Laboratoire d’Analyse, Topologie, Probabilite´s
Centre de Mathe´matiques et d’Informatique
Universite´ de Provence, Marseille, France.
A John Wiley and Sons, Ltd., Publication
ThisworkisintheWiley-DunodSeriesco-publishedbetweenDunodandJohnWiley&Sons,Ltd.

Markov Processes
and Applications
WILEY SERIES IN PROBABILITY AND STATISTICS
Established by WALTER A. SHEWHART and SAMUEL S. WILKS
Editors: DavidJ.Balding,NoelA.C.Cressie,GarrettM.Fitzmaurice,
IainM.Johnstone,GeertMolenberghs,DavidW.Scott,AdrianF.M.Smith,
RueyS.Tsay,SanfordWeisberg
Editors Emeriti: VicBarnett,J.StuartHunter,JozefL.Teugels
A complete list of titles in this series appears at the end of the volume.
Markov Processes
and Applications
Algorithms, Networks,
Genome and Finance
E´tienne Pardoux
Laboratoire d’Analyse, Topologie, Probabilite´s
Centre de Mathe´matiques et d’Informatique
Universite´ de Provence, Marseille, France.
A John Wiley and Sons, Ltd., Publication
ThisworkisintheWiley-DunodSeriesco-publishedbetweenDunodandJohnWiley&Sons,Ltd.
ThisworkisintheWiley-DunodSeriesco-publishedbetweenDunodandJohnWiley&Sons,Ltd.
Thiseditionfirstpublished2008
2008JohnWiley&SonsLtd
Registeredoffice
JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UnitedKingdom
Fordetailsofourglobaleditorialoffices,forcustomerservicesandforinformationabouthowtoapplyfor
permissiontoreusethecopyrightmaterialinthisbookpleaseseeourwebsiteatwww.wiley.com.
Therightoftheauthortobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewiththe
Copyright,DesignsandPatentsAct1988.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,
inanyformorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptas
permittedbytheUKCopyright,DesignsandPatentsAct1988,withoutthepriorpermissionofthepublisher.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmaynot
beavailableinelectronicbooks.
Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks.Allbrand
namesandproductnamesusedinthisbookaretradenames,servicemarks,trademarksorregistered
trademarksoftheirrespectiveowners.Thepublisherisnotassociatedwithanyproductorvendormentioned
inthisbook.Thispublicationisdesignedtoprovideaccurateandauthoritativeinformationinregardtothe
subjectmattercovered.Itissoldontheunderstandingthatthepublisherisnotengagedinrendering
professionalservices.Ifprofessionaladviceorotherexpertassistanceisrequired,theservicesofacompetent
professionalshouldbesought.
LibraryofCongressCataloging-in-PublicationData
Pardoux,E.(Etienne),1947-
Markovprocessesandapplications:algorithms,networks,genome,andfinance/EtiennePardoux.
p.cm.–(Wileyseriesinprobabilityandstatistics)
Includesbibliographicalreferencesandindex.
ISBN978-0-470-77271-3(cloth)
1.Markovprocesses.I.Title.
QA274.7.P3752008
519.2(cid:1)33–dc22
2008043729
AcataloguerecordforthisbookisavailablefromtheBritishLibrary.
ISBN:978-0-470-77271-3
Typesetin10/12ptTimesbyLaserwordsPrivateLimited,Chennai,India
PrintedandboundinGreatBritainbyTJInternational,Padstow,Cornwall
Contents
Preface xi
1 Simulations and the Monte Carlo method 1
1.1 Description of the method . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Convergence theorems . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Simulation of random variables . . . . . . . . . . . . . . . . . . . 5
1.4 Variance reduction techniques . . . . . . . . . . . . . . . . . . . . 9
1.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2 Markov chains 17
2.1 Definitions and elementary properties . . . . . . . . . . . . . . . . 17
2.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.1 Random walk in E =Zd . . . . . . . . . . . . . . . . . . 21
2.2.2 Bienayme´–Galton–Watson process . . . . . . . . . . . . . 21
2.2.3 A discrete time queue . . . . . . . . . . . . . . . . . . . . 22
2.3 Strong Markov property . . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Recurrent and transient states . . . . . . . . . . . . . . . . . . . . 24
2.5 The irreducible and recurrent case . . . . . . . . . . . . . . . . . . 27
2.6 The aperiodic case . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.7 Reversible Markov chain . . . . . . . . . . . . . . . . . . . . . . . 38
2.8 Rate of convergence to equilibrium . . . . . . . . . . . . . . . . . 39
2.8.1 The reversible finite state case . . . . . . . . . . . . . . . . 39
2.8.2 The general case . . . . . . . . . . . . . . . . . . . . . . . 42
2.9 Statistics of Markov chains. . . . . . . . . . . . . . . . . . . . . . 42
2.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3 Stochastic algorithms 57
3.1 Markov chain Monte Carlo. . . . . . . . . . . . . . . . . . . . . . 57
3.1.1 An application . . . . . . . . . . . . . . . . . . . . . . . . 59
3.1.2 The Ising model . . . . . . . . . . . . . . . . . . . . . . . 61
3.1.3 Bayesian analysis of images . . . . . . . . . . . . . . . . . 63
3.1.4 Heated chains. . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2 Simulation of the invariant probability . . . . . . . . . . . . . . . 64
3.2.1 Perfect simulation . . . . . . . . . . . . . . . . . . . . . . 65
vi CONTENTS
3.2.2 Coupling from the past . . . . . . . . . . . . . . . . . . . 68
3.3 Rate of convergence towards the invariant probability . . . . . . . 70
3.4 Simulated annealing . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4 Markov chains and the genome 77
4.1 Reading DNA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.1.1 CpG islands . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.1.2 Detection of the genes in a prokaryotic genome . . . . . . 79
4.2 The i.i.d. model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.3 The Markov model . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3.1 Application to CpG islands . . . . . . . . . . . . . . . . . 80
4.3.2 Search for genes in a prokaryotic genome . . . . . . . . . 81
4.3.3 Statistics of Markov chains Mk . . . . . . . . . . . . . . . 82
4.3.4 Phased Markov chains . . . . . . . . . . . . . . . . . . . . 82
4.3.5 Locally homogeneous Markov chains . . . . . . . . . . . . 82
4.4 Hidden Markov models. . . . . . . . . . . . . . . . . . . . . . . . 84
4.4.1 Computation of the likelihood . . . . . . . . . . . . . . . . 85
4.4.2 The Viterbi algorithm . . . . . . . . . . . . . . . . . . . . 86
4.4.3 Parameter estimation . . . . . . . . . . . . . . . . . . . . . 87
4.5 Hidden semi-Markov model . . . . . . . . . . . . . . . . . . . . . 92
4.5.1 Limitations of the hidden Markov model . . . . . . . . . . 92
4.5.2 What is a semi-Markov chain? . . . . . . . . . . . . . . . 92
4.5.3 The hidden semi-Markov model . . . . . . . . . . . . . . . 93
4.5.4 The semi-Markov Viterbi algorithm . . . . . . . . . . . . . 94
4.5.5 Search for genes in a prokaryotic genome . . . . . . . . . 95
4.6 Alignment of two sequences . . . . . . . . . . . . . . . . . . . . . 97
4.6.1 The Needleman–Wunsch algorithm . . . . . . . . . . . . . 98
4.6.2 Hidden Markov model alignment algorithm . . . . . . . . 99
4.6.3 A posteriori probability distribution of the alignment . . . 102
4.6.4 A posteriori probability of a given match . . . . . . . . . . 104
4.7 A multiple alignment algorithm . . . . . . . . . . . . . . . . . . . 105
4.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5 Control and filtering of Markov chains 109
5.1 Deterministic optimal control . . . . . . . . . . . . . . . . . . . . 109
5.2 Control of Markov chains . . . . . . . . . . . . . . . . . . . . . . 111
5.3 Linear quadratic optimal control . . . . . . . . . . . . . . . . . . . 111
5.4 Filtering of Markov chains . . . . . . . . . . . . . . . . . . . . . . 113
5.5 The Kalman–Bucy filter . . . . . . . . . . . . . . . . . . . . . . . 115
5.5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.5.2 Solution of the filtering problem . . . . . . . . . . . . . . 116
5.6 Linear–quadratic control with partial observation. . . . . . . . . . 120
5.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
CONTENTS vii
6 The Poisson process 123
6.1 Point processes and counting processes . . . . . . . . . . . . . . . 123
6.2 The Poisson process . . . . . . . . . . . . . . . . . . . . . . . . . 124
6.3 The Markov property . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.4 Large time behaviour . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7 Jump Markov processes 135
7.1 General facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.2 Infinitesimal generator . . . . . . . . . . . . . . . . . . . . . . . . 139
7.3 The strong Markov property . . . . . . . . . . . . . . . . . . . . . 142
7.4 Embedded Markov chain . . . . . . . . . . . . . . . . . . . . . . . 144
7.5 Recurrent and transient states . . . . . . . . . . . . . . . . . . . . 147
7.6 The irreducible recurrent case . . . . . . . . . . . . . . . . . . . . 148
7.7 Reversibility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.8 Markov models of evolution and phylogeny . . . . . . . . . . . . 154
7.8.1 Models of evolution . . . . . . . . . . . . . . . . . . . . . 156
7.8.2 Likelihood methods in phylogeny . . . . . . . . . . . . . . 160
7.8.3 The Bayesian approach to phylogeny . . . . . . . . . . . . 163
7.9 Application to discretized partial differential equations . . . . . . . 166
7.10 Simulated annealing . . . . . . . . . . . . . . . . . . . . . . . . . 167
7.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
8 Queues and networks 179
8.1 M/M/1 queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
8.2 M/M/1/K queue. . . . . . . . . . . . . . . . . . . . . . . . . . . 182
8.3 M/M/s queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
8.4 M/M/s/s queue . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
8.5 Repair shop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
8.6 Queues in series . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
8.7 M/G/∞ queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
8.8 M/G/1 queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
8.8.1 An embedded chain . . . . . . . . . . . . . . . . . . . . . 187
8.8.2 The positive recurrent case . . . . . . . . . . . . . . . . . 188
8.9 Open Jackson network . . . . . . . . . . . . . . . . . . . . . . . . 190
8.10 Closed Jackson network . . . . . . . . . . . . . . . . . . . . . . . 194
8.11 Telephone network . . . . . . . . . . . . . . . . . . . . . . . . . . 196
8.12 Kelly networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
8.12.1 Single queue . . . . . . . . . . . . . . . . . . . . . . . . . 199
8.12.2 Multi-class network . . . . . . . . . . . . . . . . . . . . . 202
8.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
9 Introduction to mathematical finance 205
9.1 Fundamental concepts . . . . . . . . . . . . . . . . . . . . . . . . 205
9.1.1 Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206