Approximate iterative algorithms

356 Pages·2014·2.801 MB·English

by  Almudevar, Anthony

Approximate Iterative Algorithms
Approximate Iterative Algorithms
Anthony Almudevar
Department of Biostatistics and Computational Biology,
University of Rochester,Rochester,NY,USA
CRCPress/BalkemaisanimprintoftheTaylor&FrancisGroup,aninformabusiness
©2014Taylor&FrancisGroup,London,UK
TypesetbyMPSLimited,Chennai,India
PrintedandBoundbyCPIGroup(UK)Ltd,Croydon,CR04YY
Allrightsreserved.Nopartofthispublicationortheinformationcontained
hereinmaybereproduced,storedinaretrievalsystem,ortransmittedinany
formorbyanymeans,electronic,mechanical,byphotocopying,recordingor
otherwise,withoutwrittenpriorpermissionfromthepublisher.
Althoughallcareistakentoensureintegrityandthequalityofthispublication
andtheinformationherein,noresponsibilityisassumedbythepublishersnor
theauthorforanydamagetothepropertyorpersonsasaresultofoperation
oruseofthispublicationand/ortheinformationcontainedherein.
LibraryofCongressCataloging-in-PublicationData
Almudevar,Anthony,author.
Approximateiterativealgorithms/AnthonyAlmudevar,Departmentof
BiostatisticsandComputationalBiology,UniversityofRochester,Rochester,NY,USA.
pagescm
Includesbibliographicalreferencesandindex.
ISBN978-0-415-62154-0(hardback)—ISBN978-0-203-50341-6(eBookPDF)
1. Approximationalgorithms. 2. Functionalanalysis. 3. Probabilities.
4. Markovprocesses. I.Title.
QA76.9.A43A46 2014
519.2(cid:1)33—dc23
2013041800
Publishedby: CRCPress/Balkema
P.O.Box11320,2301EHLeiden,TheNetherlands
e-mail:[email protected]
www.crcpress.com–www.taylorandfrancis.com
ISBN:978-0-415-62154-0(Hardback)
ISBN:978-0-203-50341-6(eBookPDF)
Table of contents
1 Introduction 1
PARTI
Mathematical background 3
2 Realanalysisandlinearalgebra 5
2.1 Definitionsandnotation 5
2.1.1 Numbers,setsandvectors 5
2.1.2 Logicalnotation 6
2.1.3 Setalgebra 6
2.1.4 Thesupremumandinfimum 7
2.1.5 Roundingoff 7
2.1.6 Functions 7
2.1.7 Sequencesandlimits 8
2.1.8 Infiniteseries 9
2.1.9 Geometricseries 11
2.1.10 Classesofrealvaluedfunctions 11
2.1.11 Graphs 12
2.1.12 Thebinomialcoefficient 13
2.1.13 Stirling’sapproximationofthefactorial 13
2.1.14 L’Hôpital’srule 14
2.1.15 Taylor’stheorem 14
2.1.16 Thelp norm 14
2.1.17 Powermeans 15
2.2 Equivalencerelationships 16
2.3 Linearalgebra 16
2.3.1 Matrices 16
2.3.2 Eigenvaluesandspectraldecomposition 18
2.3.3 Symmetric,Hermitianandpositivedefinitematrices 21
2.3.4 Positivematrices 22
2.3.5 Stochasticmatrices 24
2.3.6 Nonnegativematricesandgraphstructure 25
vi Table of contents
3 Background–measuretheory 27
3.1 Topologicalspaces 27
3.1.1 Basesoftopologies 29
3.1.2 Metricspacetopologies 29
3.2 Measurespaces 30
3.2.1 Formalconstructionofmeasures 31
3.2.2 Completionofmeasures 33
3.2.3 Outermeasure 34
3.2.4 Extensionofmeasures 35
3.2.5 Countingmeasure 36
3.2.6 Lebesguemeasure 36
3.2.7 Borelsets 36
3.2.8 Dynkinsystemtheorem 37
3.2.9 Signedmeasures 37
3.2.10 Decompositionofmeasures 38
3.2.11 Measurablefunctions 39
3.3 Integration 41
3.3.1 Convergenceofintegrals 42
3.3.2 Lp spaces 43
3.3.3 Radon-Nikodymderivative 43
3.4 Productspaces 44
3.4.1 Producttopologies 44
3.4.2 Productmeasures 45
3.4.3 TheKolmogorovextensiontheorem 49
4 Background–probabilitytheory 51
4.1 Probabilitymeasures–basicproperties 52
4.2 Momentgeneratingfunctions(MGF)andcumulantgenerating
functions(CGF) 59
4.2.1 Momentsandcumulants 61
4.2.2 MGFandCGFofindependentsums 62
4.2.3 RelationshipoftheCGFtothenormaldistribution 62
4.2.4 Probabilitygeneratingfunctions 63
4.3 Conditionaldistributions 63
4.4 Martingales 66
4.4.1 Stoppingtimes 68
4.5 Someimportanttheorems 68
4.6 Inequalitiesfortailprobabilities 70
4.6.1 Chernoffbounds 71
4.6.2 Chernoffboundforthenormaldistribution 71
4.6.3 Chernoffboundforthegammadistribution 72
4.6.4 Samplemeans 72
4.6.5 Someinequalitiesforboundedrandomvariables 73
4.7 Stochasticordering 74
4.7.1 MGForderingofthegammaandexponentialdistribution 75
4.7.2 Improvedboundsbasedonhazardfunctions 76
4.8 Theoryofstochasticlimits 77
4.8.1 Covergenceofrandomvariables 77
Table of contents vii
4.8.2 Convergenceofmeasures 78
4.8.3 Totalvariationnorm 79
4.9 Stochastickernels 82
4.9.1 Measurabilityofmeasurekernels 83
4.9.2 Continuityofmeasurekernels 84
4.10 Convergenceofsums 85
4.11 Thelawoflargenumbers 86
4.12 Extremevaluetheory 88
4.13 Maximumlikelihoodestimation 89
4.14 Nonparametricestimatesofdistributions 91
4.15 Totalvariationdistancefordiscretedistributions 92
5 Background–stochasticprocesses 97
5.1 Countingprocesses 97
5.1.1 Renewalprocesses 98
5.1.2 Poissonprocess 99
5.2 Markovprocesses 100
5.2.1 Discretestatespaces 101
5.2.2 GlobalpropertiesofMarkovchains 102
5.2.3 Generalstatespaces 106
5.2.4 Geometricergodicity 109
5.2.5 SpectralpropertiesofMarkovchains 111
5.3 Continuous-timeMarkovchains 111
5.3.1 Birthanddeathprocesses 114
5.4 Queueingsystems 114
5.4.1 Queueingsystemsasbirthanddeathprocesses 115
5.4.2 Utilizationfactor 116
5.4.3 GeneralqueueingsystemsandembeddedMarkov
chains 116
5.5 Adaptedcountingprocesses 118
5.5.1 Asymptoticbehavior 119
5.5.2 Relationshiptoadaptedevents 123
6 Functionalanalysis 125
6.1 Metricspaces 126
6.1.1 Contractivemappings 126
6.2 TheBanachfixedpointtheorem 128
6.2.1 Stoppingrulesforfixedpointalgorithms 130
6.3 Vectorspaces 131
6.3.1 Quotientspaces 133
6.3.2 Basisofavectorspace 133
6.3.3 Operators 133
6.4 Banachspaces 134
6.4.1 Banachspacesandcompleteness 136
6.4.2 Linearoperators 137
6.5 Normsandnormequivalence 139
6.5.1 Normdominance 140
6.5.2 Equivalencepropertiesofnormequivalenceclasses 140
viii Table of contents
6.6 Quotientspacesandseminorms 142
6.7 Hilbertspaces 144
6.8 ExamplesofBanachspaces 146
6.8.1 Finitedimensionalspaces 146
6.8.2 Matrixnormsandthesubmultiplicativeproperty 147
6.8.3 Weightednormsonfunctionspaces 147
6.8.4 Spanseminorms 149
6.8.5 Operatorsonspanquotientspaces 151
6.9 Measurekernelsaslinearoperators 153
6.9.1 Thecontractionpropertyofstochastickernels 153
6.9.2 Stochastickernelsandthespanseminorm 153
7 Fixedpointequations 157
7.1 Contractionasanormequivalenceproperty 158
7.2 Linearfixedpointequations 160
7.3 Thegeometricseriestheorem 161
7.4 Invarianttransformationsoffixedpointequations 163
7.5 Fixedpointalgorithmsandthespanseminorm 164
7.5.1 Approximationsinthespanseminorm 166
7.5.2 Magnitudeoffixedpointsinthespanseminorm 167
7.6 Stoppingrulesforfixedpointalgorithms 168
7.6.1 Fixedpointiterationinthespanseminorm 169
7.7 Perturbationsoffixedpointequations 169
8 Thedistributionofamaximum 173
8.1 Generalapproach 174
¯
8.2 BoundsonMbasedonMGFs 174
8.2.1 Samplemeans 176
8.2.2 Gammadistribution 177
8.3 Boundsforvaryingmarginaldistributions 178
8.3.1 Example 180
8.4 Tailprobabilitiesofmaxima 181
8.4.1 Extremevaluedistributions 182
8.4.2 TailprobabilitiesbasedonBoole’sinequality 182
8.4.3 Thenormalcase 183
8.4.4 Thegamma(α,λ)case 184
8.5 Variancemixturesbasedonrandomsamplesizes 185
8.6 Boundsformaximabasedonthefirsttwomoments 186
8.6.1 Stability 188
PARTII
General theory of approximate iterative algorithms 189
9 Background–linearconvergence 191
9.1 Linearconvergence 191
9.2 Constructionofenvelopes–thenonstochasticcase 194
Table of contents ix
9.3 Constructionofenvelopes–thestochasticcase 196
9.4 Aversionofl’Hôpital’sruleforseries 196
10 Ageneraltheoryofapproximateiterativealgorithms(AIA) 199
10.1 Ageneraltolerancemodel 201
10.2 Example:apreliminarymodel 201
10.3 ModelelementsofanAIA 202
10.3.1 Lipschitzkernels 202
10.3.2 Lipschitzconvolutions 203
10.4 AclassificationsystemforAIAs 204
10.4.1 Relativeerrormodel 206
10.5 Generalinequalities 208
10.5.1 HilbertspacemodelsofAIAs 209
10.6 Nonexpansiveoperators 213
10.6.1 Applicationofgeneralinequalitiestononexpansive
AIAs 214
10.6.2 WeaklycontractiveAIAs 216
10.6.3 Examples 216
10.6.4 Stochasticapproximation(Robbins-Monro
algorithm) 218
10.7 RatesofconvergenceforAIAs 220
10.7.1 MonotonicityoftheLipschitzkernel 221
10.7.2 CaseI–stronglycontractivemodelswithnonvanishing
bounds 221
10.7.3 CaseII–rapidlyvanishingapproximationerror 222
10.7.4 CaseIII–approximationerrordecreasingat
contractionrate 223
10.7.5 CaseIV–Approximationerrorgreaterthan
contractionrate 224
10.7.6 CaseV–Contractionratesapproaching1 224
10.7.7 Adjustmentsforrelativeerrormodels 227
10.7.8 AcomparisonofBanachspaceandHilbertspace
models 228
10.8 Stochasticapproximationasaweaklycontractivealgorithm 230
10.9 Tightnessofalgorithmtolerance 231
10.10 Finitebounds 232
10.10.1 Numericalexample 233
10.11 Summaryofconvergenceratesforstronglycontractivemodels 235
11 Selectionofapproximationschedulesforcoarse-to-fineAIAs 239
11.1 Extendingthetolerancemodel 239
11.1.1 Comparisonmodelfortoleranceschedules 241
11.1.2 Regularityconditionsforthecomputationfunction 242
11.2 Mainresult 242
11.3 Examplesofcostfunctions 243
11.4 AgeneralprincipleforAIAs 245
x Table of contents
PARTIII
Application to Markov decision processes 247
12 Markovdecisionprocesses(MDP)–background 249
12.1 Modeldefinition 250
12.2 Theoptimalcontrolproblem 253
12.2.1 Adaptivecontrolpolicies 253
12.2.2 Optimalcontrolpolicies 253
12.3 Dynamicprogrammingandlinearoperators 255
12.3.1 Thedynamicprogrammingoperator(DPO) 256
12.3.2 Finitehorizondynamicprogramming 257
12.3.3 Infinitehorizonproblem 258
12.3.4 ClassesofMDP 259
12.3.5 MeasurabilityoftheDPO 260
12.4 Dynamicprogrammingandvalueiteration 261
12.4.1 Valueiterationandoptimality 262
12.5 Regretandε-optimalsolutions 265
12.6 Banachspacestructureofdynamicprogramming 267
12.6.1 Thecontractionproperty 269
12.6.2 ContractionpropertiesoftheDPO 269
12.6.3 Theequivalenceofuniformconvergenceand
contractionfortheDPO 272
12.7 AveragecostcriterionforMDP 274
13 Markovdecisionprocesses–valueiteration 279
13.1 Valueiterationonquotientspaces 279
13.2 Contractioninthespanseminorm 281
13.2.1 ContractionpropertiesoftheDPO 281
13.3 Stoppingrulesforvalueiteration 283
13.4 Valueiterationinthespanseminorm 283
13.5 Example:M/D/1/K queueingsystem 284
13.6 Efficientcalculationof|||QJ||| 288
SP
13.7 Example:M/D/1/K systemwithoptimalcontrolofservice
capacity 291
13.8 Policyiteration 292
13.9 Valueiterationfortheaveragecostoptimization 293
14 Modelapproximationindynamicprogramming–generaltheory 295
14.1 ThegeneralinequalityforMDPs 295
14.2 Modeldistance 298
14.3 Regret 300
14.4 Acommentontheapproximationofregret 302
14.5 Example 304
15 Samplingbasedapproximationmethods 309
15.1 Modelingmaxima 310