Bayesian Inference of State Space Models: Kalman Filtering and Beyond

503 Pages·2021·10.595 MB·English

by  Kostas Triantafyllopoulos

Springer Texts in Statistics
Kostas Triantafyllopoulos
Bayesian
Inference
of State Space
Models
Kalman Filtering and Beyond
Springer Texts in Statistics
SeriesEditors
G.Allen,DepartmentofStatistics,RiceUniversity,Houston,TX,USA
R.DeVeaux,DepartmentofMathematicsandStatistics,WilliamsCollege,
Williamstown,MA,USA
R. Nugent, Department of Statistics, Carnegie Mellon University, Pittsburgh, PA,
USA
SpringerTextsinStatistics(STS)includesadvancedtextbooksfrom3rd-to4th-year
undergraduatecoursesto1st-to2nd-yeargraduatecourses.Exercisesetsshouldbe
included.TheserieseditorsarecurrentlyGeneveraI.Allen,RichardD.DeVeaux,
and Rebecca Nugent. Stephen Fienberg, George Casella, and Ingram Olkin were
editorsoftheseriesformanyyears.
Moreinformationaboutthisseriesathttp://www.springer.com/series/417
Kostas Triantafyllopoulos
Bayesian Inference of State
Space Models
Kalman Filtering and Beyond
KostasTriantafyllopoulos
SchoolofMathematics
UniversityofSheffield
Sheffield,UK
ISSN1431-875X ISSN2197-4136 (electronic)
SpringerTextsinStatistics
ISBN978-3-030-76123-3 ISBN978-3-030-76124-0 (eBook)
https://doi.org/10.1007/978-3-030-76124-0
Mathematics Subject Classification: 62F15, 62M10, 62M20, 91B84, 93E03, 93E11, 62P05, 62P20,
62P30
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerland
AG2021
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewhole orpart ofthematerial isconcerned, specifically therights oftranslation, reprinting, reuse
ofillustrations, recitation, broadcasting, reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG.
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
The discovery and developmentof state space models and the Kalman filter have
theirrootsinthedynamicalandcontrolsystemsresearchcommunities.TheKalman
filterwasproposedinitiallyasanalternativetotheWiener-Kolmogorovestimation
theoryafter Kalman appliedstate variablesto the filtering problem.Soon after its
conception,the Kalman filter was associated with significantreal-data application
such as the Apollo project at the Ames Research Centre at NASA. The Kalman
filterwassoondiscoveredbythestatisticsandeconometricsresearchcommunities,
whichwereabletoexploittheflexibilityofthestatespaceformulationinthequestto
describedynamicsystemsandtousethefilterasapowerfulforecastingframework.
Bayesian inference and forecasting were key areas of development in the 1980s
and1990s.By thattime, the state space modelsandthe Kalmanfilter were found
to be extremely popular in a wide range of applied science, including statistics,
engineering,economics,environmetricsandbiology,tonamebutafew.Partlydue
totheavailabilityofcomputeradvancement,overthelast20years,therehasbeen
anincreasinginterestinmultivariatelinearGaussian,non-linearandnon-Gaussian
modellingandrelatedstatespacemethods.ThisisfacilitatedbydeployingBayesian
computation,inparticularsequentialMonteCarloandMarkovchainMonteCarlo
methods, but also by other estimation procedures such as the unscented Kalman
filterandrelatedalgorithms.
Theprincipleideaofthisbookistobringtogethertheabovemodelswiththeir
statistical inference and make them available to a broad audience. Teaching time
series and state space modelling for over 15 years at the University of Sheffield
hasmotivatedme totake upthisbookproject.Duringtheseyears,I havemetand
collaboratedwithanumberofappliedscientistssharingakeeninterestinstatespace
models. Some of these models can be generated by a natural phenomenon such
as met in engineering or environmetrics and some can be generated by a socio-
economic structure such as in finance and its applications. The book is written
from a statistician’s perspective, and it uses a number of data sets to illustrate
the methods from a wide range of disciplines. The first six chapters discuss state
spacemethodsandBayesianinferenceforgeneraluse;thelasttwochaptersofthe
bookfocusonfinanceanddynamicalsystems.Ononehand,economicsandfinance
v
vi Preface
have been a steady flow of motivation and development of state space methods,
and on the other hand, dynamicalsystems is the key subject area where the story
of the Kalman filter began. Furthermore, the text aims to widespread state space
methodsbydevelopingsuitablesoftwaretoenablethereadertoapplythealgorithms
usingthedatasetsofthebook,butalsousingtheirowndatasets.Wehavecreated
the package BTSA (Bayesian Time Series Analysis), available via the contributed
packageCRANwebsitewithintheenvironmentforstatisticalcomputingR(https://
www.r-project.org). The package includes most functions and data sets used in
Chaps. 1–6; moreover, R commands incorporated in the text is believed to help
understandingandimplementingthealgorithmsoncetheBTSApackageisinstalled.
The book assumes a basic technical background of linear algebra, probability,
andstatistics.ThisisaboutsecondyearuniversitylevelandisreviewedinChap.2.
Thetextbookcan beusedas a one-semestercourseonlinear Gaussianstate space
methodsbycoveringChaps.1–4,perhapswithsomeinclusionofmultivariatestate
spacemodelsinChap.5.Alternatively,thebookcancoverthebasictheoryoflinear
models in Chap. 3 and then move on to non-linear and non-Gaussian models in
Chap.6,perhapsbyincludingsomepartsofChaps.7and8.Thetextbookisaimed
atstudentsatthehigherendofundergraduateorgraduatelevelanditisalsoaimed
atscientistsanddoctoralstudentsforself-study.
Sheffield,UK KostasTriantafyllopoulos
June2021
Acknowledgements
This book would not be possible without the encouragement and input of Nick
Bingham. I am indebted to him for our discussions in numerous occasions in
Sheffield and in London. I am grateful to several colleagues and friends as well
as graduate students for their feedback and support. In particular, I thank Dave
Applebaum, Peter Young, Guy Nason, Clive Anderson, Andrew Harvey, Alan
Zinober, Osman Tokhi, Giovanni Montana, Tata Subba Rao, Maurice Priestley,
Attilio Meucci, John Fry, Jeremy Oakley, and Daniel Molinari. Special thanks
are due to Jeff Harrison who introduced me to state space models and taught me
Bayesianforecasting.
IamgratefultotheSpringerteamandinparticulartoJoergSixtandRemiLodh
whoofferedmegreatsupportduringthelongperiodofthisproject.Theyhavebeen
verypatientandhavewellaccommodatedmypaceofwork.
Finally,I wouldlike to thankmyfamilywhohasbeenverysupportiveandhas
encouragedmetocompletetheproject.
vii
Contents
1 StateSpaceModels .......................................................... 1
1.1 Introduction............................................................ 1
1.1.1 TimeSeries .................................................... 1
1.1.2 ExamplesofTimeSeriesData................................ 2
1.2 WaterTankDynamicsandtheStateSpaceModel................... 3
1.3 ExamplesofStateSpaceModels ..................................... 7
1.3.1 ForecastingAir-PollutionLevels............................. 7
1.3.2 TrackingaShip................................................ 9
1.3.3 StochasticVolatility........................................... 11
1.3.4 HookeanSpringForceDynamics ............................ 13
1.4 AShortHistoryoftheKalmanFilter................................. 15
1.5 LayoutoftheBook.................................................... 19
2 MatrixAlgebra,ProbabilityandStatistics................................ 21
2.1 Vectors,MatricesandBasicOperations.............................. 21
2.2 VectorandMatrixDifferentiation..................................... 24
2.2.1 BackgroundandNotation..................................... 24
2.2.2 DifferentiationofLinearandQuadraticForms.............. 26
2.2.3 DifferentiationofDeterminantandTrace.................... 28
2.2.4 Optimisation,IntegrationandLimits......................... 31
2.3 ProbabilityandDistributionTheory.................................. 34
2.3.1 RandomVectorsandProbabilityDistributions .............. 34
2.3.2 CommonDiscreteDistributions.............................. 37
2.3.3 CommonContinuousDistributions........................... 41
2.4 Statistics................................................................ 48
2.4.1 PrincipleSet-UpandObjectives.............................. 48
2.4.2 MaximumLikelihoodEstimation:TheEMAlgorithm...... 49
2.4.3 BayesianInference............................................ 51
2.5 Exercises............................................................... 55
ix
x Contents
3 TheKalmanFilter ........................................................... 63
3.1 FromRegressiontotheStateSpaceModel .......................... 63
3.1.1 OrdinaryLeastSquares ....................................... 63
3.1.2 RecursiveLeastSquares ...................................... 67
3.1.3 TheStateSpaceModel........................................ 71
3.2 Filtering................................................................ 73
3.2.1 AFirstDerivationoftheKalmanFilter...................... 73
3.2.2 ASecondDerivationoftheKalmanFilter................... 79
3.3 Smoothing.............................................................. 83
3.3.1 Fixed-IntervalSmoothing..................................... 83
3.3.2 TheLag-OneCovarianceSmoother.......................... 88
3.4 Forecasting............................................................. 92
3.5 SteadyStateoftheKalmanFilter..................................... 96
3.5.1 Observability .................................................. 96
3.5.2 SteadyStateoftheLocalLevelModel....................... 98
3.5.3 SteadyStateofLinearStateSpaceModels .................. 100
3.6 Exercises............................................................... 104
4 ModelSpecificationandModelPerformance............................. 111
4.1 SpecificationofModelComponents.................................. 112
4.1.1 TrendStateSpaceModels .................................... 112
4.1.2 SuperpositionofStateSpaceModels......................... 120
4.1.3 FourierFormSeasonalModels............................... 124
4.1.4 Trend-SeasonalModels....................................... 131
4.1.5 Time-VaryingRegression..................................... 135
4.1.6 Time-VaryingAutoregressions ............................... 139
4.2 DecompositionofStateSpaceModels ............................... 141
4.2.1 HistoricalNoteandMotivation............................... 141
4.2.2 RationalCanonicalForm ..................................... 142
4.2.3 DecompositionofLinearStateSpaceModels............... 147
4.2.4 TurkeyDataRevisited......................................... 151
4.3 EstimationofHyperparameters....................................... 156
4.3.1 MaximumLikelihoodEstimation ............................ 156
4.3.2 SpecificationofZ UsingDiscountFactors.................. 165
t
4.3.3 Estimationofσ2:ConjugateBayesianEstimation .......... 167
4.4 ErrorAnalysis.......................................................... 174
4.5 PriorSpecification..................................................... 180
4.5.1 PriorSpecificationofβ ...................................... 181
0
4.5.2 PriorSpecificationofσ2...................................... 184
4.6 AutomaticSequentialMonitoring .................................... 186
4.6.1 ModelMonitoring............................................. 186
4.6.2 SpecificationofAlternativeModels.......................... 190
4.6.3 MonitoringfortheTobaccototalsalesdata—CP6 .......... 193
4.7 Exercises............................................................... 197