Table Of ContentApplied Mathematical Sciences
FoundingEditors
F.John
J.P.LaSalle
L.Sirovich
Volume 215
SeriesEditors
AnthonyBloch,DepartmentofMathematics,UniversityofMichigan,AnnArbor,
MI,USA
C.L.Epstein,DepartmentofMathematics,UniversityofPennsylvania,
Philadelphia,PA,USA
AlainGoriely,DepartmentofMathematics,UniversityofOxford,Oxford,UK
LeslieGreengard,NewYorkUniversity,NewYork,NY,USA
AdvisoryEditors
J.Bell,CenterforComputationalSciencesandEngineering,LawrenceBerkeley
NationalLaboratory,Berkeley,CA,USA
P.Constantin,DepartmentofMathematics,PrincetonUniversity,Princeton,NJ,
USA
R.Durrett,DepartmentofMathematics,DukeUniversity,Durham,CA,USA
R.Kohn,CourantInstituteofMathematicalSciences,NewYorkUniversity,
NewYork,NY,USA
R.Pego,DepartmentofMathematicalSciences,CarnegieMellonUniversity,
Pittsburgh,PA,USA
L.Ryzhik,DepartmentofMathematics,StanfordUniversity,Stanford,CA,USA
A.Singer,DepartmentofMathematics,PrincetonUniversity,Princeton,NJ,USA
A.Stevens,DepartmentofAppliedMathematics,UniversityofMünster,Münster,
Germany
S.Wright,ComputerSciencesDepartment,UniversityofWisconsin,Madison,WI,
USA
Themathematizationofallsciences,thefadingoftraditionalscientificboundaries,
theimpactofcomputertechnology,thegrowingimportanceofcomputermodeling
and the necessity of scientific planning all create the need both in education and
researchforbooksthatareintroductorytoandabreastofthesedevelopments.The
purposeofthisseriesistoprovidesuchbooks,suitablefortheuserofmathematics,
themathematicianinterestedinapplications,andthestudentscientist.Inparticular,
this series will provide an outlet for topics of immediate interest because of the
noveltyofitstreatmentofanapplicationorofmathematicsbeingappliedorlying
close to applications. These books should be accessible to readers versed in
mathematics or science and engineering, and will feature a lively tutorial style, a
focus on topics of current interest, and present clear exposition of broad appeal.
AcomplimenttotheAppliedMathematicalSciencesseriesistheTextsinApplied
Mathematicsseries,whichpublishestextbookssuitableforadvancedundergraduate
andbeginninggraduatecourses.
·
Daniela Calvetti Erkki Somersalo
Bayesian Scientific
Computing
DanielaCalvetti ErkkiSomersalo
DepartmentofMathematics,Applied DepartmentofMathematics,Applied
Mathematics,andStatistics Mathematics,andStatistics
CaseWesternReserveUniversity CaseWesternReserveUniversity
Cleveland,OH,USA Cleveland,OH,USA
ISSN 0066-5452 ISSN 2196-968X (electronic)
AppliedMathematicalSciences
ISBN 978-3-031-23823-9 ISBN 978-3-031-23824-6 (eBook)
https://doi.org/10.1007/978-3-031-23824-6
MathematicsSubjectClassification:65C40,65F22,65F08,62C10,62F15
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SwitzerlandAG2023
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
ToOrianaandGiovanniMadan
Preface
Fifteenyearsago,whentheideaofusingprobabilitytomodelunknownparameters
tobeestimatedcomputationallywasalesscommonlyacceptedideathanitistoday,
writing a book to show how natural and rather straightforward it was to merge
Bayesianinferenceandscientificcomputingfeltabraveanddaringact.Thenatural
environmentforsuchasymbioticrelationtodevelopwasinverseproblems,awell-
establishedresearchareawherebothBayesianinferenceandcomputationalmethods
were establishing their own separate strongholds. One of the main reasons for the
separationisthatprobabilitywasnotpartoftheusualtrainingofanumericalanalyst,
justlikescientificcomputingwasnotpartofthetypicalbackgroundofastatistician
oraprobabilist.Nonetheless,theconcurrentsimilarityandcomplementarityofthe
two fields suggested that if there could be a way to create some synergy between
them,itwouldprobablyleadtonovelandunexpectedfindings.
We ourselves, coming from numerical analysis and Bayesian-physics-based
inverseproblems,wereveryawareofsomeoftheobstaclestobeovercome,including
havingtofillsomeholesinthebackgroundwithoutgettingtoodeepintotheintrica-
ciesofthesefieldsandfeelinglikewehadventuredtoofaroutofourowncomfort
zone. The book Introduction to Bayesian Scientific Computing: Ten Lectures in
SubjectiveComputing[18]wasalight-heartedgatewaytowhatwecalledBayesian
ScientificComputing,wherewedeliberatelychosetonotincludetheoremsorproofs,
emphasizing instead the problem-solving power of the approach. The book was
writtenwiththepurposeofawakingthecuriosityofunlikelyreaders,andinastyle
that,wehoped,wouldmakethereadingexperiencemorelikefollowinganunfolding
storyratherthanscientificreading.Likewise,whenitcametocompilingalistofrefer-
ences,wewentforaquiteradicalversionofOccam’srazor.Inhindsight,ourfirst
bookonBayesianscientificcomputingreflectedourenthusiasmforthecontentand
our desire to make it easy for newcomers to look into it, preemptively answering
many of the questions that the reader may have had and would not dare to ask, in
fearofsoundingnaïve.Overtheyears,astheideaofcombiningclassicalnumerical
analysisandBayesianinferencebecamemoremainstream,thequestionsthatpeople
askedbecamedeeper,andwestartedtofeelaboutourbookthesamewayaswedid
aboutthebell-bottompantsandthemullethaircutsofouryoungeryears.
vii
viii Preface
In the last 15 years, pairing Bayesian inference and scientific computing has
become more natural, also thanks to the tremendous growth of uncertainty quan-
tification, an area of applied mathematics where Bayesian inference and scientific
computing fit very naturally. Over the same period of time, Bayesian scientific
computinghasadvancedalot,andwebegantofeelthatitwastimetoupdatethe2007
bookandwriteamoregrown-upversionthatwouldincludethenewdevelopmentsin
thefield.Thatwouldalsogiveusachancetorevisitsomeofthetopicsthatwe,inour
initialenthusiasm,hadsometimestreatedinaverycavalierway.Amongthetopics
that were either missing, or not systematically developed in the 2007 monograph,
are hierarchical models and sparsity promotion priors. Sparse solutions of inverse
problems,andmethodsforcomputingthem,havebeenthetopicofactiveresearch
inthelast15years,inparticular,followingthepopularityof(cid:2) andtotalvariation
1
regularization.Sincesparsityispartoftheaprioribeliefabouttheunknown,itisvery
naturaltoexpressitinBayesianterms.Thesystematicanalysisofsparsity-promoting
priorscarriedoutinthelast15yearswithinthelargerfamilyofhierarchicalpriors
has shed light on how to characterize sparsity in probabilistic terms. Hierarchical
priorsandtheircomputationallyfriendlyformulations,examinedatdifferentlevels
and from various points of view over several chapters, are prominently featured
inthismonograph,andtheconnectionbetweencertainchoicesofhyperpriorsand
Tikhonovregularizationisalsoestablished.Anothermajortopicthatwascompletely
omittedfrom[18]issequentialBayesianmethods,whichnowoccupythelasttwo
chapters of the book. Particle filters, Kalman filters and Ensemble Kalman filters
combine,inanelegantandorganicway,manyoftheBayesianscientificcomputing
topics introduced in the earlier chapters. The evolution and observation equations
can be interpreted as prior and likelihood, and the way in which the posterior is
updatedfromonetimesteptothenextresemblesthenaturallearningprocess.Like-
wise,samplingmethodshaveamoreprominentposition,commensuratewiththeir
keyroleinmanyBayesianscientificcomputingtasks.TherationalforMonteCarlo
integrationinhighdimensionsisanexcellentexampleoftheneedforsamplingfrom
adistribution,andisusedhereasagatewaytosomeofthemostpopularsampling
schemes, including the intuitive importance sampling, the classical Gibbs sampler
andMetropolis–Hastingsalgorithms,allthewaytothecleverpreconditionedCrank–
Nicolsonsampler.
Othersignificantwaysinwhichthisbookdiffersfrom[18]areinthelevelofdetail
and notations. During the years, we have experienced that some of our shorthand
notations in the previous book led to confusion and potential ambiguities, and in
order to avoid that, some notations in the new book are arguably a bit heavier but
lesspronetomisinterpretations.Wehavetriedtoretainsomeofthelightnessofthe
previous book, and we hope that the style manages to elucidate the origins of the
ideasandmakesthebookarewardingreadingexperience,andnotjustasourceof
reference.
Onethingbetweenthetwobookprojectsisinvariant:Liketheoldbook,thenew
oneisalsotheresultofadialoguewithourstudents.Thenewbookcorrespondsto
thecurriculumofthegraduatecourseBayesianScientificComputingtaughtoverthe
yearsatCaseWesternReserveUniversity,andmuchofthechangesandadditionsare
Preface ix
aresultofthequestionsandcommentsofthesmartandinquisitiveCWRUstudents
fromalloverthecampus.Moreover,wehavehadtheopportunitytoteachpartsofthis
material in several other universities and research institutes, including Instituto de
MatemáticaPuraeAplicada(IMPA)inRiodeJaneiro,Brazilin2011;Universityof
Copenhagen,Denmarkin2014;UniversityofNaples“FedericoII”inNaples,Italy
in2015;BasqueCenterforAppliedMathematics(BCAM)inBilbao,Spainin2015;
UniversityofWarwickinWarwick,UKin2015;UniversityofRome“LaSapienza”in
Rome,Italyin2015;MOXatMilanPolytechnicUniversityinMilan,Italyin2019;
Danish Technical University in Lyngby, Denmark in 2019, as well as Gran Sasso
ScienceInstitute(GSSI)inL’Aquila,Italyin2021.OurwarmestthanksgotoJorge
Zubelli, Sami Brandt, Gerardo Toraldo, Salvatore Cuomo, Luca Gerardo Giorda,
AndrewStuart,FrancescaPitolli,BarbaraVantaggi,SimonaPerotto,PerChristian
HansenandNicolaGuglielmiformakingthesevisitspossible,andespeciallytothe
fantasticgroupsofgraduatestudents,postdocsandlocalresearchersparticipatingin
thosecourses,whoseenthusiasmconvincedusthatweshouldcontinuetodevelop
thematerialandthatrefreshingthebookwastherightthingtodo.Wearealsovery
gratefultoHowardElman,whoseinsightfulcommentsonanearfinaldraftofthis
bookhelpedimprovethepresentationofsometopics.
Ifmanyofthetopicinthisbookhaveplayedacentralroleinourresearchprojects,
ourresearchinterestshavehadastronginfluenceonthestructureofthebook.We
gratefullyacknowledgepartialsupportfortheworkofDanielaCalvettibythegrants
NSF-DMS1522334,NSF-DMS1951446,SimonsFellowshipinMathematics“The
inverseproblemofmagnetoencephalography,”andofErkkiSomersalobythegrants
NSF-DMS1016183,NSF-DMS1312424andNSF-DMS1714617.
Cleveland,USA DanielaCalvetti
June2022 ErkkiSomersalo
Preface to the 2007 Book Introduction to
Bayesian Scientific Computing
Thebookofnature,accordingtoGalilei,iswritteninthelanguageofmathematics.
The nature of mathematics is being exact, and its exactness is underlined by the
formalismusedbymathematicianstowriteit.Thisformalism,characterizedbytheo-
remsandproofs,andsyncopatedwithoccasionallemmas,remarksandcorollaries,
is so deeply ingrained that mathematicians feel uncomfortable when the pattern is
broken, to the point of giving the impression that the attitude of mathematicians
towards the way mathematics should be written is almost moralistic. There is a
definitionoftenquoted,“Amathematicianisapersonwhoprovestheorems,”anda
similar,morealchemisticone,creditedtoPaulErdös,butmorelikelygoingbackto
AlfrédRényi,statingthat“Amathematicianisamachinethattransformscoffeeinto
theorems.”1 Therefore it seems to be the form, not the content, that characterizes
mathematics,similarlytowhathappensinanyformalmoralisticcodewhereinform
takesprecedenceovercontent.
This book is deliberately written in a very different manner, without a single
theoremorproof.Sincemoralityhasitssubjectivecomponent,toparaphraseManuel
VasquezMontalban,wecouldcallitTenImmoralMathematicalRecipes.2Doesthe
lackoftheoremsandproofsmeanthatthebookismoreinaccuratethantraditional
booksofmathematics?Orisitpossiblyjustasignoflackofcoffee?Thisisourfirst
openquestion.
Exactness is an interesting concept. Italo Calvino, in his Lezioni Americane,3
listedexactnessasoneofthevaluesthathewouldhavewantedtotakealongtothe
twenty-firstcentury.Exactness,forCalvino,meantpreciselinguisticexpression,but
in a particular sense. To explain what he meant by exactness, he used a surprising
exampleofexactexpression:thepoetryofGiacomoLeopardi,withallitsambiguities
andsuggestiveimages.AccordingtoCalvino,whenobsessedwithaformallanguage
1That said, academic mathematics departments should invest on high-quality coffee beans and
decentcoffeemakers,inhopeofbettertheorems.AsPaulTurán,athirdHungarianmathematician,
remarked,“weakcoffeeisfitonlytoproducelemmas.”
2M.V.Montalban:Ricetteimmorali(orig.Lasrecetasinmorales,1981),Feltrinelli,1992.
3I.Calvino:LezioniAmericane,OscarMondadori,1988.
xi
