Bayesian Modeling and Computation in Python

421 Pages·2021·40.681 MB·English

by  Osvaldo A. Martin, Ravin Kumar, Junpeng Lao

Bayesian Modeling and
Computation in Python
CHAPMAN & HALL/CRC
Texts in Statistical Science Series
Joseph K. Blitzstein, Harvard University, USA
Julian J. Faraway, University of Bath, UK
Martin Tanner, Northwestern University, USA
Jim Zidek, University of British Columbia, Canada
Recently Published Titles
A First Course in Linear Model Theory, Second Edition
Nalini Ravishanker, Zhiyi Chi, Dipak K. Dey
Foundations of Statistics for Data Scientists
With R and Python
Alan Agresti and Maria Kateri
Fundamentals of Causal Inference
With R
Babette A. Brumback
Sampling
Design and Analysis, Third Edition
Sharon L. Lohr
Theory of Statistical Inference
Anthony Almudevar
Probability, Statistics, and Data
A Fresh Approach Using R
Darrin Speegle and Brain Claire
Bayesian Modeling and Computation in Python
Osvaldo A. Martin, Ravin Kumar and Junpeng Lao
Bayes Rules!
An Introduction to Applied Bayesian Modeling
Alicia Johnson, Miles Ott and Mine Dogucu
Stochastic Processes with R
An Introduction
Olga Korosteleva
Introduction to Design and Analysis of Scientific Studies
Nathan Taback
Practical Time Series Analysis for Data Science
Wayne A. Woodward, Bivin Philip Sadler and Stephen Robertson
For more information about this series, please visit: https://www.routledge.com/
Chapman--HallCRC-Texts-in-Statistical-Science/book-series/CHTEXSTASCI
Bayesian Modeling and
Computation in Python
Osvaldo A. Martin, Ravin Kumar and Junpeng Lao
First edition published 2022
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
CRC Press is an imprint of Taylor & Francis Group, LLC
© 2022 Taylor & Francis Group, LLC
Reasonable efforts have been made to publish reliable data and information, but the author and publisher
cannot assume responsibility for the validity of all materials or the consequences of their use. The authors
and publishers have attempted to trace the copyright holders of all material reproduced in this publica-
tion and apologize to copyright holders if permission to publish in this form has not been obtained. If
any copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, trans-
mitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter
invented, including photocopying, microfilming, and recording, or in any information storage or retrieval
system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, access www.copyright.com or
contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-
8400. For works that are not available on CCC please contact [e-mail]
Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used
only for identification and explanation without intent to infringe.
ISBN: 978-0-367-89436-8 (hbk)
ISBN: 978-1-032-18029-8 (pbk)
ISBN: 978-1-003-01916-9 (ebk)
DOI: 10.1201/9781003019169
Publisher’s note: This book has been prepared from camera-ready copy provided by the authors.
Typeset in CMR10
by KnowledgeWorks Global Ltd.
To Romina and Abril, and their caring love. To everyone who helped me
get here.
Osvaldo Martin
To the educators, both formal and informal, who unconditionally shared
their knowledge and wisdom making me who I am today. In order of ap-
pearance in my life, Tim Pegg, Mr. Michael Collins, Mrs. Sara LaFram-
boise Saadeh, Professor Mehrdad Haghi, Professor Winny Dong, Profes-
sor Dixon Davis, Jason Errington, Chris Lopez, John Norman, Professor
Ananth Krishnamurthy, and Kurt Campbell.
Thank you
Ravin Kumar
To Yuli.
Junpeng Lao

Contents
Foreword xiii
Preface xv
Symbols xxi
1 Bayesian Inference 1
1.1 Bayesian Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Bayesian Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Bayesian Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 A DIY Sampler, Do Not Try This at Home . . . . . . . . . . . . . . . . . . 6
1.3 Say Yes to Automating Inference, Say No to Automated Model Building . 10
1.4 A Few Options to Quantify Your Prior Information . . . . . . . . . . . . . 13
1.4.1 Conjugate Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.4.2 Objective Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.4.3 Maximum Entropy Priors . . . . . . . . . . . . . . . . . . . . . . . . 20
1.4.4 Weakly Informative Priors and Regularization Priors . . . . . . . . . 23
1.4.5 Using Prior Predictive Distributions to Assess Priors . . . . . . . . . 25
1.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2 Exploratory Analysis of Bayesian Models 31
2.1 There is Life After Inference, and Before Too! . . . . . . . . . . . . . . . . 31
2.2 Understanding Your Assumptions . . . . . . . . . . . . . . . . . . . . . . . 32
2.3 Understanding Your Predictions . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4 Diagnosing Numerical Inference . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4.1 Effective Sample Size. . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.4.2 Potential Scale Reduction Factor Rˆ . . . . . . . . . . . . . . . . . . . 43
2.4.3 Monte Carlo Standard Error . . . . . . . . . . . . . . . . . . . . . . 43
2.4.4 Trace Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.4.5 Autocorrelation Plots . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.4.6 Rank Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.4.7 Divergences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.4.8 Sampler Parameters and Other Diagnostics . . . . . . . . . . . . . . 51
2.5 Model Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.5.1 Cross-validation and LOO . . . . . . . . . . . . . . . . . . . . . . . . 53
2.5.2 Expected Log Predictive Density . . . . . . . . . . . . . . . . . . . . 56
2.5.3 Pareto Shape Parameter, κˆ . . . . . . . . . . . . . . . . . . . . . . . 57
2.5.4 Interpreting p loo When Pareto κˆ is Large. . . . . . . . . . . . . . . 58
2.5.5 LOO-PIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.5.6 Model Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
vii
viii Contents
3 Linear Models and Probabilistic Programming Languages 67
3.1 Comparing Two (or More) Groups . . . . . . . . . . . . . . . . . . . . . . . 67
3.1.1 Comparing Two PPLs . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.2.1 Linear Penguins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.2.2 Predictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.2.3 Centering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.3 Multiple Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.3.1 Counterfactuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.4 Generalized Linear Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.4.1 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.4.2 Classifying Penguins . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
3.4.3 Interpreting Log Odds . . . . . . . . . . . . . . . . . . . . . . . . . . 98
3.5 Picking Priors in Regression Models . . . . . . . . . . . . . . . . . . . . . . 100
3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4 Extending Linear Models 107
4.1 Transforming Covariates . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.2 Varying Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.3 Interaction Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.4 Robust Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.5 Pooling, Multilevel Models, and Mixed Effects . . . . . . . . . . . . . . . . 119
4.5.1 Unpooled Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.5.2 Pooled Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.5.3 Mixing Group and Common Parameters . . . . . . . . . . . . . . . . 125
4.6 Hierarchical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
4.6.1 Posterior Geometry Matters . . . . . . . . . . . . . . . . . . . . . . . 131
4.6.2 Predictions at Multiple Levels. . . . . . . . . . . . . . . . . . . . . . 137
4.6.3 Priors for Multilevel Models . . . . . . . . . . . . . . . . . . . . . . . 140
4.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5 Splines 145
5.1 Polynomial Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5.2 Expanding the Feature Space . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.3 Introducing Splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
5.4 Building the Design Matrix using Patsy . . . . . . . . . . . . . . . . . . . . 152
5.5 Fitting Splines in PyMC3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
5.6 Choosing Knots and Prior for Splines . . . . . . . . . . . . . . . . . . . . . 157
5.6.1 Regularizing Prior for Splines . . . . . . . . . . . . . . . . . . . . . . 158
5.7 Modeling CO Uptake with Splines . . . . . . . . . . . . . . . . . . . . . . 160
2
5.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
6 Time Series 169
6.1 An Overview of Time Series Problems . . . . . . . . . . . . . . . . . . . . . 169
6.2 Time Series Analysis as a Regression Problem . . . . . . . . . . . . . . . . 170
6.2.1 Design Matrices for Time Series . . . . . . . . . . . . . . . . . . . . 175
6.2.2 Basis Functions and Generalized Additive Model . . . . . . . . . . . 178
6.3 Autoregressive Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
6.3.1 Latent AR Process and Smoothing . . . . . . . . . . . . . . . . . . . 187
6.3.2 (S)AR(I)MA(X) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
6.4 State Space Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Contents ix
6.4.1 Linear Gaussian State Space Models and Kalman filter. . . . . . . . 195
6.4.2 ARIMA, Expressed as a State Space Model . . . . . . . . . . . . . . 198
6.4.3 Bayesian Structural Time Series . . . . . . . . . . . . . . . . . . . . 202
6.5 Other Time Series Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
6.6 Model Criticism and Choosing Priors . . . . . . . . . . . . . . . . . . . . . 206
6.6.1 Priors for Time Series Models . . . . . . . . . . . . . . . . . . . . . . 207
6.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
7 Bayesian Additive Regression Trees 213
7.1 Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
7.1.1 Ensembles of Decision Trees . . . . . . . . . . . . . . . . . . . . . . . 216
7.2 The BART Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
7.3 Priors for BART . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
7.3.1 Prior Independence. . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
7.3.2 Prior for the Tree Structure T . . . . . . . . . . . . . . . . . . . . . 218
j
7.3.3 Prior for the Leaf Values µ and Number of Trees m. . . . . . . . . 218
ij
7.4 Fitting Bayesian Additive Regression Trees . . . . . . . . . . . . . . . . . . 218
7.5 BART Bikes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
7.6 Generalized BART Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
7.7 Interpretability of BARTs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.7.1 Partial Dependence Plots . . . . . . . . . . . . . . . . . . . . . . . . 223
7.7.2 Individual Conditional Expectation. . . . . . . . . . . . . . . . . . . 225
7.8 Variable Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
7.9 Priors for BART in PyMC3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
7.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
8 Approximate Bayesian Computation 233
8.1 Life Beyond Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
8.2 Approximating the Approximated Posterior . . . . . . . . . . . . . . . . . . 235
8.3 Fitting a Gaussian the ABC-way . . . . . . . . . . . . . . . . . . . . . . . . 235
8.4 Choosing the Distance Function, (cid:15) and the Summary Statistics . . . . . . . 238
8.4.1 Choosing the Distance . . . . . . . . . . . . . . . . . . . . . . . . . . 239
8.4.2 Choosing (cid:15) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
8.4.3 Choosing Summary Statistics . . . . . . . . . . . . . . . . . . . . . . 241
8.5 g-and-k Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
8.6 Approximating Moving Averages . . . . . . . . . . . . . . . . . . . . . . . . 247
8.7 Model Comparison in the ABC Context . . . . . . . . . . . . . . . . . . . . 249
8.7.1 Marginal Likelihood and LOO . . . . . . . . . . . . . . . . . . . . . 250
8.7.2 Model Choice via Random Forest . . . . . . . . . . . . . . . . . . . . 254
8.7.3 Model Choice for MA Model . . . . . . . . . . . . . . . . . . . . . . 256
8.8 Choosing Priors for ABC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
8.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
9 End to End Bayesian Workflows 261
9.1 Workflows, Contexts, and Questions . . . . . . . . . . . . . . . . . . . . . . 261
9.1.1 Applied Example: Airlines Flight Delays Problem. . . . . . . . . . . 263
9.2 Getting Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
9.2.1 Sample Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
9.2.2 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
9.2.3 Observational Studies . . . . . . . . . . . . . . . . . . . . . . . . . . 265
9.2.4 Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265