Statistical Machine Learning: A Unified Framework (Chapman & Hall/CRC Texts in Statistical Science)

525 Pages·2020·6.928 MB·English

by  Richard Golden

Statistical Machine
Learning
CHAPMAN & HALL/CRC
Texts in Statistical Science Series
Joseph K. Blitzstein, Harvard University, USA
Julian J. Faraway, University of Bath, UK
Martin Tanner, Northwestern University, USA
Jim Zidek, University of British Columbia, Canada
Recently Published Titles
Sampling
Design and Analysis, Second Edition
Sharon L. Lohr
The Analysis of Time Series
An Introductionwith R, Seventh Edition
Chris ChatfieldandHaipeng Xing
Time Series
A Data Analysis Approach Using R
Robert H. Shumway and David S. Stoffer
Practical Multivariate Analysis, Sixth Edition
Abdelmonem Afifi, Susanne May, Robin A. Donatello, and Virginia A. Clark
Time Series: A First Course with Bootstrap Starter
Tucker S. McElroyandDimitris N. Politis
Probability and Bayesian Modeling
Jim Albert and Jingchen Hu
Surrogates
Gaussian Process Modeling,Design, andOptimization for the Applied Sciences
Robert B.Gramacy
Statistical Analysis of Financial Data
With Examples in R
James Gentle
Statistical Rethinking
A Bayesian Course with Examples in R and STAN, Second Edition
Richard McElreath
Randomization, Bootstrap and Monte Carlo Methods in Biology
Fourth Edition
Bryan F. J. Manly, Jorje A. Navarro Alberto
Statistical Machine Learning
A Unified Framework
Richard M. Golden
For more information about this series, please visit: https://www.crcpress.com/Chapman--
HallCRC-Texts-in-Statistical-Science/book-series/CHTEXSTASCI
Statistical Machine
Learning
A Unified Framework
Richard M. Golden
MATLAB(cid:13)R is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does
notwarranttheaccuracyofthetextorexercisesinthisbook.Thisbook’suseordiscussionofMATLAB(cid:13)R
softwareorrelatedproductsdoesnotconstituteendorsementorsponsorshipbyTheMathWorksofapar-
ticularpedagogicalapproachorparticularuseoftheMATLAB(cid:13)R software.
Firsteditionpublished2020
byCRCPress
6000BrokenSoundParkwayNW,Suite300,BocaRaton,FL33487-2742
andbyCRCPress
2ParkSquare,MiltonPark,Abingdon,Oxon,OX144RN
(cid:13)c 2020RichardM.Golden
CRCPressisanimprintofTaylor&FrancisGroup,LLC
Reasonableeffortshavebeenmadetopublishreliabledataandinformation,buttheauthorandpublisher
cannotassumeresponsibilityforthevalidityofallmaterialsortheconsequencesoftheiruse.Theauthors
andpublishershaveattemptedtotracethecopyrightholdersofallmaterialreproducedinthispublication
and apologize to copyright holders if permission to publish in this form has not been obtained. If any
copyrightmaterialhasnotbeenacknowledgedpleasewriteandletusknowsowemayrectifyinanyfuture
reprint.
ExceptaspermittedunderU.S.CopyrightLaw,nopartofthisbookmaybereprinted,reproduced,trans-
mitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter
invented, including photocopying, microfilming, and recording, or in any information storage or retrieval
system,withoutwrittenpermissionfromthepublishers.
For permission to photocopy or use material electronically from this work, access www.copyright.com or
contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-
8400.[e-mail]
Trademarknotice:Productorcorporatenamesmaybetrademarksorregisteredtrademarks,andareused
onlyforidentificationandexplanationwithoutintenttoinfringe.
Library of Congress Cataloging-in-Publication Data
Names:Golden,RichardM.,author.
Title:Statisticalmachinelearning:aunifiedframework/RichardM.Golden.
Description:Firstedition.|BocaRaton,FL:CRCPress,2020.|Includes
bibliographicalreferencesandindex.
Identifiers:LCCN2020004853|ISBN9781138484696(hardback)|ISBN9780367494223
(paperback)|ISBN9781351051507(ebook)
Subjects:LCSH:Machinelearning--Statisticalmethods.|Computeralgorithms.
Classification:LCCQ325.5.G652020|DDC006.3/10727--dc23
LCrecordavailableathttps://lccn.loc.gov/2020004853
ISBN:978-1-138-48469-6(hbk)
ISBN:978-1-351-05150-7(ebk)
TypesetinCMR
byNovaTechsetPrivateLimited,Bengaluru&Chennai,India
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Symbols xi
Preface xiii
I Inference and Learning Machines 1
1 A Statistical Machine Learning Framework 3
1.1 Statistical Machine Learning: A Brief Overview . . . . . . . . . . . . . . . 3
1.2 Machine Learning Environments . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 Feature Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1.1 Feature Vector Representations. . . . . . . . . . . . . . . . 4
1.2.1.2 The Curse of Feature Vector Dimensionality . . . . . . . . 6
1.2.1.3 Feature Vector Engineering Methods. . . . . . . . . . . . . 7
1.2.2 Stationary Statistical Environments . . . . . . . . . . . . . . . . . . 7
1.2.3 Strategies for Teaching Machine Learning Algorithms . . . . . . . . 8
1.2.4 Prior Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.4.1 Feature Maps Specify Relevant Statistical Regularities . . . 9
1.2.4.2 Similar Inputs Predict Similar Responses . . . . . . . . . . 10
1.2.4.3 Only a Small Number of Model Parameters Are Relevant . 10
1.2.4.4 Different Feature Detectors Share Parameters. . . . . . . . 10
1.3 Empirical Risk Minimization Framework . . . . . . . . . . . . . . . . . . . 12
1.3.1 ANN Graphical Notation . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.2 Risk Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.3.3 Regularization Terms . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.3.4 Optimization Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.3.4.1 Batch Gradient Descent Learning . . . . . . . . . . . . . . 16
1.3.4.2 Adaptive Gradient Descent Learning. . . . . . . . . . . . . 18
1.4 Theory-Based System Analysis and Design . . . . . . . . . . . . . . . . . . 19
1.4.1 Stage 1: System Specification . . . . . . . . . . . . . . . . . . . . . . 19
1.4.2 Stage 2: Theoretical Analyses . . . . . . . . . . . . . . . . . . . . . . 19
1.4.3 Stage 3: Physical Implementation . . . . . . . . . . . . . . . . . . . . 20
1.4.4 Stage 4: System Behavior Evaluation . . . . . . . . . . . . . . . . . . 20
1.5 Supervised Learning Machines . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.5.1 Discrepancy Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.5.2 Basis Functions and Hidden Units . . . . . . . . . . . . . . . . . . . 26
1.5.3 Recurrent Neural Networks (RNNs) . . . . . . . . . . . . . . . . . . 32
1.6 Unsupervised Learning Machines . . . . . . . . . . . . . . . . . . . . . . . . 36
1.7 Reinforcement Learning Machines . . . . . . . . . . . . . . . . . . . . . . . 49
1.7.1 Reinforcement Learning Overview . . . . . . . . . . . . . . . . . . . 51
1.7.2 Value Function Reinforcement Passive Learning . . . . . . . . . . . . 55
1.7.3 Policy Gradient Reinforcement Reactive Learning. . . . . . . . . . . 57
1.8 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
v
vi Contents
2 Set Theory for Concept Modeling 65
2.1 Set Theory and Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.2 Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
2.2.1 Types of Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
2.2.2 Directed Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
2.2.3 Undirected Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
2.3 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
2.4 Metric Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
2.5 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3 Formal Machine Learning Algorithms 83
3.1 Environment Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.1.1 Time Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.1.2 Event Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.2 Machine Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.2.1 Dynamical Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.2.2 Iterated Maps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.2.3 Vector Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.3 Intelligent Machine Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
3.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
II Deterministic Learning Machines 97
4 Linear Algebra for Machine Learning 99
4.1 Matrix Notation and Operators . . . . . . . . . . . . . . . . . . . . . . . . 99
4.2 Linear Subspace Projection Theorems . . . . . . . . . . . . . . . . . . . . . 106
4.3 Linear System Solution Theorems . . . . . . . . . . . . . . . . . . . . . . . 112
4.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5 Matrix Calculus for Machine Learning 119
5.1 Convergence and Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.1.1 Deterministic Convergence . . . . . . . . . . . . . . . . . . . . . . . 119
5.1.2 Continuous Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.2 Vector Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.2.1 Vector Derivative Definitions . . . . . . . . . . . . . . . . . . . . . . 129
5.2.2 Theorems for Computing Matrix Derivatives . . . . . . . . . . . . . 130
5.2.3 Useful Derivative Calculations for Deep Learning . . . . . . . . . . . 133
5.2.4 Gradient Backpropagation for Deep Learning . . . . . . . . . . . . . 138
5.3 Objective Function Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 142
5.3.1 Taylor Series Expansions . . . . . . . . . . . . . . . . . . . . . . . . 142
5.3.2 Gradient Descent Type Algorithms . . . . . . . . . . . . . . . . . . . 143
5.3.3 Critical Point Classification . . . . . . . . . . . . . . . . . . . . . . . 146
5.3.3.1 Identifying Critical Points. . . . . . . . . . . . . . . . . . . 146
5.3.3.2 Identifying Flat Regions . . . . . . . . . . . . . . . . . . . . 147
5.3.3.3 Identifying Local Minimizers . . . . . . . . . . . . . . . . . 148
5.3.3.4 Identifying Saddlepoints . . . . . . . . . . . . . . . . . . . . 150
5.3.3.5 Identifying Global Minimizers . . . . . . . . . . . . . . . . 151
5.3.4 Lagrange Multipliers . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
5.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Contents vii
6 Convergence of Time-Invariant Dynamical Systems 169
6.1 Dynamical System Existence Theorems . . . . . . . . . . . . . . . . . . . . 170
6.2 Invariant Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
6.3 Lyapunov Convergence Theorems . . . . . . . . . . . . . . . . . . . . . . . 175
6.3.1 Lyapunov Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.3.2 Invariant Set Theorems . . . . . . . . . . . . . . . . . . . . . . . . . 176
6.3.2.1 Finite State Space Convergence Analyses . . . . . . . . . . 176
6.3.2.2 Continuous State Space Convergence Analyses . . . . . . . 178
6.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7 Batch Learning Algorithm Convergence 187
7.1 Search Direction and Stepsize Choices . . . . . . . . . . . . . . . . . . . . . 188
7.1.1 Search Direction Selection . . . . . . . . . . . . . . . . . . . . . . . . 188
7.1.2 Stepsize Selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.2 Descent Algorithm Convergence Analysis . . . . . . . . . . . . . . . . . . . 196
7.3 Descent Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
7.3.1 Gradient and Steepest Descent . . . . . . . . . . . . . . . . . . . . . 203
7.3.2 Newton-Type Descent . . . . . . . . . . . . . . . . . . . . . . . . . . 204
7.3.2.1 Newton-Raphson Algorithm . . . . . . . . . . . . . . . . . 204
7.3.2.2 Levenberg-Marquardt Algorithm . . . . . . . . . . . . . . . 205
7.3.3 L-BFGS and Conjugate Gradient Descent Methods . . . . . . . . . . 206
7.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
III Stochastic Learning Machines 211
8 Random Vectors and Random Functions 213
8.1 Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
8.1.1 Sigma-Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
8.1.2 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
8.2 Random Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
8.2.1 Measurable Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 218
8.2.2 Discrete, Continuous, and Mixed Random Vectors . . . . . . . . . . 222
8.3 Existence of the Radon-Nikody´m Density (Optional Reading) . . . . . . . 226
8.3.1 Lebesgue Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
8.3.2 The Radon-Nikody´m Probability Density Function . . . . . . . . . . 228
8.3.3 Vector Support Specification Measures . . . . . . . . . . . . . . . . . 229
8.4 Expectation Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
8.4.1 Random Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
8.4.2 Expectations of Random Functions . . . . . . . . . . . . . . . . . . . 234
8.4.3 Conditional Expectation and Independence . . . . . . . . . . . . . . 236
8.5 Concentration Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
8.6 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
9 Stochastic Sequences 243
9.1 Types of Stochastic Sequences . . . . . . . . . . . . . . . . . . . . . . . . . 244
9.2 Partially Observable Stochastic Sequences . . . . . . . . . . . . . . . . . . 246
9.3 Stochastic Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
9.3.1 Convergence with Probability One . . . . . . . . . . . . . . . . . . . 250
9.3.2 Convergence in Mean Square . . . . . . . . . . . . . . . . . . . . . . 253
9.3.3 Convergence in Probability . . . . . . . . . . . . . . . . . . . . . . . 254
9.3.4 Convergence in Distribution . . . . . . . . . . . . . . . . . . . . . . . 255
9.3.5 Stochastic Convergence Relationships . . . . . . . . . . . . . . . . . 257
viii Contents
9.4 Combining and Transforming Stochastic Sequences . . . . . . . . . . . . . 259
9.5 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
10 Probability Models of Data Generation 263
10.1 Learnability of Probability Models . . . . . . . . . . . . . . . . . . . . . . . 263
10.1.1 Correctly Specified and Misspecified Models . . . . . . . . . . . . . . 263
10.1.2 Smooth Parametric Probability Models . . . . . . . . . . . . . . . . 267
10.1.3 Local Probability Models . . . . . . . . . . . . . . . . . . . . . . . . 268
10.1.4 Missing-Data Probability Models . . . . . . . . . . . . . . . . . . . . 268
10.2 Gibbs Probability Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
10.3 Bayesian Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
10.3.1 Factoring a Chain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
10.3.2 Bayesian Network Factorization . . . . . . . . . . . . . . . . . . . . . 277
10.4 Markov Random Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
10.4.1 The Markov Random Field Concept . . . . . . . . . . . . . . . . . . 283
10.4.2 MRF Interpretation of Gibbs Distributions . . . . . . . . . . . . . . 286
10.5 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
11 Monte Carlo Markov Chain Algorithm Convergence 299
11.1 Monte Carlo Markov Chain (MCMC) Algorithms . . . . . . . . . . . . . . 300
11.1.1 Countably Infinite First-Order Chains on Finite State Spaces . . . . 300
11.1.2 Convergence Analysis of Monte Carlo Markov Chains . . . . . . . . 303
11.1.3 Hybrid MCMC Algorithms . . . . . . . . . . . . . . . . . . . . . . . 306
11.1.4 Finding Global Minimizers and Computing Expectations . . . . . . 307
11.1.5 Assessing and Improving MCMC Convergence Performance . . . . . 310
11.1.5.1 Assessing Convergence for Expectation Approximation . . 310
11.1.5.2 MCMC Convergence Challenges and Heuristics . . . . . . . 311
11.2 MCMC Metropolis-Hastings (MH) Algorithms . . . . . . . . . . . . . . . . 313
11.2.1 Metropolis-Hastings Algorithm Definition . . . . . . . . . . . . . . . 313
11.2.2 Convergence Analysis of Metropolis-Hastings Algorithms . . . . . . 316
11.2.3 Important Special Cases of the Metropolis-Hastings Algorithm . . . 317
11.2.4 Metropolis-Hastings Machine Learning Applications . . . . . . . . . 319
11.3 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
12 Adaptive Learning Algorithm Convergence 325
12.1 Stochastic Approximation (SA) Theory . . . . . . . . . . . . . . . . . . . . 326
12.1.1 Passive versus Reactive Statistical Environments . . . . . . . . . . . 326
12.1.1.1 Passive Learning Environments . . . . . . . . . . . . . . . . 326
12.1.1.2 Reactive Learning Environments . . . . . . . . . . . . . . . 326
12.1.2 Average Downward Descent . . . . . . . . . . . . . . . . . . . . . . . 327
12.1.3 Annealing Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
12.1.4 The Main Stochastic Approximation Theorem . . . . . . . . . . . . 329
12.1.5 Assessing Stochastic Approximation Algorithm Convergence. . . . . 335
12.2 Learning in Passive Statistical Environments Using SA . . . . . . . . . . . 337
12.2.1 Implementing Different Optimization Strategies . . . . . . . . . . . . 337
12.2.2 Improving Generalization Performance . . . . . . . . . . . . . . . . . 342
12.3 Learning in Reactive Statistical Environments Using SA . . . . . . . . . . 348
12.3.1 Policy Gradient Reinforcement Learning . . . . . . . . . . . . . . . . 348
12.3.2 Stochastic Approximation Expectation Maximization. . . . . . . . . 351
12.3.3 Markov Random Field Learning (Contrastive Divergence) . . . . . . 354
12.3.4 Generative Adversarial Network (GAN) Learning . . . . . . . . . . . 355
12.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Contents ix
IV Generalization Performance 359
13 Statistical Learning Objective Function Design 361
13.1 Empirical Risk Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
13.2 Maximum Likelihood (ML) Estimation Methods . . . . . . . . . . . . . . . 370
13.2.1 ML Estimation: Probability Theory Interpretation . . . . . . . . . . 370
13.2.2 ML Estimation: Information Theory Interpretation . . . . . . . . . . 377
13.2.2.1 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
13.2.2.2 Cross-Entropy Minimization: ML Estimation . . . . . . . . 380
13.2.3 Properties of Cross-Entropy Global Minimizers . . . . . . . . . . . . 382
13.2.4 Pseudolikelihood Empirical Risk Function . . . . . . . . . . . . . . . 385
13.2.5 Missing-Data Likelihood Empirical Risk Function. . . . . . . . . . . 386
13.3 Maximum a Posteriori (MAP) Estimation Methods . . . . . . . . . . . . . 389
13.3.1 Parameter Priors and Hyperparameters . . . . . . . . . . . . . . . . 389
13.3.2 Maximum a Posteriori (MAP) Risk Function . . . . . . . . . . . . . 391
13.3.3 Bayes Risk Interpretation of MAP Estimation. . . . . . . . . . . . . 394
13.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
14 Simulation Methods for Evaluating Generalization 399
14.1 Sampling Distribution Concepts . . . . . . . . . . . . . . . . . . . . . . . . 402
14.1.1 K-Fold Cross-Validation . . . . . . . . . . . . . . . . . . . . . . . . . 402
14.1.2 Sampling Distribution Estimation with Unlimited Data . . . . . . . 403
14.2 Bootstrap Methods for Sampling Distribution Simulation . . . . . . . . . . 405
14.2.1 Bootstrap Approximation of Sampling Distribution . . . . . . . . . . 405
14.2.2 Monte Carlo Bootstrap Sampling Distribution Estimation . . . . . . 407
14.3 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
15 Analytic Formulas for Evaluating Generalization 417
15.1 Assumptions for Asymptotic Analysis . . . . . . . . . . . . . . . . . . . . . 417
15.2 Theoretical Sampling Distribution Analysis . . . . . . . . . . . . . . . . . . 423
15.3 Confidence Regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
15.4 Hypothesis Testing for Model Comparison Decisions . . . . . . . . . . . . . 437
15.4.1 Classical Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . 438
15.4.2 Bayesian Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . 441
15.5 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
16 Model Selection and Evaluation 447
16.1 Cross-Validation Risk Model Selection Criteria . . . . . . . . . . . . . . . . 448
16.2 Bayesian Model Selection Criteria . . . . . . . . . . . . . . . . . . . . . . . 456
16.2.1 Bayesian Model Selection Problem . . . . . . . . . . . . . . . . . . . 456
16.2.2 Laplace Approximation for Multidimensional Integration . . . . . . 457
16.2.3 Bayesian Information Criteria . . . . . . . . . . . . . . . . . . . . . . 459
16.3 Model Misspecification Detection Model Selection Criteria . . . . . . . . . 464
16.3.1 Nested Models Method for Assessing Model Misspecification . . . . 464
16.3.2 Information Matrix Discrepancy Model Selection Criteria . . . . . . 465
16.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
References 473
Algorithm Index 491
Subject Index 495