Table Of ContentLearning Probabilistic Graphical
Models in R
Familiarize yourself with probabilistic graphical models
through real-world problems and illustrative code
examples in R
David Bellot
BIRMINGHAM - MUMBAI
Learning Probabilistic Graphical Models in R
Copyright © 2016 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: April 2016
Production reference: 1270416
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78439-205-5
www.packtpub.com
Credits
Author Project Coordinator
David Bellot Kinjal Bari
Reviewers Proofreader
Mzabalazo Z. Ngwenya Safis Editing
Prabhanjan Tattar
Indexer
Mariammal Chettiyar
Acquisition Editor
Divya Poojari
Graphics
Abhinash Sahu
Content Development Editor
Trusha Shriyan
Production Coordinator
Nilesh Mohite
Technical Editor
Vivek Arora
Cover Work
Nilesh Mohite
Copy Editor
Stephen Copestake
About the Author
David Bellot is a PhD graduate in computer science from INRIA, France, with a
focus on Bayesian machine learning. He was a postdoctoral fellow at the University
of California, Berkeley, and worked for companies such as Intel, Orange, and
Barclays Bank. He currently works in the financial industry, where he develops
financial market prediction algorithms using machine learning. He is also a
contributor to open source projects such as the Boost C++ library.
About the Reviewers
Mzabalazo Z. Ngwenya holds a postgraduate degree in mathematical statistics
from the University of Cape Town. He has worked extensively in the field of
statistical consulting and has considerable experience working with R. Areas of
interest to him are primarily centered around statistical computing. Previously,
he has been involved in reviewing the following Packt Publishing titles: Learning
RStudio for R Statistical Computing, Mark P.J. van der Loo and Edwin de Jonge;
R Statistical Application Development by Example Beginner's Guide, Prabhanjan
Narayanachar Tattar; Machine Learning with R, Brett Lantz; R Graph Essentials, David
Alexandra Lillis; R Object-oriented Programming, Kelly Black; Mastering Scientific
Computing with R, Paul Gerrard and Radia Johnson; and Mastering Data Analysis with
R, Gergely Darócz.
Prabhanjan Tattar is currently working as a senior data scientist at Fractal Analytics,
Inc. He has 8 years of experience as a statistical analyst. Survival analysis and statistical
inference are his main areas of research/interest. He has published several research
papers in peer-reviewed journals and authored two books on R: R Statistical Application
Development by Example, Packt Publishing; and A Course in Statistics with R, Wiley. The
R packages gpk, RSADBE, and ACSWR are also maintained by him.
www.PacktPub.com
eBooks, discount offers, and more
Did you know that Packt offers eBook versions of every book published, with PDF
and ePub files available? You can upgrade to the eBook version at www.PacktPub.
com and as a print book customer, you are entitled to a discount on the eBook copy.
Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign
up for a range of free newsletters and receive exclusive discounts and offers on Packt
books and eBooks.
TM
https://www2.packtpub.com/books/subscription/packtlib
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital
book library. Here, you can search, access, and read Packt's entire library of books.
Why subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print, and bookmark content
• On demand and accessible via a web browser
Table of Contents
Preface v
Chapter 1: Probabilistic Reasoning 1
Machine learning 4
Representing uncertainty with probabilities 5
Beliefs and uncertainty as probabilities 6
Conditional probability 7
Probability calculus and random variables 7
Sample space, events, and probability 7
Random variables and probability calculus 8
Joint probability distributions 10
Bayes' rule 11
Interpreting the Bayes' formula 13
A first example of Bayes' rule 13
A first example of Bayes' rule in R 16
Probabilistic graphical models 20
Probabilistic models 20
Graphs and conditional independence 21
Factorizing a distribution 23
Directed models 24
Undirected models 25
Examples and applications 26
Summary 31
Chapter 2: Exact Inference 33
Building graphical models 35
Types of random variable 36
Building graphs 37
Probabilistic expert system 37
Basic structures in probabilistic graphical models 40
Variable elimination 44
[ i ]
Table of Contents
Sum-product and belief updates 47
The junction tree algorithm 51
Examples of probabilistic graphical models 62
The sprinkler example 62
The medical expert system 63
Models with more than two layers 64
Tree structure 66
Summary 68
Chapter 3: Learning Parameters 69
Introduction 71
Learning by inference 75
Maximum likelihood 79
How are empirical and model distribution related? 79
The ML algorithm and its implementation in R 82
Application 86
Learning with hidden variables – the
EM algorithm 88
Latent variables 89
Principles of the EM algorithm 90
Derivation of the EM algorithm 91
Applying EM to graphical models 93
Summary 94
Chapter 4: Bayesian Modeling – Basic Models 97
The Naive Bayes model 98
Representation 100
Learning the Naive Bayes model 101
Bayesian Naive Bayes 104
Beta-Binomial 106
The prior distribution 111
The posterior distribution with the conjugacy property 112
Which values should we choose for the Beta parameters? 113
The Gaussian mixture model 115
Definition 116
Summary 122
Chapter 5: Approximate Inference 125
Sampling from a distribution 126
Basic sampling algorithms 129
Standard distributions 129
Rejection sampling 133
An implementation in R 135
[ ii ]
Table of Contents
Importance sampling 142
An implementation in R 144
Markov Chain Monte-Carlo 152
General idea of the method 153
The Metropolis-Hastings algorithm 154
MCMC for probabilistic graphical models in R 162
Installing Stan and RStan 163
A simple example in RStan 164
Summary 165
Chapter 6: Bayesian Modeling – Linear Models 167
Linear regression 169
Estimating the parameters 170
Bayesian linear models 176
Over-fitting a model 176
Graphical model of a linear model 179
Posterior distribution 181
Implementation in R 184
A stable implementation 188
More packages in R 194
Summary 195
Chapter 7: Probabilistic Mixture Models 197
Mixture models 198
EM for mixture models 200
Mixture of Bernoulli 207
Mixture of experts 210
Latent Dirichlet Allocation 215
The LDA model 216
Variational inference 220
Examples 221
Summary 224
Appendix 227
References 227
Books on the Bayesian theory 227
Books on machine learning 228
Papers 228
Index 229
[ iii ]
