Applied Deep Learning with Keras: Solve complex real-life problems with the simplicity of Keras

412 Pages·2019·24.153 MB·English

by  Bhagwat, Ritesh;Abdolahnejad, Mahla;Moocarme, Matthew

Applied Deep
Learning with Keras
Solve complex real-life problems with the simplicity
of Keras
Ritesh Bhagwat, Mahla Abdolahnejad, and Matthew Moocarme
Applied Deep Learning with Keras
Copyright © 2019 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system,
or transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of
the information presented. However, the information contained in this book is sold
without warranty, either express or implied. Neither the author, nor Packt Publishing,
and its dealers and distributors will be held liable for any damages caused or alleged to
be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
Authors: Ritesh Bhagwat, Mahla Abdolahnejad, and Matthew Moocarme
Reviewer: Elder Santos
Managing Editor: Aditya Shah
Acquisitions Editor: Kunal Sawant
Production Editor: Shantanu Zagade
Editorial Board: David Barnes, Ewan Buckingham, Simon Cox, Manasa Kumar,
Alex Mazonowicz, Douglas Paterson, Dominic Pereira, Shiny Poojary, Erol Staveley,
Ankita Thakur, Mohita Vyas and Jonathan Wray
First Published: April 2019
Production Reference: 1220419
Published by Packt Publishing Ltd.
Livery Place, 35 Livery Street
Birmingham B3 2PB, UK
ISBN: 978-1-83855-507-8
Table of Contents
Preface i
Introduction to Machine Learning with Keras 1
Introduction .................................................................................................... 2
Data Representation ..................................................................................... 3
Tables of Data ....................................................................................................... 4
Loading Data ........................................................................................................ 5
Exercise 1: Loading a Dataset from the UCI Machine
Learning Repository ............................................................................................ 7
Data Preprocessing ...................................................................................... 11
Exercise 2: Cleaning the Data ........................................................................... 12
Appropriate Representation of the Data ........................................................ 23
Exercise 3: Appropriate Representation of the Data ..................................... 24
Life Cycle of Model Creation ............................................................................. 25
Machine Learning Libraries ........................................................................ 25
scikit-learn .................................................................................................... 26
Keras .............................................................................................................. 28
Advantages of Keras .......................................................................................... 29
Disadvantages of Keras ..................................................................................... 29
More than Building Models .............................................................................. 30
Model Training ............................................................................................. 31
Classifiers and Regression Models .................................................................. 31
Classification Tasks ............................................................................................ 32
Regression Tasks ................................................................................................ 33
Training and Test Datasets ............................................................................... 34
Model Evaluation Metrics ................................................................................. 35
Exercise 4: Creating a Simple Model ................................................................ 36
Model Tuning ................................................................................................ 40
Baseline Models ................................................................................................. 40
Exercise 5: Determining a Baseline Model ...................................................... 40
Regularization .................................................................................................... 42
Cross-Validation ................................................................................................. 43
Activity 1: Adding Regularization to the Model .............................................. 44
Summary ....................................................................................................... 45
Machine Learning versus Deep Learning 47
Introduction .................................................................................................. 48
Advantages of ANNs over Traditional Machine Learning Algorithms ......... 50
Advantages of Traditional Machine Learning Algorithms over ANNs ......... 51
Hierarchical Data Representation ................................................................... 51
Linear Transformations .............................................................................. 52
Scalars, Vectors, Matrices, and Tensors .......................................................... 53
Tensor Addition .................................................................................................. 53
Exercise 6: Perform Various Operations with Vectors, Matrices,
and Tensors ........................................................................................................ 54
Reshaping ........................................................................................................... 58
Matrix Transposition ......................................................................................... 58
Exercise 7: Matrix Reshaping and Transposition ........................................... 59
Matrix Multiplication ......................................................................................... 62
Exercise 8: Matrix Multiplication ...................................................................... 64
Exercise 9: Tensor Multiplication ..................................................................... 65
Introduction to Keras .................................................................................. 68
Layer Types ......................................................................................................... 69
Activation Functions .......................................................................................... 70
Model Fitting ....................................................................................................... 72
Activity 2: Creating a Logistic Regression Model Using Keras ...................... 73
Summary ....................................................................................................... 74
Deep Learning with Keras 77
Introduction .................................................................................................. 78
Building Your First Neural Network .......................................................... 78
Logistic Regression to a Deep Neural Network ............................................. 79
Activation Functions .......................................................................................... 81
Forward Propagation for Making Predictions ............................................... 84
Loss Function ...................................................................................................... 85
Backpropagation for Computing Derivatives of Loss Function ................... 86
Gradient Descent for Learning Parameters ................................................... 87
Exercise 10: Neural Network Implementation with Keras ........................... 90
Activity 3: Building a Single-Layer Neural Network for Performing
Binary Classification .......................................................................................... 94
Model Evaluation ......................................................................................... 96
Evaluating a Trained Model with Keras ........................................................... 96
Splitting Data into Training and Test Sets ...................................................... 97
Underfitting and Overfitting ............................................................................. 99
Early Stopping ................................................................................................. 104
Activity 4: Diabetes Diagnosis with Neural Networks ................................ 105
Summary ..................................................................................................... 108
Evaluate Your Model with Cross-Validation
using Keras Wrappers 111
Introduction ................................................................................................ 112
Cross-Validation ......................................................................................... 112
Drawbacks of Splitting a Dataset Only Once ............................................... 113
K-Fold Cross-Validation .................................................................................. 115
Leave-One-Out Cross-Validation ................................................................... 117
Comparing the K-Fold and LOO Methods .................................................... 118
Cross-Validation for Deep Learning Models ........................................... 119
Keras Wrapper with scikit-learn .................................................................... 119
Exercise 11: Building the Keras Wrapper with scikit-learn
for a Regression Problem ............................................................................... 120
Cross-Validation with scikit-learn ................................................................. 122
Cross-Validation Iterators in scikit-learn ..................................................... 123
Exercise 12: Evaluate Deep Neural Networks with Cross-Validation ....... 125
Activity 5: Model Evaluation Using Cross-Validation
for a Diabetes Diagnosis Classifier ............................................................... 127
Model Selection with Cross-validation .................................................... 128
Cross-Validation for Model Evaluation versus Model Selection ............... 129
Exercise 13: Write User-Defined Functions to Implement
Deep Learning Models with Cross-Validation ............................................. 131
Activity 6: Model Selection Using Cross-Validation
for the Diabetes Diagnosis Classifier ............................................................ 137
Activity 7: Model Selection Using Cross-validation
on the Boston House Prices Dataset ............................................................ 139
Summary ..................................................................................................... 141
Improving Model Accuracy 143
Introduction ................................................................................................ 144
Regularization ............................................................................................ 144
The Need for Regularization .......................................................................... 145
Reducing Overfitting with Regularization .................................................... 146
L1 and L2 Regularization ........................................................................... 147
L1 and L2 Regularization Formulation ......................................................... 147
L1 and L2 Regularization Implementation in Keras ................................... 149
Activity 8: Weight Regularization on a Diabetes Diagnosis Classifier ...... 150
Dropout Regularization ............................................................................. 151
Principles of Dropout Regularization ........................................................... 152
Reducing Overfitting with Dropout .............................................................. 153
Exercise 14: Dropout Implementation in Keras .......................................... 154
Activity 9: Dropout Regularization on Boston Housing Dataset ............... 157
Other Regularization Methods ................................................................. 158
Early Stopping ................................................................................................. 159
Exercise 15: Implementing Early Stopping in Keras ................................... 161
Data Augmentation ........................................................................................ 168
Adding Noise ................................................................................................... 168
Hyperparameter Tuning with scikit-learn .............................................. 169
Grid Search with scikit-learn ......................................................................... 170
Randomized Search with scikit-learn ........................................................... 172
Activity 10: Hyperparameter Tuning on the Diabetes
Diagnosis Classifier ......................................................................................... 174
Summary ..................................................................................................... 175
Model Evaluation 177
Introduction ................................................................................................ 178
Accuracy ...................................................................................................... 178
Exercise 16: Calculating Null Accuracy on a Dummy
Healthcare Dataset ......................................................................................... 179
Advantages and Limitations of Accuracy ..................................................... 181
Imbalanced Datasets ................................................................................. 182
Working with Imbalanced Datasets ............................................................. 183
Confusion Matrix ....................................................................................... 183
Metrics Computed from a Confusion Matrix ............................................... 185
Exercise 17: Computing Accuracy and Null Accuracy
with Healthcare Data ..................................................................................... 188
Activity 11: Computing the Accuracy and Null Accuracy
of a Neural Network When We Change the Train/Test Split ..................... 199
Activity 12: Derive and Compute Metrics Based
on a Confusion Matrix .................................................................................... 200
Exercise 18: Calculate the ROC and AUC Curves ......................................... 201
Summary ..................................................................................................... 205
Computer Vision with Convolutional Neural Networks 207
Introduction ................................................................................................ 208
Computer Vision ........................................................................................ 208
Convolutional Neural Networks .............................................................. 209
Architecture of a CNN ............................................................................... 209
Input Image ..................................................................................................... 210
Convolution Layer ........................................................................................... 210
Pooling Layer ................................................................................................... 212
Flattening ......................................................................................................... 214
Image Augmentation ................................................................................. 216
Advantages of Image Augmentation ............................................................ 217
Exercise 19: Build a CNN and Identify Images of Cats and Dogs .............. 218
Activity 13: Amending our Model with Multiple Layers and the
Use of SoftMax ................................................................................................ 221
Exercise 20: Amending our model by reverting to the Sigmoid
activation function .......................................................................................... 222
Exercise 21: Changing the Optimizer from Adam to SGD .......................... 224
Exercise 22: Classifying a New Image ........................................................... 227
Activity 14: Classify a New Image .................................................................. 229
Summary ..................................................................................................... 230
Transfer Learning and Pre-Trained Models 233
Introduction ................................................................................................ 234
Pre-Trained Sets and Transfer Learning ................................................. 234
Feature Extraction .......................................................................................... 235
Fine-Tuning a Pre-Trained Network ....................................................... 238
The ImageNet Dataset ................................................................................... 238
Some Pre-Trained Networks in Keras .......................................................... 238
Exercise 23: Identify an Image Using the VGG16 Network ........................ 239
Activity 15: Use the VGG16 Network to Train a Deep Learning
Network to Identify Images ........................................................................... 242
Exercise 24: Classification of Images That Are Not Present
in the ImageNet Database. ............................................................................ 243
Exercise 25: Fine-Tune the VGG16 Model ..................................................... 247
Exercise 26: Image Classification with ResNet ............................................ 252
Activity 16: Image Classification with ResNet ............................................. 255
Summary ..................................................................................................... 256