Deep Learning for Computer Vision with Python, Volume 2

210 Pages·2017·9.53 MB·english

by  Adrian Rosebrock

Deep Learning for Computer Vision with
Python
Practitioner Bundle
Dr. Adrian Rosebrock
1stEdition(1.1.0)
Copyright(cid:13)c 2017AdrianRosebrock,PyImageSearch.com
PUBLISHED BY PYIMAGESEARCH
PYIMAGESEARCH.COM
Thecontentsofthisbook,unlessotherwiseindicated,areCopyright(cid:13)c 2017AdrianRosebrock,
PyimageSearch.com. Allrightsreserved. Bookslikethisaremadepossiblebythetimeinvestedby
theauthors. Ifyoureceivedthisbookanddidnotpurchaseit,pleaseconsidermakingfuturebooks
possible by buying a copy at https://www.pyimagesearch.com/deep-learning-computer-vision-
python-book/today.
Firstprinting,September2017
Tomyfather,Joe;mywife,Trisha;
andthefamilybeagles,JosieandJemma.
Withouttheirconstantloveandsupport,
thisbookwouldnotbepossible.

Contents
1 Introduction .................................................. 11
2 Data Augmentation ........................................... 13
2.1 What Is Data Augmentation? 13
2.2 Visualizing Data Augmentation 14
2.3 Comparing Training With and Without Data Augmentation 17
2.3.1 TheFlowers-17Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.2 Aspect-awarePreprocessing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.3 Flowers-17: NoDataAugmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.4 Flowers-17: WithDataAugmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4 Summary 29
3 Networks as Feature Extractors ................................ 31
3.1 Extracting Features with a Pre-trained CNN 32
3.1.1 WhatIsHDF5? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1.2 WritingFeaturestoanHDF5Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2 The Feature Extraction Process 37
3.2.1 ExtractingFeaturesFromAnimals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2.2 ExtractingFeaturesFromCALTECH-101 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.2.3 ExtractingFeaturesFromFlowers-17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3 Training a Classifier on Extracted Features 43
3.3.1 ResultsonAnimals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.2 ResultsonCALTECH-101 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.3 ResultsonFlowers-17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4 Summary 46
4 Understanding rank-1 & rank-5 Accuracies .................... 49
4.1 Ranked Accuracy 49
4.1.1 Measuringrank-1andrank-5Accuracies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.1.2 ImplementingRankedAccuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.1.3 RankedAccuracyonFlowers-17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.1.4 RankedAccuracyonCALTECH-101 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.2 Summary 54
5 Fine-tuning Networks ......................................... 57
5.1 Transfer Learning and Fine-tuning 57
5.1.1 IndexesandLayers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.1.2 NetworkSurgery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.1.3 Fine-tuning,fromStarttoFinish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.2 Summary 69
6 Improving Accuracy with Network Ensembles .................. 71
6.1 Ensemble Methods 71
6.1.1 Jensen’sInequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6.1.2 ConstructinganEnsembleofCNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6.1.3 EvaluatinganEnsemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.2 Summary 80
7 Advanced Optimization Methods .............................. 83
7.1 Adaptive Learning Rate Methods 83
7.1.1 Adagrad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
7.1.2 Adadelta . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
7.1.3 RMSprop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7.1.4 Adam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7.1.5 Nadam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
7.2 Choosing an Optimization Method 86
7.2.1 ThreeMethodsYouShouldLearnhowtoDrive: SGD,Adam,andRMSprop . . 86
7.3 Summary 87
8 Optimal Pathway to Apply Deep Learning ..................... 89
8.1 A Recipe for Training 89
8.2 Transfer Learning or Train from Scratch 93
8.3 Summary 94
9 Working with HDF5 and Large Datasets ......................... 95
9.1 Downloading Kaggle: Dogs vs. Cats 95
9.2 Creating a Configuration File 96
9.2.1 YourFirstConfigurationFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
9.3 Building the Dataset 98
9.4 Summary 102
10 Competing in Kaggle: Dogs vs. Cats .......................... 103
10.1 Additional Image Preprocessors 103
10.1.1 MeanPreprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
10.1.2 PatchPreprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
10.1.3 CropPreprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
10.2 HDF5 Dataset Generators 109
10.3 Implementing AlexNet 112
10.4 Training AlexNet on Kaggle: Dogs vs. Cats 117
10.5 Evaluating AlexNet 120
10.6 Obtaining a Top-5 Spot on the Kaggle Leaderboard 123
10.6.1 ExtractingFeaturesUsingResNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
10.6.2 TrainingaLogisticRegressionClassifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
10.7 Summary 128
11 GoogLeNet ................................................. 131
11.1 The Inception Module (and its Variants) 132
11.1.1 Inception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
11.1.2 Miniception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
11.2 MiniGoogLeNet on CIFAR-10 134
11.2.1 ImplementingMiniGoogLeNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
11.2.2 TrainingandEvaluatingMiniGoogLeNetonCIFAR-10 . . . . . . . . . . . . . . . . . . . 140
11.2.3 MiniGoogLeNet: Experiment#1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
11.2.4 MiniGoogLeNet: Experiment#2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
11.2.5 MiniGoogLeNet: Experiment#3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
11.3 The Tiny ImageNet Challenge 146
11.3.1 DownloadingTinyImageNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
11.3.2 TheTinyImageNetDirectoryStructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
11.3.3 BuildingtheTinyImageNetDataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
11.4 DeeperGoogLeNet on Tiny ImageNet 153
11.4.1 ImplementingDeeperGoogLeNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
11.4.2 TrainingDeeperGoogLeNetonTinyImageNet . . . . . . . . . . . . . . . . . . . . . . . . 161
11.4.3 CreatingtheTrainingScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
11.4.4 CreatingtheEvaluationScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
11.4.5 DeeperGoogLeNetExperiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
11.5 Summary 168
12 ResNet ...................................................... 171
12.1 ResNet and the Residual Module 171
12.1.1 GoingDeeper: ResidualModulesandBottlenecks . . . . . . . . . . . . . . . . . . . . . 172
12.1.2 RethinkingtheResidualModule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
12.2 Implementing ResNet 175
12.3 ResNet on CIFAR-10 180
12.3.1 TrainingResNetonCIFAR-10Withthectrl+cMethod . . . . . . . . . . . . . . . . . . 181
12.3.2 ResNetonCIFAR-10: Experiment#2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
12.4 Training ResNet on CIFAR-10 with Learning Rate Decay 188
12.5 ResNet on Tiny ImageNet 192
12.5.1 UpdatingtheResNetArchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
12.5.2 TrainingResNetonTinyImageNetWiththectrl+cMethod . . . . . . . . . . . . . . 194
12.5.3 TrainingResNetonTinyImageNetwithLearningRateDecay. . . . . . . . . . . . . 198
12.6 Summary 202