Deep Learning and Missing Data in Engineering Systems

188 Pages·2019·6.16 MB·English

by  Collins Achepsah Leke

Studies in Big Data 48
Collins Achepsah Leke
Tshilidzi Marwala
Deep Learning
and Missing Data
in Engineering
Systems
Studies in Big Data
Volume 48
Series editor
Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland
e-mail: [email protected]
Theseries“StudiesinBigData”(SBD)publishesnewdevelopmentsandadvances
in the various areas of Big Data- quickly and with a high quality. The intent is to
coverthetheory,research,development,andapplicationsofBigData,asembedded
inthefieldsofengineering,computerscience,physics,economicsandlifesciences.
The books of the series refer to the analysis and understanding of large, complex,
and/or distributed data sets generated from recent digital sources coming from
sensorsorotherphysicalinstrumentsaswellassimulations,crowdsourcing,social
networks or other internet transactions, such as emails or video click streams and
others. The series contains monographs, lecture notes and edited volumes in Big
Data spanning the areas of computational intelligence including neural networks,
evolutionary computation, soft computing, fuzzy systems, as well as artificial
intelligence, data mining, modern statistics and operations research, as well as
self-organizing systems. Of particular value to both the contributors and the
readership are the short publication timeframe and the world-wide distribution,
which enable both wide and rapid dissemination of research output.
More information about this series at http://www.springer.com/series/11970
Collins Achepsah Leke Tshilidzi Marwala
(cid:129)
Deep Learning and Missing
Data in Engineering Systems
123
Collins Achepsah Leke Tshilidzi Marwala
Faculty of Engineering andBuilt Faculty of Engineering andBuilt
Environment Environment
University of Johannesburg University of Johannesburg
Auckland Park, SouthAfrica Auckland Park, SouthAfrica
ISSN 2197-6503 ISSN 2197-6511 (electronic)
Studies in BigData
ISBN978-3-030-01179-6 ISBN978-3-030-01180-2 (eBook)
https://doi.org/10.1007/978-3-030-01180-2
LibraryofCongressControlNumber:2018960236
Intel® and Core™ are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other
countries, Intel Corporation,2200 Mission College Boulevard, Santa Clara, CA 95054, USA, https://
www.intel.com
MATLAB® is a registered trademark of The MathWorks, Inc., 1 Apple Hill Drive, Natick, MA
01760-2098,USA,http://www.mathworks.com
©SpringerNatureSwitzerlandAG2019
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Deep Learning and Missing Data in Engineering Systems discuss concepts and
applications of artificial intelligence, specifically, deep learning. The artificial
intelligence techniques that are studied include multilayer autoencoder networks
and deep autoencoder networks. Also studied in this book are computational and
swarm intelligence techniques which include ant colony optimization, ant lion
optimizer,batalgorithm,cuckoosearchoptimization,fireflyalgorithmandinvasive
weed optimization algorithm. In addition, this book explores using deep autoen-
codernetworkswithavaryingnumberofhiddenlayers.Thisbookalsostudiesthe
reconstruction of images from reduced dimensions obtained from the bottleneck
layer of the deep autoencoder.
These techniques are used to solve the missing data problem in an image
recognition and reconstruction context.
This book is an interesting reference for graduate students, researchers and
artificial intelligence practitioners with interest in deep learning and image
processing.
Auckland Park, South Africa Collins Achepsah Leke, Ph.D.
July 2018 Tshilidzi Marwala, Ph.D.
v
Acknowledgements
We would like to thank the following people for their contribution to this manu-
script:MsiziKhoza,Dr.MlungisiDuma,Prof.FulufheloNelwamondo,Dr.Vukosi
Marivate, Dr. Marcos Alvares, Dr. Pramod Parida, Dr. Ali Hasan, Dr. Ilyes
Boulkaibet and Dr. Satyakama Paul.
We also thank colleagues and practitioners that have collaborated directly and
indirectly to the writing of the manuscript including Dr. Alain Richard Ndjiongue,
StephanMartinNlom,MokesioluwaFanoro,ChwayitaMacandaandNqobileDudu.
We thank Prof. Bhekisipho Twala.
WededicatethisbooktoLekeNeeNchangehAgathaFonkeng,LekeNchabanu
Gwendoline,Dr.LekeClarenceFonkeng,Dr.LekeBetechuohBrain,MbaliDenga
Marwala,LwaziThendoMarwala,NhloniphoKhathusheloMarwalaandDr.Jabulile
VuyiswaManana.
July 2018 Collins Achepsah Leke
Tshilidzi Marwala
vii
Contents
1 Introduction to Missing Data Estimation . . . . . . . . . . . . . . . . . . . . 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 The Evolution of Missing Data Estimation Methods . . . . . . . . . 2
1.3 Missing Data Proportions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Missing Data Mechanisms. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4.1 Missing Completely at Random (MCAR) . . . . . . . . . . 4
1.4.2 Missing at Random (MAR) . . . . . . . . . . . . . . . . . . . . 4
1.4.3 Non-ignorable Case or Missing not
at Random (MNAR). . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4.4 Missing by Natural Design (MBND). . . . . . . . . . . . . . 6
1.5 Missing Data Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.6 Classical Missing Data Techniques . . . . . . . . . . . . . . . . . . . . . 7
1.6.1 Listwise or Casewise Deletion . . . . . . . . . . . . . . . . . . 7
1.6.2 Pairwise Deletion. . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6.3 Mean–Mode Substitution . . . . . . . . . . . . . . . . . . . . . . 8
1.6.4 Imputation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6.4.1 Single-Imputation-Based Techniques . . . . . . . 8
1.6.4.2 Multiple-Imputation-Based Techniques. . . . . . 9
1.6.5 Regression Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.7 Machine Learning Approaches to Missing Data . . . . . . . . . . . . 10
1.7.1 Decision Trees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.7.2 Artificial Neural Networks (ANNs) . . . . . . . . . . . . . . . 11
1.7.2.1 Auto-associative Neural Network. . . . . . . . . . 12
1.7.3 Support Vector Machine (SVM) . . . . . . . . . . . . . . . . . 14
1.8 Machine Learning Optimization. . . . . . . . . . . . . . . . . . . . . . . . 14
1.8.1 Genetic Algorithm (GA). . . . . . . . . . . . . . . . . . . . . . . 15
1.8.2 Particle Swarm Optimization (PSO). . . . . . . . . . . . . . . 15
1.8.3 Simulated Annealing (SA) . . . . . . . . . . . . . . . . . . . . . 16
1.9 Challenges to Missing Data Imputation . . . . . . . . . . . . . . . . . . 16
ix
x Contents
1.10 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2 Introduction to Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2 Restricted Boltzmann Machines (RBMs) . . . . . . . . . . . . . . . . . 22
2.3 Contrastive Divergence (CD). . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4 Deep Belief Networks (DBNs) . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5 Convolutional Neural Networks (CNNs) . . . . . . . . . . . . . . . . . 28
2.5.1 Convolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.5.2 Non-linearity (ReLU) . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.5.3 Pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5.4 Fully Connected Layer . . . . . . . . . . . . . . . . . . . . . . . . 34
2.6 Recurrent Neural Networks (RNNs). . . . . . . . . . . . . . . . . . . . . 34
2.7 Deep Autoencoders (DAE) . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3 Missing Data Estimation Using Bat Algorithm . . . . . . . . . . . . . . . . 41
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.3 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.5 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4 Missing Data Estimation Using Cuckoo Search Algorithm . . . . . . . 57
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2 Cuckoo Search (CS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.3 Proposed Approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.4 Experimental Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.5 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5 Missing Data Estimation Using Firefly Algorithm. . . . . . . . . . . . . . 73
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.2 Review of Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2.1 Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2.1.1 Missing Data Mechanisms. . . . . . . . . . . . . . . 75
5.2.1.2 Missing Data Patterns . . . . . . . . . . . . . . . . . . 76
5.2.2 Deep Learning (DL). . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.2.3 Restricted Boltzmann Machine (RBM) . . . . . . . . . . . . 76
5.2.4 Contrastive Divergence (CD) . . . . . . . . . . . . . . . . . . . 78
5.2.5 Autoencoder (AE) . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.2.6 Firefly Algorithm (FA). . . . . . . . . . . . . . . . . . . . . . . . 80
Contents xi
5.3 Experimental Design and Procedure. . . . . . . . . . . . . . . . . . . . . 81
5.4 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6 Missing Data Estimation Using Ant Colony Optimization
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.2 Ant Colony Optimization (ACO). . . . . . . . . . . . . . . . . . . . . . . 93
6.3 Experimental Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.4 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.5 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
7 Missing Data Estimation Using Ant-Lion Optimizer Algorithm . . . 103
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.2 Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.3 Ant-Lion Optimizer (ALO) . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.4 Missing Data Estimation Model. . . . . . . . . . . . . . . . . . . . . . . . 108
7.5 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.6 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
8 Missing Data Estimation Using Invasive Weed Optimization
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
8.2 Invasive Weed Optimization (IWO) Algorithm. . . . . . . . . . . . . 116
8.3 Missing Data Estimation Model. . . . . . . . . . . . . . . . . . . . . . . . 118
8.4 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
8.5 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
9 Missing Data Estimation Using Swarm Intelligence Algorithms
from Reduced Dimensions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
9.1 Auto-Associative Neural Network . . . . . . . . . . . . . . . . . . . . . . 129
9.2 Missing Data Estimation Model. . . . . . . . . . . . . . . . . . . . . . . . 130
9.3 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
9.3.1 Ant Colony Optimization Results . . . . . . . . . . . . . . . . 135
9.3.2 Ant-Lion Optimizer Results . . . . . . . . . . . . . . . . . . . . 136
9.3.3 Bat Algorithm Results . . . . . . . . . . . . . . . . . . . . . . . . 138
9.3.4 Cuckoo Search Algorithm Results. . . . . . . . . . . . . . . . 140
9.3.5 Firefly Algorithm Results . . . . . . . . . . . . . . . . . . . . . . 141
9.3.6 Invasive Weed Optimization Results . . . . . . . . . . . . . . 144
9.4 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146