Table Of ContentStudies in Computational Intelligence 711
Brijesh Verma
Ligang Zhang
David Stockwell
Roadside
Video Data
Analysis: Deep
Learning
Studies in Computational Intelligence
Volume 711
Series editor
Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland
e-mail: [email protected]
About this Series
The series “Studies in Computational Intelligence” (SCI) publishes new develop-
mentsandadvancesinthevariousareasofcomputationalintelligence—quicklyand
with a high quality. The intent is to cover the theory, applications, and design
methods of computational intelligence, as embedded in the fields of engineering,
computer science, physics and life sciences, as well as the methodologies behind
them. The series contains monographs, lecture notes and edited volumes in
computational intelligence spanning the areas of neural networks, connectionist
systems, genetic algorithms, evolutionary computation, artificial intelligence,
cellular automata, self-organizing systems, soft computing, fuzzy systems, and
hybrid intelligent systems. Of particular value to both the contributors and the
readership are the short publication timeframe and the worldwide distribution,
which enable both wide and rapid dissemination of research output.
More information about this series at http://www.springer.com/series/7092
Brijesh Verma Ligang Zhang
(cid:129)
David Stockwell
Roadside Video Data
Analysis: Deep Learning
123
Brijesh Verma DavidStockwell
Schoolof Engineering andTechnology Schoolof Engineering andTechnology
Central Queensland University Central Queensland University
Brisbane, QLD Brisbane, QLD
Australia Australia
LigangZhang
Schoolof Engineering andTechnology
Central Queensland University
Brisbane, QLD
Australia
ISSN 1860-949X ISSN 1860-9503 (electronic)
Studies in Computational Intelligence
ISBN978-981-10-4538-7 ISBN978-981-10-4539-4 (eBook)
DOI 10.1007/978-981-10-4539-4
LibraryofCongressControlNumber:2017936905
©SpringerNatureSingaporePteLtd.2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerNatureSingaporePteLtd.
Theregisteredcompanyaddressis:152BeachRoad,#21-01/04GatewayEast,Singapore189721,Singapore
Preface
Video data analysis has become an increasingly important research area with
widespread applications in automatic surveillance of transportation infrastructure
including roads, rail and airports. As the amount of video data collected grows, so
doestheopportunityforfurtherprocessingwithnewartificialintelligencemethods.
Onetypeofusefulvideodatathathasseenlittleresearchistheroadsidevideodata
that is collected using video-mounted vehicles. These video data may augment or
replace road-based surveys of the conditions of roadside objects such as trees,
grasses, roads and traffic signs and can be potentially used in many real-world
applications such as roadside vegetation growth condition monitoring, effective
roadside management to reduce the possible hazards to drivers and vehicles, and
developingautomaticvehiclesthatareabletoautomaticallysenseroadsideobjects
and traffic signs.
Mostexisting studies onvideo data analysisareprimarily focusedonanalyzing
generic object categories in the data content in public benchmark datasets. Very
limited research has focused on the analysis of roadside video data, although the
significance of developing smart techniques for roadside video data analysis has
beenwidelyrecognized.Oneofthemainreasonsisprobablybecausethereisalack
ofacomprehensivepublicdatasetthatwasspecificallycreatedforroadsideobjects.
Another reason is the various types of variations and environmental conditions
encountered along road sides, which are still challenging issues in the computer
visionfield.Thegreatvariabilityintheappearanceandstructure ofobjectsaswell
asthevarioustypesofenvironmentaleffectssuchasunderexposure,overexposure,
shadows, and sunlight reflectance make accurate segmentation and recognition of
objects difficult. The current literature lacks a comprehensive review of existing
machine learning algorithms, particularly deep learning techniques, on roadside
data analysis.
This book highlights the methods and applications for roadside video data
analysis. It describes various system architectures and methodologies that are
specificallybuiltupondifferenttypesoflearningalgorithmsforroadsidevideodata
processing, with detailed analysis of the segmentation, feature extraction and
classification. The use of deep learning to solve the roadside video data
v
vi Preface
segmentation and classification problems is one of the major highlights of this
book. Deep neural net learning has become popular in machine learning and data
mining areas. However, the benefits of a deep feature free approach must be bal-
anced against the considerations of accuracy and robustness, and most real-world
learningsystems requiresome hand engineering offeatures andarchitectures. This
book examines via empirical testing the types of features and architectures that
contribute to the performance of multi-layer neural nets on real-world scene anal-
ysis.Wethendemonstratenovelarchitecturesthatperformsceneclassificationwith
equalorbetteraccuracytopreviousmethodsandinvestigatethefeatureengineering
intoconvolutionalneuralnetworks.Further,weprovideanindustrialperspectiveto
help align theoretical concerns with real-world results. Finally, as a case study of
roadside video data analysis, we demonstrate an application of vegetation biomass
estimationtechniquesforroadsidefireriskassessment.Overall,thisbookcompiles
the most useful strategies in the field of scene analysis to help researchers identify
the most appropriate features and architectures for their applications.
Brisbane, Australia Brijesh Verma
Ligang Zhang
David Stockwell
Acknowledgements
The authors express their gratitude for help from the Department of Transport and
Main Roads in Queensland, Australia for the creation of the data and assistance
with field resources for the fire risk survey. This work was supported under
Australian Research Council’s Linkage Projects funding scheme (project number
LP140100939).
Many students and research fellows in the Centre for Intelligent Systems at
CentralQueenslandUniversityhavecontributedtoresearchpresentedinthisbook.
The authors would like to thank the following researchers: Dr Sujan Chowdhury,
DrTejyKinattukaraJobachan,DrPeterMcLeod,DrM.Asafuddoula,Mrs.Toshi
Sinha and Mrs. Fatma Shaheen.
vii
Contents
1 Introduction . .... .... .... ..... .... .... .... .... .... ..... .. 1
1.1 Background .. .... .... ..... .... .... .... .... .... ..... .. 1
1.2 Collection of Roadside Video Data . .... .... .... .... ..... .. 2
1.2.1 Industry Data ... ..... .... .... .... .... .... ..... .. 2
1.2.2 Benchmark Data. ..... .... .... .... .... .... ..... .. 7
1.3 Applications Using Roadside Video Data .... .... .... ..... .. 10
1.4 Outline of the Book.... ..... .... .... .... .... .... ..... .. 11
References ... .... .... .... ..... .... .... .... .... .... ..... .. 12
2 Roadside Video Data Analysis Framework.. .... .... .... ..... .. 13
2.1 Overview.... .... .... ..... .... .... .... .... .... ..... .. 13
2.2 Methodology . .... .... ..... .... .... .... .... .... ..... .. 14
2.2.1 Pre-processing of Roadside Video Data.... .... ..... .. 14
2.2.2 Segmentation of Roadside Video Data into Objects.... .. 18
2.2.3 Feature Extraction from Objects.. .... .... .... ..... .. 19
2.2.4 Classification of Roadside Objects.... .... .... ..... .. 20
2.2.5 Applications of Classified Roadside Objects. .... ..... .. 21
2.3 Related Work. .... .... ..... .... .... .... .... .... ..... .. 23
2.3.1 Vegetation Segmentation and Classification. .... ..... .. 24
2.3.2 Generic Object Segmentation and Classification.. ..... .. 28
2.4 Matlab Code for Data Processing... .... .... .... .... ..... .. 32
References ... .... .... .... ..... .... .... .... .... .... ..... .. 37
3 Non-deep Learning Techniques for Roadside Video Data
Analysis. .... .... .... .... ..... .... .... .... .... .... ..... .. 41
3.1 Neural Network Learning..... .... .... .... .... .... ..... .. 41
3.1.1 Introduction .... ..... .... .... .... .... .... ..... .. 41
3.1.2 Neural Network Learning Approach... .... .... ..... .. 42
3.1.3 Experimental Results .. .... .... .... .... .... ..... .. 45
3.1.4 Summary .. .... ..... .... .... .... .... .... ..... .. 48
ix
x Contents
3.2 Support Vector Machine Learning.. .... .... .... .... ..... .. 49
3.2.1 Introduction .... ..... .... .... .... .... .... ..... .. 49
3.2.2 SVM Learning Approach... .... .... .... .... ..... .. 49
3.2.3 Experimental Results .. .... .... .... .... .... ..... .. 53
3.2.4 Summary .. .... ..... .... .... .... .... .... ..... .. 56
3.3 Clustering Learning.... ..... .... .... .... .... .... ..... .. 56
3.3.1 Introduction .... ..... .... .... .... .... .... ..... .. 56
3.3.2 Clustering Learning Approach ... .... .... .... ..... .. 57
3.3.3 Experimental Results .. .... .... .... .... .... ..... .. 62
3.3.4 Summary .. .... ..... .... .... .... .... .... ..... .. 73
3.4 Fuzzy C-Means Learning..... .... .... .... .... .... ..... .. 74
3.4.1 Introduction .... ..... .... .... .... .... .... ..... .. 74
3.4.2 Fuzzy C-Means Learning Approach... .... .... ..... .. 75
3.4.3 Experimental Results .. .... .... .... .... .... ..... .. 78
3.4.4 Summary .. .... ..... .... .... .... .... .... ..... .. 80
3.5 Ensemble Learning .... ..... .... .... .... .... .... ..... .. 80
3.5.1 Introduction .... ..... .... .... .... .... .... ..... .. 80
3.5.2 Ensemble Learning Approach.... .... .... .... ..... .. 80
3.5.3 Experimental Results .. .... .... .... .... .... ..... .. 84
3.5.4 Summary .. .... ..... .... .... .... .... .... ..... .. 86
3.6 Majority Voting Based Hybrid Learning . .... .... .... ..... .. 86
3.6.1 Introduction .... ..... .... .... .... .... .... ..... .. 86
3.6.2 Majority Voting Approach.. .... .... .... .... ..... .. 87
3.6.3 Experimental Results .. .... .... .... .... .... ..... .. 90
3.6.4 Summary .. .... ..... .... .... .... .... .... ..... .. 94
3.7 Region Merging Learning .... .... .... .... .... .... ..... .. 95
3.7.1 Introduction .... ..... .... .... .... .... .... ..... .. 95
3.7.2 Region Merging Approach.. .... .... .... .... ..... .. 95
3.7.3 Components of Approach... .... .... .... .... ..... .. 100
3.7.4 Experimental Results .. .... .... .... .... .... ..... .. 109
3.7.5 Summary .. .... ..... .... .... .... .... .... ..... .. 115
References ... .... .... .... ..... .... .... .... .... .... ..... .. 116
4 Deep Learning Techniques for Roadside Video Data Analysis... .. 119
4.1 Introduction.. .... .... ..... .... .... .... .... .... ..... .. 119
4.2 Related Work. .... .... ..... .... .... .... .... .... ..... .. 120
4.3 Automatic Versus Manual Feature Extraction . .... .... ..... .. 122
4.3.1 Introduction .... ..... .... .... .... .... .... ..... .. 122
4.3.2 Comparison Framework.... .... .... .... .... ..... .. 122
4.3.3 Experimental Results .. .... .... .... .... .... ..... .. 124
4.3.4 Summary .. .... ..... .... .... .... .... .... ..... .. 125
4.4 Single Versus Ensemble Architectures... .... .... .... ..... .. 127
4.4.1 Introduction .... ..... .... .... .... .... .... ..... .. 127
4.4.2 Comparison Framework.... .... .... .... .... ..... .. 128
