Introduction to Intelligent Surveillance: Surveillance Data Capture, Transmission, and Analytics

229 Pages·2019·8.85 MB·English

by  Wei Qi Yan

Texts in Computer Science
Wei Qi Yan
Introduction
to Intelligent
Surveillance
Surveillance Data Capture,
Transmission, and Analytics
Third Edition
Texts in Computer Science
Series editors
David Gries, Department of Computer Science, Cornell University, Ithaca, NY,
USA
Orit Hazzan, Faculty of Education in Technology and Science, Technion—Israel
Institute of Technology, Haifa, Israel
More information about this series at http://www.springer.com/series/3191
Wei Qi Yan
Introduction to Intelligent
Surveillance
Surveillance Data Capture,
Transmission, and Analytics
Third Edition
123
Wei QiYan
Department ofComputer Science
Auckland University of Technology
Auckland,New Zealand
ISSN 1868-0941 ISSN 1868-095X (electronic)
Textsin Computer Science
ISBN978-3-030-10712-3 ISBN978-3-030-10713-0 (eBook)
https://doi.org/10.1007/978-3-030-10713-0
LibraryofCongressControlNumber:2018966855
1stedition:©SpringerInternationalPublishingSwitzerland2016
2ndedition:©SpringerInternationalPublishingAG2017
3rdedition:©SpringerNatureSwitzerlandAG2019
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Digital surveillance is a leading-edge security problem and has been unprece-
dentedlyappliedtomonitorandprotectourlivesalmostanywhere.Itisubiquitous
andpracticalandcould greatly lessensecuritystaff’shuman labor.Withtheaidof
intelligence, surveillance systems have the capability to automatically complete
tasks such as object detection, recognition, tracking as well as event life cycle
including event detection, recognition, search, retrieval, mining, reasoning, etc.
Withthedevelopmentofartificialintelligence(AI),thesystemshavetakeninfrom
the progression of supercomputing, cloud computing, big data, deep learning, etc.
Thisbookstartsfromcameracalibration,alongwithsurveillancedatacapturing,
scrambling and descrambling, secure transmission with secure network environ-
ment; afterward, the surveillance at object level has been introduced such as
detection,recognition,andtracking.Biometricsispresentedasanimportantpartof
this book. A picture is more than a thousand of words, an event is more than
thousands of pictures. An event is the basic unit of knowledge which bridges the
gapbetweenphysicalworldandsemanticobjects;thelifecycleofaneventincludes
generatingnewonesandoperationsonevents,etc.Theknowledgebasedonevents
could be used for discovery and exploration, and this will be employed for
surveillance alarm making; at the end of this book, fundamentals of supercom-
puting(FPGA,GPU,andparallelcomputing),cloudcomputing,mobilecomputing,
deep learning, etc., will be emphasized.
This book is based on our research and teaching experience, we have used the
content of this book for postgraduate teaching in the higher education. The stuff
ofthewholecourseincludingassignmentsandexaminationshasbeenverifiedfora
plurality of times and could serve the readers of this book well.
This book was written for research students and engineers as well as scientists
who are interested in intelligent surveillance.
Auckland, New Zealand Wei Qi Yan
October 2018
v
Acknowledgements
The first edition of this book has been published in March 2016. In the past two
years,wehaveendowedinallaspectstomakethebookfullandperfect.Following
the feedback from readers and audiences responses, the author has further omitted
mistakes and typos, detailed the mathematical descriptions and algorithms, as well
as updated each chapter with the latest contents and references.
ThesecondeditionofthisbookwaspublishedinJune2017andemphasizedon
red-hot technology such as deep learning, mobile and cloud computing, and big
data in intelligent surveillance. The author’s endeavor was about how surveillance
research and teaching could take in nutrition from the progress of other fields and
whattheaudienceandreaderscouldpaytheirattentiontowhentheyreadthisbook.
The third edition of this book is emphasized on human behavior analysis,
privacypreservation,andthedetailsofdeeplearningandartificialintelligence(AI).
We integrate the latest development in AI and machine learning into this book for
meeting the trends of today’s research. The book shows how machine intelligence
could assistsecuritypeopleinsurveillancewithregard tothefundamentalaspects:
observation, learning, presentation, and reasoning or references.
Thankstoourpeercolleaguesandstudentswhosematerialswerereferencedand
who have given invaluable comments on this book. Special thanks to my super-
visedstudents:Dr.J.Weir,Dr.D.Kieran,Mr.Y.Jiao,Mr.G.Wang,Mr.W.Cui,
Mr. D. Gu, Mr. J. Wang, Dr. Y. Zhang, Mr. L. Zhou, Mr. J. Lu, Mr. J. Shen,
Mr. D. Shen, Mr. K. Zheng, Ms Y. Ren, Mr. R. Li, Mr. P. Li, Mr. Z. Liu,
Ms. Y. Shen, Ms. H. Wang, Mr. C. Xin, Ms. Q. Zhang, Ms X. Zhang, Dr. Q. Gu,
our colleagues: Dr. M. Nguyen, Prof. R. Klette, et al.
vii
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Introduction to Surveillance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Surveillance History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Open Problems in Intelligent Surveillance. . . . . . . . . . . . . 6
1.2 Introduction to Sensor Deployment and Calibration . . . . . . . . . . . 8
1.2.1 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.2 Surveillance Cameras. . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.2.3 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.3 Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2 Surveillance Data Capturing and Compression. . . . . . . . . . . . . . . . . 23
2.1 Data Capturing Using FSM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 Data Capturing Using DNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.3 Surveillance Data Compression . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4 Digital Image Encoding and Decoding. . . . . . . . . . . . . . . . . . . . . 33
2.5 Digital Video Encoding and Decoding. . . . . . . . . . . . . . . . . . . . . 38
2.6 Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3 Surveillance Data Secure Transmissions. . . . . . . . . . . . . . . . . . . . . . 45
3.1 Introduction to Network Communications . . . . . . . . . . . . . . . . . . 45
3.1.1 Computer Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.1.2 Network Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.1.3 Network Threats and Attacks . . . . . . . . . . . . . . . . . . . . . . 50
3.2 Data Scrambling and Descrambling Technologies. . . . . . . . . . . . . 59
3.2.1 Cryptographic mod Operation. . . . . . . . . . . . . . . . . . . . . . 59
3.2.2 Cryptographic xor Operation . . . . . . . . . . . . . . . . . . . . . . 64
3.2.3 Image Encryption on Frequency Domain . . . . . . . . . . . . . 68
3.3 Data Secure Transmission over Internet and Networks . . . . . . . . . 69
3.4 Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
ix
x Contents
4 Surveillance Data Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.1 Object Computable Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.2 Object Segmentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.3 Object Annotation, Classification, and Recognition. . . . . . . . . . . . 96
4.4 Object Locating and Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.4.1 Support Vector Machine (SVM). . . . . . . . . . . . . . . . . . . . 103
4.4.2 Artificial Neural Networks (ANN) . . . . . . . . . . . . . . . . . . 110
4.4.3 Deep Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.5 Metadata of Surveillance Data. . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.6 Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5 Biometrics for Surveillance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.1 Introduction to Biometrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.2 Biometrics Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.2.1 Measurement and Analysis. . . . . . . . . . . . . . . . . . . . . . . . 132
5.2.2 Fingerprint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5.2.3 Palm Print. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5.2.4 Human Face . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.2.5 Gender Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.2.6 EBGM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.2.7 Twin Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.2.8 Pedestrians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.2.9 Suspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
5.3 Human Actions, Behaviors, and Gestures. . . . . . . . . . . . . . . . . . . 139
5.4 Human Privacy and Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.4.1 Pixelization and Gaussian Blurring . . . . . . . . . . . . . . . . . . 141
5.4.2 Audio Progressive Scrambling . . . . . . . . . . . . . . . . . . . . . 142
5.4.3 Covering and Overlapping . . . . . . . . . . . . . . . . . . . . . . . . 143
5.5 Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6 Visual Event Computing I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
6.1 Definition of an Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
6.2 Event Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
6.3 Event Detection and Recognition. . . . . . . . . . . . . . . . . . . . . . . . . 158
6.4 Event Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.5 Event Clustering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
6.6 Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Contents xi
7 Visual Event Computing II. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
7.1 Event Search and Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
7.1.1 Lexical Search and Retrieval . . . . . . . . . . . . . . . . . . . . . . 167
7.1.2 Global and Local Search . . . . . . . . . . . . . . . . . . . . . . . . . 169
7.2 Event Mining and Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
7.2.1 Event Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
7.2.2 Event Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
7.3 Event Exploration and Discovery . . . . . . . . . . . . . . . . . . . . . . . . 177
7.3.1 Event Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.3.2 Event Exploration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.4 Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
8 Surveillance Alarm Making . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
8.1 Alarm Setting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
8.1.1 Rule-Based Alarming. . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
8.1.2 Probability-Based Alarming . . . . . . . . . . . . . . . . . . . . . . . 182
8.1.3 System-Based Alarming. . . . . . . . . . . . . . . . . . . . . . . . . . 184
8.2 Decision Making . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
8.2.1 Simple Decision Making . . . . . . . . . . . . . . . . . . . . . . . . . 184
8.2.2 Complex Decision Making. . . . . . . . . . . . . . . . . . . . . . . . 189
8.2.3 Alarming Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
8.3 Alarming for a Traffic Ticketing System . . . . . . . . . . . . . . . . . . . 191
8.4 Incident Responses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
8.5 Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
9 Surveillance Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
9.1 Challenges of Modern Computing. . . . . . . . . . . . . . . . . . . . . . . . 195
9.1.1 Parallel Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
9.1.2 Multicore Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
9.1.3 Cluster Computing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
9.1.4 Supercomputing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
9.1.5 High-Performance Computing . . . . . . . . . . . . . . . . . . . . . 199
9.1.6 Multithread Computing . . . . . . . . . . . . . . . . . . . . . . . . . . 200
9.1.7 Graphics Processing Unit (GPU) . . . . . . . . . . . . . . . . . . . 201
9.2 OpenCL Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
9.2.1 OpenCL Programming. . . . . . . . . . . . . . . . . . . . . . . . . . . 201
9.2.2 Examples of OpenCL Programming . . . . . . . . . . . . . . . . . 204
9.3 MATLAB Parallel Computing. . . . . . . . . . . . . . . . . . . . . . . . . . . 205
9.4 Mobile Computing for Surveillance. . . . . . . . . . . . . . . . . . . . . . . 206