Cloud Computing in Remote Sensing

293 Pages·2019·15.825 MB·English

by  Lizhe Wang, Jining Yan, Yan Ma

Cloud Computing in
Remote Sensing

Cloud Computing in
Remote Sensing
Lizhe Wang
Jining Yan
Yan Ma
CRCPress
Taylor&FrancisGroup
6000BrokenSoundParkwayNW,Suite300
BocaRaton,FL33487-2742
©2020byTaylor&FrancisGroup,LLC
CRCPressisanimprintofTaylor&FrancisGroup,anInformabusiness
NoclaimtooriginalU.S.Governmentworks
Printedonacid-freepaper
InternationalStandardBookNumber-13:978-1-138-59456-2(Hardback)
Thisbookcontainsinformationobtainedfromauthenticandhighlyregardedsources.Rea-
sonable efforts have been made to publish reliable data and information, but the author
and publisher cannot assume responsibility for the validity of all materials or the conse-
quences of their use. The authors and publishers have attempted to trace the copyright
holdersofallmaterialreproducedinthispublicationandapologizetocopyrightholdersif
permissiontopublishinthisformhasnotbeenobtained.Ifanycopyrightmaterialhasnot
beenacknowledgedpleasewriteandletusknowsowemayrectifyinanyfuturereprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted,
reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other
means,nowknownorhereafterinvented,includingphotocopying,microfilming,andrecord-
ing,orinanyinformationstorageorretrievalsystem,withoutwrittenpermissionfromthe
publishers.
For permission to photocopy or use material electronically from this work, please access
www.copyright.com(http://www.copyright.com/)orcontacttheCopyrightClearanceCen-
ter, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-
for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system
ofpaymenthasbeenarranged.
Trademark Notice:Productorcorporatenamesmaybetrademarksorregisteredtrade-
marks,andareusedonlyforidentificationandexplanationwithoutintenttoinfringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Preface xi
1 Remote Sensing and Cloud Computing 1
1.1 Remote Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Remote sensing definition . . . . . . . . . . . . . . . . . 1
1.1.2 Remote sensing big data . . . . . . . . . . . . . . . . . 2
1.1.3 Applications of remote sensing big data . . . . . . . . 3
1.1.4 Challenges of remote sensing big data . . . . . . . . . 5
1.1.4.1 Data integration challenges . . . . . . . . . . 5
1.1.4.2 Data processing challenges . . . . . . . . . . 5
1.2 Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.1 Cloud service models . . . . . . . . . . . . . . . . . . . 6
1.2.2 Cloud deployment models . . . . . . . . . . . . . . . . . 7
1.2.3 Security in the Cloud . . . . . . . . . . . . . . . . . . . 7
1.2.4 Open-source Cloud frameworks . . . . . . . . . . . . . 8
1.2.4.1 OpenStack . . . . . . . . . . . . . . . . . . . 8
1.2.4.2 Apache CloudStack . . . . . . . . . . . . . . 10
1.2.4.3 OpenNebula . . . . . . . . . . . . . . . . . . 10
1.2.5 Big data in the Cloud . . . . . . . . . . . . . . . . . . 12
1.2.5.1 Big data management in the Cloud . . . . . 12
1.2.5.2 Big data analytics in the Cloud . . . . . . . 12
1.3 Cloud Computing in Remote Sensing . . . . . . . . . . . . . 14
2 Remote Sensing Data Integration in a Cloud Computing
Environment 17
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Background on Architectures for Remote Sensing Data Integra-
tion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.1 Distributed integration of remote sensing data . . . . 18
2.2.2 OODT: a data integration framework . . . . . . . . . 19
2.3 Distributed Integration of Multi-Source Remote Sensing Data 20
2.3.1 The ISO 19115-based metadata transformation . . . . 20
2.3.2 Distributed multi-source remote sensing data integration 22
v
vi Contents
2.4 Experiment and Analysis . . . . . . . . . . . . . . . . . . . . 24
2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3 Remote Sensing Data Organization and Management in a
Cloud Computing Environment 29
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Preliminaries and Related Techniques . . . . . . . . . . . . . . 31
3.2.1 Spatial organization of remote sensing data . . . . . . . 31
3.2.2 MapReduce and Hadoop . . . . . . . . . . . . . . . . . 32
3.2.3 HBase . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2.4 Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 LSI Organization Model of Multi-Source Remote Sensing Data 35
3.4 Remote Sensing Big Data Management in a Parallel File System 38
3.4.1 Full-text index of multi-source remote sensing metadata 38
3.4.2 Distributed data retrieval . . . . . . . . . . . . . . . . 40
3.5 Remote Sensing Big Data Management in the Hadoop Ecosys-
tem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.5.1 Data organization and storage component . . . . . . . 42
3.5.2 Data index and search component . . . . . . . . . . . 43
3.6 Metadata Retrieval Experiments in a Parallel File System . . 45
3.6.1 LSI model-based metadata retrieval experiments in a
parallel file system . . . . . . . . . . . . . . . . . . . . 45
3.6.2 Comparative experiments and analysis . . . . . . . . . 48
3.6.2.1 Comparative experiments . . . . . . . . . . . 48
3.6.2.2 Results analysis . . . . . . . . . . . . . . . . 49
3.7 Metadata Retrieval Experiments in the Hadoop Ecosystem . . 51
3.7.1 Time comparisons of storing metadata in HBase . . . 52
3.7.2 Time comparisons of loading metadata from HBase to
Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . 52
3.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4 High Performance Remote Sensing Data Processing in a
Cloud Computing Environment 55
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2 High Performance Computing for RS Big Data: State of the Art 58
4.2.1 Cluster computing for RS data processing . . . . . . . 58
4.2.2 Cloud computing for RS data processing . . . . . . . . 59
4.2.2.1 Programming models for big data . . . . . . 60
4.2.2.2 Resource management and provisioning . . . 60
4.3 Requirements and Challenges: RSCloud for RS Big Data . . . 61
4.4 pipsCloud: High Performance Remote Sensing Clouds . . . . 62
4.4.1 The system architecture of pipsCloud . . . . . . . . . 63
4.4.2 RS data management and sharing . . . . . . . . . . . 65
Contents vii
4.4.2.1 HPGFS: distributed RS data storage with
application-aware data layouts and copies . . . 67
4.4.2.2 RS metadata management with NoSQL
database . . . . . . . . . . . . . . . . . . . . 68
4.4.2.3 RS data index with Hilbert R+tree . . . . . 69
4.4.2.4 RS data subscription and distribution . . . . . 71
4.4.3 VE-RS: RS-specific HPC environment as a service . . 72
4.4.3.1 On-demand HPC cluster platforms with
bare-metal provisioning . . . . . . . . . . . . 73
4.4.3.2 Skeletal programming for RS big data process-
ing. . . . . . . . . . . . . . . . . . . . . . . . 76
4.4.4 VS-RS: Cloud-enabled RS data processing system . . . 77
4.4.4.1 Dynamic workflow processing for RS applica-
tions in the Cloud . . . . . . . . . . . . . . . 78
4.5 Experiments and Discussion . . . . . . . . . . . . . . . . . . 82
4.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5 Programming Technologies for High Performance Remote
SensingDataProcessinginaCloudComputingEnvironment 89
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.3 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . 92
5.3.1 Massive RS data . . . . . . . . . . . . . . . . . . . . . 92
5.3.2 Parallel programmability . . . . . . . . . . . . . . . . 93
5.3.3 Data processing speed . . . . . . . . . . . . . . . . . . 94
5.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 94
5.4.1 Generic algorithm skeletons for remote sensing applica-
tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4.1.1 Categories of remote sensing algorithms . . . 98
5.4.1.2 Generic RS farm-pipeline skeleton . . . . . . 98
5.4.1.3 Generic RS image-wrapper skeleton . . . . . 102
5.4.1.4 Generic feature abstract skeleton . . . . . . . 105
5.4.2 Distributed RS data templates . . . . . . . . . . . . . 108
5.4.2.1 RSData templates . . . . . . . . . . . . . . . 108
5.4.2.2 Dist RSData templates . . . . . . . . . . . . . 111
5.5 Experiments and Discussion . . . . . . . . . . . . . . . . . . 115
5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6 Construction and Management of Remote Sensing Produc-
tion Infrastructures across Multiple Satellite Data Centers 121
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.3 Infrastructures Overview . . . . . . . . . . . . . . . . . . . . 124
viii Contents
6.3.1 Target environment . . . . . . . . . . . . . . . . . . . 124
6.3.2 MDCPS infrastructures overview . . . . . . . . . . . . 125
6.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 128
6.4.1 Data management . . . . . . . . . . . . . . . . . . . . 128
6.4.1.1 Spatial metadata management for
co-processing . . . . . . . . . . . . . . . . . . 130
6.4.1.2 Distributed file management . . . . . . . . . . 131
6.4.2 Workflow management . . . . . . . . . . . . . . . . . . 133
6.4.2.1 Workflow construction . . . . . . . . . . . . . 136
6.4.2.2 Task scheduling . . . . . . . . . . . . . . . . . 137
6.4.2.3 Workflow fault-tolerance . . . . . . . . . . . . 141
6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.5.1 Related experiments on dynamic data management . . 142
6.5.2 Related experiments on workflow management . . . . 146
6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.6.1 System architecture . . . . . . . . . . . . . . . . . . . . 147
6.6.2 System feasibility . . . . . . . . . . . . . . . . . . . . . 148
6.6.3 System scalability . . . . . . . . . . . . . . . . . . . . 148
6.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . 148
7 Remote Sensing Product Production in an OpenStack-Based
Cloud Computing Environment 151
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.2 Background and Related Work . . . . . . . . . . . . . . . . . 153
7.2.1 Remote sensing products . . . . . . . . . . . . . . . . 153
7.2.1.1 Fine processing products . . . . . . . . . . . 154
7.2.1.2 Inversion index products . . . . . . . . . . . 154
7.2.1.3 Thematic products. . . . . . . . . . . . . . . 154
7.2.2 Remote sensing production system . . . . . . . . . . . 155
7.3 Cloud-Based Remote Sensing Production System . . . . . . 156
7.3.1 Program framework . . . . . . . . . . . . . . . . . . . 156
7.3.2 System architecture . . . . . . . . . . . . . . . . . . . . 157
7.3.3 Knowledge base and inference rules. . . . . . . . . . . 159
7.3.3.1 The upper and lower hierarchical relationship
database . . . . . . . . . . . . . . . . . . . . 159
7.3.3.2 Input/output database of every kind of remote
sensing product . . . . . . . . . . . . . . . . 160
7.3.3.3 Inference rules for production demand data
selection . . . . . . . . . . . . . . . . . . . . . 161
7.3.3.4 Inference rules for workflow organization . . . 161
7.3.4 Business logic . . . . . . . . . . . . . . . . . . . . . . . 162
7.3.5 Active service patterns . . . . . . . . . . . . . . . . . . 165
7.4 Experiment and Case Study . . . . . . . . . . . . . . . . . . . 167
7.4.1 Global scale remote sensing production . . . . . . . . . 167
Contents ix
7.4.2 Regional scale mosaic production . . . . . . . . . . . . 168
7.4.3 Local scale change detection. . . . . . . . . . . . . . . 170
7.4.3.1 Remote sensing data cube. . . . . . . . . . . . 171
7.4.3.2 Local scale time-series production . . . . . . . 171
7.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
8 KnowledgeDiscoveryandInformationAnalysisfromRemote
Sensing Big Data 175
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
8.2 Preliminaries and Related Work . . . . . . . . . . . . . . . . 176
8.2.1 Knowledge discovery categories . . . . . . . . . . . . . 176
8.2.2 Knowledge discovery methods . . . . . . . . . . . . . . 178
8.2.3 Related work . . . . . . . . . . . . . . . . . . . . . . . 179
8.3 Architecture Overview . . . . . . . . . . . . . . . . . . . . . 180
8.3.1 Target data and environment . . . . . . . . . . . . . . 180
8.3.2 FRSDC architecture overview . . . . . . . . . . . . . . . 181
8.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 182
8.4.1 Feature data cube . . . . . . . . . . . . . . . . . . . . 182
8.4.1.1 Spatial feature object in FRSDC . . . . . . . 182
8.4.1.2 Data management . . . . . . . . . . . . . . . 182
8.4.2 Distributed executed engine . . . . . . . . . . . . . . . 184
8.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
8.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
9 AutomaticConstructionofCloudComputingInfrastructures
in Remote Sensing 191
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
9.2 Definition of the Remote Sensing Oriented Cloud Computing
Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9.2.1 Generally used cloud computing infrastructure . . . . 193
9.2.2 Remote sensing theme oriented cloud computing infras-
tructure . . . . . . . . . . . . . . . . . . . . . . . . . . 193
9.3 Design and Implementation of Remote Sensing Oriented Cloud
Computing Infrastructure . . . . . . . . . . . . . . . . . . . . 195
9.3.1 System architecture design . . . . . . . . . . . . . . . 195
9.3.2 System workflow design . . . . . . . . . . . . . . . . . 196
9.3.3 System module design . . . . . . . . . . . . . . . . . . 198
9.4 Key Technologies of Remote Sensing Oriented Cloud Infrastruc-
ture Automatic Construction . . . . . . . . . . . . . . . . . . 200
9.4.1 Automatic deployment based on OpenStack and Salt-
Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
9.4.2 Resource monitoring based on Ganglia . . . . . . . . . 203
9.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205