Data Management and Data Processing Support on Array-Based Scientific Data

267 Pages·2015·2.26 MB·English

Data Management and Data Processing Support on
Array-Based Scientific Data
Dissertation
Presented in Partial Fulfillment of the Requirements for the Degree Doctor
of Philosophy in the Graduate School of The Ohio State University
By
Yi Wang, B.S., M.S.
Graduate Program in Computer Science and Engineering
The Ohio State University
2015
Dissertation Committee:
Gagan Agrawal,Advisor
Feng Qin
SrinivasanParthasarathy
© Copyrightby
YiWang
2015
Abstract
Scientific simulations are now being performed at finer temporal and spatial scales,
leading to an explosionof the output data (mostly in array-based formats), and challenges
in effectivelystoring,managing,querying,disseminating,analyzing, and visualizingthese
datasets. Many paradigmsand tools used today for large-scale scientificdata management
and data processing are often too heavy-weight and have inherent limitations, making it
extremelyhard tocopewiththe‘bigdata’challenges inavarietyofscientificdomains.
Our overallgoal is to providehigh-performancedata management and data processing
support on array-based scientific data, targeting data-intensive applications and various
scientific array storages. We believe that such high-performance support can significantly
reduce the prohibitively expensive costs of data translation, data transfer, data ingestion,
dataintegration,dataprocessing,anddatastorageinvolvedinmanyscientificapplications,
leadingto betterperformance, ease-of-use, and responsiveness.
On one hand, we have investigated four data management topics as follows. First, we
builta light-weightdatamanagement layeroverscientificdatasetsstored in HDF5 format,
which is one of the popular array formats. Unlike many popular data transport protocols
such as OPeNDAP, which requires costly data translation and data transfer before
accessing remote data, our implementation can support server-side flexible subsetting and
aggregation,withhighparallelefficiency. Second, toavoidthehighupfrontdataingestion
costs of loading large-scale array data into array databases like SciDB, we designed a
ii
system referred to as SAGA, which can provide database-like support over native array
storage. Specifically, we focused on implementing a number of structural (grid, sliding,
hierarchical, and circular) aggregations, which are unique in array data model. Third, we
proposed a novel approximate aggregation approach over array data using bitmap
indexing. This approach can operate on the compact bitmap indices rather than the
original raw datasets, and can support fast, accurate and flexible aggregations over any
array or its subset without data reorganization. Fourth, we extended bitmap indexing to
assist the data mining task subgroup discovery over array data. Like the aggregation
approach, our algorithm can operate entirely on bitmap indices, and it can efficiently
handle a key challenge associated with array data – a subgroup identified over array data
can bedescribed byvalue-based and/ordimension-basedattributes.
On theotherhand, we focused on bothofflineand in-situdataprocessingparadigmsin
the context of MapReduce. To process disk-resident scientific data in various data
formats, we developed a customizable MapReduce-like framework, SciMATE, which can
be adapted to support transparent processing on any of the scientific data formats. Thus,
unnecessary data integration and data reloading incurred by applying traditional
MapReduce paradigm to scientific data processing can be avoided. We then designed
another MapReduce-like framework, Smart, to support efficient in-situ scientific analytics
in both time sharing and space sharing modes. In contrast to offline processing, our
implementation can avoid, either completely or to a very large extent, both data transfer
and datastoragecosts.
iii
This isdedicatedto theones Ilove: myparentsandmyfiancee.
iv
Acknowledgments
I would never have been able to complete my dissertation without the guidance of my
committee members, help from friends, and support from my family and fiancee over the
years.
Foremost, I would like to express my sincerest gratitude to my advisor Prof. Gagan
Agrawal for his continuous support of my Ph.D career. He is the researcher with the
broadest interests I have ever seen, and he has offered me excellent opportunities to delve
into a variety of research topics, including high-performance computing, programming
models, databases, data mining, and bioinformatics. This helps me to learn diverse
techniques,gaindifferentmethodologies,anddevelopmypotentialinanall-aroundway. I
am also grateful that he is not a harsh boss who pushes immense workload with close
supervision, but a thoughtful mentor who teaches me how to be self-motivated and enjoy
the research. My 5-year exploration would not have been so rewarding without his
patience, encouragement,and advice.
Besides my advisor, I would like to thank my candidacy and dissertation committee
members, Prof. Feng Qin, Prof. Srinivasan Parthasarathy, Prof. Han-Wei Shen, and Prof.
Spyros Blanas, for their valuable time and insightful advices. I would also like to thank
Dr. Arnab Nandi, Dr. Kun Huang, Dr. Hatice Gulcin Ozer, Dr. Xin Gao, Dr. Jim-Jingyan
Wang for the collaborations during my PhD career. I also need to acknowledge a CSE
v
staff, Aaron Jenkins, and I will always remember the nice time I worked with him in my
first quarter.
My sincere thanks also go to my hosts Dr. Marco Yu and Dr. Donghua Xu, and my
mentorDr. NanBoden,fortheirhelpinbothworkandlifeduringthesummerinternshipat
Google. Undertheirguidance,Ilearneddiversetechniquesfastandwasevenabletoapply
themtotheresearchprojectsaftertheinternship. Workingwiththemwasquitearewarding
experience that I might not be able to obtain from academia. The insightful discussions
with them can always bring me some inspirations of interesting research directions, some
of which I have already fulfilled in the last year of my PhD study. Their enthusiasm and
charms letmemakethedecisionto jointhesameteamat Googleafter graduation.
I also would like to thank the colleagues in our group, Data-Intensive and High
Performance Computing Research Group. My Ph.D. career would not have been so
colorful without the friendship from them. Former and present members of our lab have
provided me with numerous help, in both daily life and work, and they are: Wenjing Ma,
Vignesh Ravi, Tantan Liu, Tekin Bicer, Jiedan Zhu, Vignesh San, Xin Huo, Ashish
Nagavaram, Liyang Tang, Yu Su, Linchuan Chen, Mehmet Can Kurt, Mucahid Kutlu,
Jiaqi Liu, Roee Ebenstein, Sameh Shohdy, Gangyi Zhu, Peng Jiang, Jiankai Sun, Qiwei
Yang, andDavid Siegal.
Lastbutnottheleast,Iexpressspecialthankstomyparentsandmyfiancee. Myparents
havebeen always considerateand supportive. Myfiancee has been alwayscheering meup
and standingby methroughtheupsand downs.
vi
Vita
July29th,1987 ............................. Born -Wuhan, China
2010 .......................................B.S. SoftwareEngineering,
Wuhan University,Wuhan,China
2014 .......................................M.S. Computer Science & Engineering,
TheOhioStateUniversity
May 2014-August2014 .....................Software Engineer Intern, Google Inc.,
MountainView,CA
Publications
Research Publications
Yi Wang, Linchuan Chen, and Gagan Agrawal. “Supporting User-Defined Estimation
and Early Termination on a MapReduce-Like Framework for Online Analytics”. Under
Submission.
Yi Wang, Gagan Agrawal, Tekin Bicer, and Wei Jiang. “Smart: A MapReduce-Like
Framework for In-Situ Scientific Analytics”. In Proceeedings of the International
Conference for High Performance Computing, Networking, Storage and Analysis (SC),
November,2015.
Yi Wang, Gagan Agrawal, Tekin Bicer, and Wei Jiang. “Smart: A MapReduce-Like
Framework for In-Situ Scientific Analytics”. Technical Report, OSU-CISRC-4/15-TR05,
April,2015.
Yi Wang, Yu Su, Gagan Agrawal and Tantan Liu. “SciSD: Novel Subgroup Discovery
over Scientific Datasets Using Bitmap Indices”. Technical Report, OSU-CISRC-3/15-
TR03, March, 2015.
vii
Gangyi Zhu, Yi Wang, Gagan Agrawal. “SciCSM: Novel Contrast Set Mining over
Scientific Datasets Using Bitmap Indices (Short Paper)”. In Proceedings of the
International Conference on Scientific and Statistical Database Management (SSDBM),
June2015.
YiWang,YuSu,andGaganAgrawal. “ANovelApproachforApproximateAggregations
OverArrays”. In ProceedingsoftheInternationalConferenceon ScientificandStatistical
DatabaseManagement(SSDBM),June2015.
Yu Su, Yi Wang, and Gagan Agrawal. “In-Situ Bitmaps Generation and Efficient Data
Analysis based on Bitmaps”. In Proceedings of the International Symposium on High-
PerformanceParallelandDistributedComputing(HPDC), June2015.
Jim Jing-Yan Wang, Yi Wang, Shiguang Zhao, and Xin Gao. “Maximum Mutual
Information Regularized Classification”. In Engineering Applications of Artificial
Intelligence(ENG APPLARTIF INTEL), 2015.
YiWang,ArnabNandi,andGaganAgrawal. “SAGA:ArrayStorageasaDBwithSupport
forStructural Aggregations”. In Proceedingsof theInternationalConferenceon Scientific
andStatisticalDatabaseManagement(SSDBM),June2014.
Yi Wang, Gagan Agrawal, Gulcin Ozer, and Kun Huang “Removing Sequential
Bottlenecks in Analysis of Next-Generation Sequencing Data”. In Proceedings of the
International Workshop on High Performance Computational Biology (HiCOMB) in
Conjunction with International Parallel & Distributed Processing Symposium (IPDPS),
May 2014.
YuSu, YiWang,GaganAgrawal,andRajkumarKettimuthu. “SDQuery DSI:Integrating
DataManagement Support witha WideArea DataTransferProtocol”. In Proceeedingsof
the International Conference for High Performance Computing, Networking, Storage and
Analysis(SC), November,2013.
Yi Wang, Yu Su, and Gagan Agrawal. “Supporting a Light-Weight Data Management
LayerOverHDF5”. InProceedingsoftheInternationalSymposiumonCluster,Cloudand
GridComputing(CCGrid), May,2013.
Yi Wang, Wei Jiang, and Gagan Agrawal. “SciMATE: A Novel MapReduce-Like
Framework for Multiple Scientific Data Formats”. In Proceedings of the International
SymposiumonCluster,Cloud andGridComputing(CCGrid), May,2012.
viii
Fields of Study
MajorField: ComputerScience and Engineering
ix