Big Data Analytics: Methods and Applications

278 Pages·2016·7.33 MB·English

by  Saumyadipta Pyne

Saumyadipta Pyne · B.L.S. Prakasa Rao
S.B. Rao Editors
Big Data
Analytics
Methods and Applications
Big Data Analytics
⋅
Saumyadipta Pyne B.L.S. Prakasa Rao
S.B. Rao
Editors
Big Data Analytics
Methods and Applications
123
Editors
SaumyadiptaPyne S.B.Rao
Indian Institute of Public Health CRRaoAIMSCS
Hyderabad University of HyderabadCampus
India Hyderabad
India
B.L.S. PrakasaRao
CRRaoAIMSCS
University of HyderabadCampus
Hyderabad
India
ISBN978-81-322-3626-9 ISBN978-81-322-3628-3 (eBook)
DOI 10.1007/978-81-322-3628-3
LibraryofCongressControlNumber:2016946007
©SpringerIndia2016
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
foranyerrorsoromissionsthatmayhavebeenmade.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringer(India)Pvt.Ltd.
Theregisteredcompanyaddressis:7thFloor,VijayaBuilding,17BarakhambaRoad,NewDelhi110001,India
Foreword
Bigdataistransformingthetraditionalwaysofhandlingdatatomakesenseofthe
world from which it is collected. Statisticians, for instance, are used to developing
methods for analysis of data collected for a specific purpose in a planned way.
Sample surveys and design of experiments are typical examples.
Big data, in contrast, refers to massive amounts of very high dimensional and
even unstructured data which are continuously produced and stored with much
cheaper cost than they are used to be. High dimensionality combined with large
sample size creates unprecedented issues such as heavy computational cost and
algorithmic instability.
Themassivesamples inbigdata aretypicallyaggregatedfrom multiplesources
at different time points using different technologies. This can create issues of
heterogeneity, experimental variations, and statistical biases, and would therefore
require the researchers and practitioners to develop more adaptive and robust
procedures.
Toward this, I am extremely happy to see in this title not just a compilation of
chapterswrittenbyinternationalexpertswhoworkindiversedisciplinesinvolving
BigData,butalsoararecombination,withinasinglevolume,ofcutting-edgework
in methodology, applications, architectures, benchmarks, and data standards.
Iamcertainthatthetitle,editedbythreedistinguishedexpertsintheirfields,will
informandengagethemindofthereaderwhileexploringanexcitingnewterritory
in science and technology.
Calyampudi Radhakrishna Rao
C.R. Rao Advanced Institute of Mathematics,
Statistics and Computer Science,
Hyderabad, India
v
Preface
The emergence of the field of Big Data Analytics has prompted the practitioners
andleadersinacademia,industry,andgovernmentsacrosstheworldtoaddressand
decide on different issues in an increasingly data-driven manner. Yet, often Big
Datacouldbetoocomplextobehandledbytraditionalanalyticalframeworks.The
variedcollectionofthemescoveredinthistitleintroducesthereadertotherichness
of the emerging field of Big Data Analytics in terms of both technical methods as
well as useful applications.
The idea of this title originated when we were organizing the “Statistics 2013,
InternationalConferenceonSocio-EconomicChallengesandSustainableSolutions
(STAT2013)” at the C.R. Rao Advanced Institute of Mathematics, Statistics and
Computer Science (AIMSCS) in Hyderabad to mark the “International Year of
Statistics”inDecember2013.Astheconvener,Prof.SaumyadiptaPyneorganized
a special session dedicated to lectures by several international experts working on
large data problems, which ended with a panel discussion on the research chal-
lenges and directions in this area. Statisticians, computer scientists, and data ana-
lysts from academia, industry and government administration participated in a
lively exchange.
Following the success of that event, we felt the need to bring together a col-
lection of chapters written by Big Data experts in the form of a title that can
combine new algorithmic methods, Big Data benchmarks, and various relevant
applications from this rapidly emerging area of interdisciplinary scientific pursuit.
The presenttitle combines some ofthekeytechnical aspectswith case studies and
domain applications, which makes the materials more accessible to the readers. In
fact, when Prof. Pyne taught his materials in a Master’s course on “Big and
High-dimensional Data Analytics” at the University of Hyderabad in 2013 and
2014, it was well-received.
vii
viii Preface
We thank all the authors of the chapters for their valuable contributions to this
title.Also,Wesincerelythankallthereviewersfortheirvaluabletimeanddetailed
comments. We also thank Prof. C.R. Rao for writing the foreword to the title.
Hyderabad, India Saumyadipta Pyne
June 2016 B.L.S. Prakasa Rao
S.B. Rao
Contents
Big Data Analytics: Views from Statistical and Computational
Perspectives... .... .... .... ..... .... .... .... .... .... ..... .... 1
Saumyadipta Pyne, B.L.S. Prakasa Rao and S.B. Rao
Massive Data Analysis: Tasks, Tools, Applications, and Challenges.... 11
Murali K. Pusala, Mohsen Amini Salehi, Jayasimha R. Katukuri,
Ying Xie and Vijay Raghavan
Statistical Challenges with Big Data in Management Science ..... .... 41
Arnab Laha
Application of Mixture Models to Large Datasets . .... .... ..... .... 57
Sharon X. Lee, Geoffrey McLachlan and Saumyadipta Pyne
An Efficient Partition-Repetition Approach in Clustering
of Big Data ... .... .... .... ..... .... .... .... .... .... ..... .... 75
Bikram Karmakar and Indranil Mukhopadhayay
Online Graph Partitioning with an Affine Message Combining
Cost Function . .... .... .... ..... .... .... .... .... .... ..... .... 95
Xiang Chen and Jun Huan
Big Data Analytics Platforms for Real-Time Applications in IoT.. .... 115
Yogesh Simmhan and Srinath Perera
Complex Event Processing in Big Data Systems... .... .... ..... .... 137
Dinkar Sitaram and K.V. Subramaniam
Unwanted Traffic Identification in Large-Scale University
Networks: A Case Study. .... ..... .... .... .... .... .... ..... .... 163
Chittaranjan Hota, Pratik Narang and Jagan Mohan Reddy
Application-Level Benchmarking of Big Data Systems . .... ..... .... 189
Chaitanya Baru and Tilmann Rabl
ix
x Contents
Managing Large-Scale Standardized Electronic Health Records... .... 201
Shivani Batra and Shelly Sachdeva
Microbiome Data Mining for Microbial Interactions and
Relationships.. .... .... .... ..... .... .... .... .... .... ..... .... 221
Xingpeng Jiang and Xiaohua Hu
A Nonlinear Technique for Analysis of Big Data in Neuroscience . .... 237
Koel Das and Zoran Nenadic
Big Data and Cancer Research .... .... .... .... .... .... ..... .... 259
Binay Panda
About the Editors
Saumyadipta Pyne is Professor at the Public Health Foundation of India, at the
Indian Institute of Public Health, Hyderabad, India. Formerly, he was P.C.
MahalanobisChairProfessorandheadofBioinformaticsattheC.R.RaoAdvanced
Institute of Mathematics, Statistics and Computer Science. He is also Ramalin-
gaswamiFellowofDepartmentofBiotechnology,theGovernmentofIndia,andthe
founderchairmanoftheComputerSocietyofIndia’sSpecialInterestGrouponBig
Data Analytics. Professor Pyne has promoted research and training in Big Data
Analytics, globally, including as the workshop co-chair of IEEE Big Data in 2014
and2015heldintheU.S.A.HisresearchinterestsincludeBigDataproblemsinlife
sciencesandhealthinformatics,computationalstatisticsandhigh-dimensionaldata
modeling.
B.L.S.PrakasaRao istheRamanujanChairProfessorattheC.R.RaoAdvanced
Institute of Mathematics, Statistics and Computer Science, Hyderabad, India.
Formerly, he was director at the Indian Statistical Institute, Kolkata, and the Homi
BhabhaChairProfessorattheUniversityofHyderabad.HeisaBhatnagarawardee
from the Government of India, fellow of all the three science academies in India,
fellowofInstituteofMathematicalStatistics,U.S.A.,andarecipientofthenational
awardinstatisticsinmemoryofP.V.SukhatmefromtheGovernmentofIndia.He
has also received the Outstanding Alumni award from Michigan State University.
With over 240 papers published in several national and international journals of
repute, Prof. Prakasa Rao is the author or editor of 13 books, and member of the
editorial boards of several national and international journals. He was, most
recently, the editor-in-chieffor journals—Sankhya A and Sankhya B. His research
interests include asymptotic theory of statistical inference, limit theorems in
probability theory and inference for stochastic processes.
S.B. Rao was formerly director of the Indian Statistical Institute, Kolkata, and
director of the C.R. Rao Advanced Institute of Mathematics, Statistics and Com-
puter Science, Hyderabad. His research interests include theory and algorithms in
graph theory, networks and discrete mathematics with applications in social,
xi