NEW OUTLIER DETECTION TECHNIQUES FOR DATA STREAMS Approved by

145 Pages·2016·2.77 MB·English

NEWOUTLIERDETECTION
TECHNIQUESFORDATASTREAMS
Approvedby:
Dr. MichaelHahsler
Dr. MargaretH.Dunham
Dr. SukumaranNair
Dr. JeffTian
Dr. PingGui
NEWOUTLIERDETECTION
TECHNIQUESFORDATASTREAMS
ADissertationPresentedtotheGraduateFacultyofthe
BobbyB.LyleSchoolofEngineering
SouthernMethodistUniversity
in
PartialFulfillmentoftheRequirements
forthedegreeof
DoctorofPhilosophy
witha
MajorinComputerScience
by
CharlieIsaksson
(M.S.C.S.,MidSwedenUniversity,2006)
Dec17,2016
ACKNOWLEDGMENTS
Iamtrulyhumbledandgratefulforthegreatnumberofindividualsthathassupported
and encouraged me over the past nine years to fulfill my biggest dream. Dr. Margaret H.
Dunham and Dr. Michael Hahsler have been my two mentors and friends throughout this
rewardingjourney. IwouldliketoextendspecialthankstoDr. Hahslerforhelpingmefind
mypathbackandrecognizeyourbackgroundknowledgeandpatience.
IwouldliketoextendmygratitudetothefacultyandstaffmembersintheDepartment
ofComputerScienceandEngineeringatSouthernMethodistUniversity.
Fortheothermembersofmydissertationcommittee,Dr. SukumaranNair,Dr. JeffTian
and Dr. Ping Gui, thank you for all the feedback and patience you had with me. I whole-
heartedly enjoyed the challenge of researching a critical issue that currently is important
forvariousindustries.
Finally, a special recognition goes out to my family and friends who supported and
encouragedmeduringmypursuitofthedoctorateincomputerscience. Thankstomykids
forgivingmethestrengthtokeepgoing. Iloveyoumorethanyouwilleverknow.
iii
Isaksson, Charlie M.S.C.S.,MidSwedenUniversity,2006
NewOutlierDetection
TechniquesForDataStreams
Advisor: Professor MichaelHahsler
DoctorofPhilosophydegreeconferredDec17,2016
DissertationcompletedNov09,2016
Theavailabilityandreliabilityofdatahavebecomeessentialinourmodernsociety. In
fact, it has become critical in every domain to maintain high-quality data even though that
datamayoriginateathighvelocityandinlargequantities. Todayitiswellunderstoodthat
data enables businesses to achieve their full potential by providing valuable insights into
their business as well as potentially offering them an advantage over their competitors. To
achievesuchagoalrequiresasignificantinvestmentinbothbigdatainfrastructureanddata
mining capabilities. Data Mining is the process of finding hidden patterns within a large
dataset. Imperative to Data Mining is the ability to detect outliers, data points that deviate
fromtherestofthedatapointsbecauseoutlierscandramaticallyaltertheresultoftheanal-
ysis. Although outliers occur infrequently, it is hard to identify them since there are many
potential sources for outliers (such as human errors, machine errors, environmental varia-
tions, faulty sensors). Finding outliers in large dataset requires extremely efficient outlier
detection techniques. It becomes even harder to detect an outlier within a data stream as it
imposes a single pass restriction and data often arrives at a very fast rate. Also streaming
data may contain redundant information, which can reduce outlier detection performance
and efficiency. To avoid this redundancy while maintaining the correctness of the data, it
becomes necessary to summarize the data stream. The Extensible Markov Model (EMM)
hasbeenproventobeagoodcandidateformeetingtheserequirementstodetectoutliersin
iv
data stream applications. EMM uses data stream clustering models and takes into account
temporalandorderingaspectsusingaMarkovChain(MC),apowerfultemporalmodelthat
allowsstudyingacomplexsystemandmakingpredictionsaboutevents. ExtensibleMarkov
model is a time changing MC that has the ability of learning and dynamically adapting its
structuretotheenvironmentaswellasupdatingthestatetransitionprobabilitybasedonthe
incoming data. The model generated by EMM allows analysis of a particular time frame
asanMC,and,astimepasses,thismodelwillcontinuetoadapt,evolve,andlearnwiththe
ongoingdatastream. ThisisduetotheclosecouplingoftheclusteringmodelwithanMC
model. Combining these two models delivers a spatiotemporal model that satisfies all the
requirements from a data stream (big data) infrastructure standpoint. In this dissertation,
the data pattern finding capability of EMM has been extended in several ways. Firstly, a
sophisticated mining task on the synopsis is investigated to detect Distributed Denial of
Service (DDOS) network intrusion. A performance study is then conducted of different
outlier detection techniques and compared with EMM, and this leads to two additional ex-
tensionstofurtherimproveEMM’sperformance. SO-Stream,anewself-organizingcluster
structure that allows the algorithm to obtain the threshold for each micro-cluster dynami-
cally, is proposed, and then SO-Stream is extended by integrating a Markov Model (MM).
The new algorithm is called Adaptive Streaming Markov Model (ASMM), which is de-
signed to handle concept drift, spatiotemporal outliers, and high volume and velocity data
streams while preserving higher accuracy and cluster quality. The dissertation concludes
with directions for future work including distributed ASMM’s that can be integrated into
big data frameworks, ASMM’s for telecom applications and a visualization technique for
multidimensionaldatathatisgreatlyneededforbetterinterpretationofoutliermodels.
v
TABLEOFCONTENTS
LISTOFFIGURES ............................................................... ix
LISTOFTABLES ................................................................ xi
CHAPTER
1. INTRODUCTION .......................................................... 1
1.1. Motivation ............................................................ 2
1.2. FocusoftheDissertationandConclusions .............................. 3
1.3. OrganizationoftheDissertation ........................................ 4
2. BACKGROUND............................................................ 6
2.1. OutlierDetection ...................................................... 6
2.2. DataStreams.......................................................... 8
2.3. OutlierDetectioninDataStreams ...................................... 9
2.4. SpatiotemporalOutlierDetectioninDataStream........................ 13
3. RISKLEVELINGOFNETWORKTRAFFICANOMALIES................. 17
3.1. Introduction........................................................... 17
3.2. RelatedWork.......................................................... 19
3.3. Methodology.......................................................... 23
3.4. ExperimentsandAnalysis.............................................. 31
3.4.1. Dataset........................................................ 31
3.4.2. Experiments ................................................... 32
3.5. ChapterSummary ..................................................... 34
4. ACOMPARATIVESTUDYOFOUTLIERDETECTIONALGORITHMS.... 36
vi
4.1. Introduction........................................................... 36
4.1.1. ExtensibleMarkovModel ...................................... 38
4.1.2. DensityBasedLocalOutliers(LOFApproach) .................. 41
4.1.3. DensityBasedLocalOutliers(LSC-MineApproach) ............ 41
4.2. ExperimentalResults .................................................. 42
4.2.1. TimeAnalysis ................................................. 43
4.2.2. ExperimentsonRealLifeDataandSyntheticDatasets........... 45
4.3. ChapterSummary ..................................................... 48
5. SO-STREAM: SELF ORGANIZING DENSITY-BASED CLUSTERING
OVERDATASTREAM..................................................... 56
5.1. Introduction........................................................... 56
5.2. RelatedWork.......................................................... 58
5.3. Methodology.......................................................... 61
5.3.1. SOStreamOverview ........................................... 61
5.3.2. Density-BasedCentroid ........................................ 62
5.3.3. SOStreamAlgorithm. .......................................... 65
5.3.4. OnlineMerging................................................ 67
5.4. Experiments........................................................... 71
5.4.1. SyntheticData................................................. 71
5.4.2. Real-WorldDataset ............................................ 72
5.4.3. ParameterAnalysis ............................................ 74
5.4.4. ScalabilityandComplexityofSOStream........................ 76
5.5. ChapterSummary ..................................................... 78
vii
6. ASMM:DETECTINGSPETIO-TEMPORALOUTLIERSWITHADAP-
TIVESTREAMINGMARKOVMODEL .................................... 79
6.1. Introduction........................................................... 79
6.2. Methodology.......................................................... 86
6.2.1. ExtensibleMarkovModelAlgorithm ........................... 86
6.2.2. AdaptiveStreamingMarkovModelAlgorithm .................. 90
6.2.3. EMMRare..................................................... 99
6.3. Experiments...........................................................100
6.3.1. Datasets .......................................................101
6.3.2. ParameterAnalysis ............................................112
6.3.3. ScalabilityandComplexityofASMM ..........................113
6.4. ChapterSummary .....................................................114
7. CONCLUSION.............................................................118
7.1. SummaryofResults ...................................................118
7.2. DirectionsforFutureResearch .........................................121
APPENDIX
REFERENCES ...................................................................123
viii
LISTOFFIGURES
Figure Page
2.1 Aclassificationofoutlierdetectiontechniques............................ 6
2.2 Aworkflowfromatraditionalspatiotemporaloutlierdetectionframework. .. 14
2.3 TheworkflowfromEMMoutlierdetectionframework. .................... 16
3.1 LogarithmoftrafficvolumeshowstheDDoSattacks ...................... 33
4.1 AdvantagesoftheLOFapproach. Modifiedfrom[78]..................... 42
4.2 Run time for LOF, LSC-Mine, and EMM with MinPts=20 and EMM
Threshold=0.99. ................................................... 44
5.1 (a)Datapointsofstreamwith5overlappingclustersand(b)ShowSOStream
capability to distinguish overlapped clusters. For visualizing cluster
structure,wedonotutilizeFadingorMerging. ....................... 72
5.2 SOStreamclusteringqualityhorizon=1K,Streamspeed=1K.Thequal-
ityEvaluationforMR-StreamandD-Streamisretrievedfrom[68]...... 73
5.3 SOStreammemorycostoverthelengthofthedatastream. TheMemory
EvaluationforMR-Streamisretrievedfrom[68]. ..................... 76
5.4 SOStream execute time using high dimensional KDD CUP99 dataset
with34numericalattributes. Thesamplingdatarateisevery25Kpoints. 77
6.1 ExampleofEMMdirectedgraph. ....................................... 87
6.2 BasicexamplethatshowhighleveloperationsofASMM. ................. 90
6.3 The Sensors were arranged in the lab according to the above diagram.
Obtainedfrom[88] ................................................ 102
6.4 Show subplotsfrom time period[8398:9000]. It is evidentthat humidity
suffers from spatial outlier. However, due to large data size we are
unabletodisplaytemporaloutliers................................... 109
ix
6.5 Show subplots from normalized Server System Health dataset. Hence,
thehighlightedredareaincludeboththespatialandtemporaloutliers. .. 110
6.6 DistributionofASMM’sclusterscountbasedondifferentbuffersizefor
theKDDCUP’99data.............................................. 116
6.7 Clusterssizedecreaseswithincreasedbuffersize. Thenumberofclusters
stabilizesbetweenbuffersize15to35................................ 117
6.8 ASMM and EMM memory cost over different threshold values using
KDDCUP99dataset. .............................................. 117
6.9 ASMMandEMMexecutetimeusinghighdimensionalKDDCUP99dataset. 117
x