toward accurate and efficient outlier detection in high - SMARTech

154 Pages·2007·1.48 MB·English

by  Minh Quoc Nguyen,,,

TOWARD ACCURATE AND EFFICIENT OUTLIER
DETECTION IN HIGH DIMENSIONAL AND LARGE
DATA SETS
A Thesis
Presented to
The Academic Faculty
by
Minh Quoc Nguyen
In Partial Fulfillment
of the Requirements for the Degree
Doctor of Philosophy in the
School of Computer Science
Georgia Institute of Technology
August 2010
TOWARD ACCURATE AND EFFICIENT OUTLIER
DETECTION IN HIGH DIMENSIONAL AND LARGE
DATA SETS
Approved by:
Professor Leo Mark, Advisor Professor Ling Liu
College of Computing College of Computing
Georgia Institute of Technology Georgia Institute of Technology
Professor Edward Omiecinski, Advisor Professor Jinho Kim
College of Computing Department of Computer Science
Georgia Institute of Technology Kangwon National University
Professor Sham Navathe Date Approved: 5 April 2010
College of Computing
Georgia Institute of Technology
To my parents,
my wife,
and my daughter.
iii
ACKNOWLEDGEMENTS
First andforemost, Iwould like to acknowledge my advisors, Professors Leo Mark and
Edward Omiecinski for their priceless effort in guiding me throughout my thesis. I
would like to attribute my thesis to my advisors for their support and encouragement.
I am also thankful for their patience in helping me prepare my resume for my job
search.
I would like to thank my committee members: Professors Leo Mark, Edward
Omiecinski, Sham Navathe, Ling Liu, and Jinho Kim. I am thankful for Navathe’s
advice on writing my thesis proposal. Mark, Omiecinski, and Navathe were helpful
with my teaching assistantship. In my first year, Liu and Navathe made me feel
comfortable to be at Georgia Tech with their warm welcome.
I also acknowledge the Vietnam Education Foundation (VEF) for providing fund-
ing during my first two years at Georgia Tech and for providing invaluable networking
to support my career.
I will never forget my labmates: Rocky, Saurav, Weiyun, Sangeetha, and Wei.
Your friendships made the lab my home. You have been very supportive.
I am thankful for my parents who always believe in me and encourage me to
pursue the path that I have chosen. And lastly, I owe my wife a special thank you
for her encouragement and endless support during my Ph.D. study. Her patience,
endurance, and love give me courage and concentration to reach my goal.
iv
TABLE OF CONTENTS
DEDICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
I INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Outlier Detection in Large and High Dimensional Data . . . . . . . 3
1.2 Detecting Patterns of Anomalous Activities . . . . . . . . . . . . . 4
1.3 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . 5
II RELATED WORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1 Tabular data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.1 Statistical Methods . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.2 Supervised Methods . . . . . . . . . . . . . . . . . . . . . . 9
2.1.3 Semi-supervised Methods . . . . . . . . . . . . . . . . . . . 11
2.1.4 Unsupervised Methods . . . . . . . . . . . . . . . . . . . . . 12
2.1.5 Distance-based and Density-based . . . . . . . . . . . . . . 14
2.2 Anomaly in Sequential Data . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 Detecting Unusual Sequences . . . . . . . . . . . . . . . . . 15
2.2.2 Detecting Unusual Subsequences in a Sequence . . . . . . . 17
2.2.3 Detecting Unusual Values in a Sequence . . . . . . . . . . . 18
2.3 Graph-based Anomaly . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Robust Outlier Methods . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
III EFFICIENT LOCAL DENSITY-BASED OUTLIER DETECTION USING
RANDOMIZATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
v
3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3 Local Outlier Factor . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.4 Generalized Local Density-based Outlier . . . . . . . . . . . . . . . 26
3.5 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.5.1 Query Time of New Point . . . . . . . . . . . . . . . . . . . 31
3.5.2 Time Complexity Analysis . . . . . . . . . . . . . . . . . . . 33
3.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.6.1 2D Example . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.6.2 Real Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.6.3 Dimensionality . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.6.4 Speed Comparison . . . . . . . . . . . . . . . . . . . . . . . 40
3.6.5 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.6.6 Convergence Rate . . . . . . . . . . . . . . . . . . . . . . . 41
3.6.7 Other Parameters . . . . . . . . . . . . . . . . . . . . . . . 42
3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
IV ANOMALOUS PATTERN DETECTION USING ADAPTIVE NEAREST
NEIGHBOR APPROACH . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.2 Adaptive Nearest Neighbors . . . . . . . . . . . . . . . . . . . . . . 44
4.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.1 LOF, KMEAN, SNN . . . . . . . . . . . . . . . . . . . . . . 52
4.3.2 Outcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
V ACCURATELY RANKING OUTLIERS WITH DELTA DEVIATION AC-
CUMULATION AND MIXTURE OF VARIANCES . . . . . . . . . . . 59
5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.1.1 Mixture of Variances . . . . . . . . . . . . . . . . . . . . . . 60
5.1.2 Accumulated Sub-dimensional Variations . . . . . . . . . . . 62
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
vi
5.3 Concept of Outlier in High Dimensions . . . . . . . . . . . . . . . . 66
5.3.1 Filtering and Normalization . . . . . . . . . . . . . . . . . . 68
5.4 Sub-dimensional Outlier . . . . . . . . . . . . . . . . . . . . . . . . 69
5.5 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.6.1 Synthetic Dataset . . . . . . . . . . . . . . . . . . . . . . . 79
5.6.2 KDD CUP ’99 Dataset . . . . . . . . . . . . . . . . . . . . . 81
5.6.3 The Effect of the Filter Parameter . . . . . . . . . . . . . . 84
5.6.4 Selecting ρ . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
0
5.6.5 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
VI EFFICIENTANOMALOUSCORRELATEDATTRIBUTEPATTERNDE-
TECTION IN MULTI-DIMENSIONAL DATA . . . . . . . . . . . . . . 90
6.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
6.2 Comparison with outlier detection and clustering . . . . . . . . . . 91
6.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.4 Relational Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.5 Core Rules and Patterns . . . . . . . . . . . . . . . . . . . . . . . . 96
6.6 Pattern Construction . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.7 Core Rule Construction . . . . . . . . . . . . . . . . . . . . . . . . 99
6.8 Attribute Selection for Core Rule . . . . . . . . . . . . . . . . . . . 103
6.9 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.9.1 Building Core Rules . . . . . . . . . . . . . . . . . . . . . . 105
6.9.2 Pruning Candidates . . . . . . . . . . . . . . . . . . . . . . 105
6.9.3 Interval-Tuple Construction . . . . . . . . . . . . . . . . . . 106
6.10 Parameter Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.11 Running-time Complexity . . . . . . . . . . . . . . . . . . . . . . . 108
6.12 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.12.1 Our Method . . . . . . . . . . . . . . . . . . . . . . . . . . 113
vii
6.13 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.14 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
VII CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
VIII FUTURE WORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
8.1 Randomized Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 122
8.2 Anomalous Pattern Detection Using Correlated Attributes . . . . . 123
8.3 Complexity in Data Model . . . . . . . . . . . . . . . . . . . . . . . 124
REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
VITA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
viii
LIST OF TABLES
1 Outlier detection methods categorized based on data type and the
availability of training data . . . . . . . . . . . . . . . . . . . . . . . 21
2 Outlier detection methods categorized based on distribution assump-
tion and the availability of training data . . . . . . . . . . . . . . . . 21
3 Magic, Physics, and KDD Cup ’99 dataset description . . . . . . . . . 37
4 Running time: Randomization versus LOF . . . . . . . . . . . . . . . 40
5 Top 10 LOF outliers in Sam’s club data . . . . . . . . . . . . . . . . . 53
6 Clusters in Sam’s club data generated by SNN . . . . . . . . . . . . 53
7 Interesting patterns clustered by Outcast . . . . . . . . . . . . . . . . 55
8 Notations and basic definitions . . . . . . . . . . . . . . . . . . . . . . 64
9 Seven outliers are detected . . . . . . . . . . . . . . . . . . . . . . . . 80
10 Nine outlier clusters are detected in 2D dataset . . . . . . . . . . . . 83
11 Detected attack connections in KDD CUP dataset . . . . . . . . . . . 84
12 Dimensional and aggregated outlier scores for p and p . . . . . . . 88
7 36
13 An example dataset contains three groups. . . . . . . . . . . . . . . . 92
14 A subset of correlated observations . . . . . . . . . . . . . . . . . . . 93
15 Statistics of X , X , X and X . . . . . . . . . . . . . . . . . . . . . 93
1 2 3 4
16 List of attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
17 Top outliers (10%) clustered by kmean . . . . . . . . . . . . . . . . . 111
18 Top outliers (10%) clustered by SNN . . . . . . . . . . . . . . . . . . 112
19 The results of clustering using SNN on the entire dataset . . . . . . . 112
20 Top 10 unusual patterns ordered by size . . . . . . . . . . . . . . . . 114
21 Other unusual patterns ordered by size . . . . . . . . . . . . . . . . . 114
22 Unusual patterns ordered by score . . . . . . . . . . . . . . . . . . . . 115
23 The recall rate of the attack types in the top 10 patterns . . . . . . . 115
ix
LIST OF FIGURES
1 An example of a PST tree . . . . . . . . . . . . . . . . . . . . . . . . 17
2 Outliers with respect to their local subspaces. . . . . . . . . . . . . . 27
3 Randomized local density-based outlier method . . . . . . . . . . . . 30
4 Compute candidate outliers . . . . . . . . . . . . . . . . . . . . . . . 31
5 Partition a set using random split points . . . . . . . . . . . . . . . . 31
6 Tree structure of randomized LOF . . . . . . . . . . . . . . . . . . . . 32
7 2D test data for local outlier . . . . . . . . . . . . . . . . . . . . . . . 35
8 Similarity in ranking for randomized outlier detection . . . . . . . . . 36
9 Effect of dimensionality on the detection rate. . . . . . . . . . . . . . 39
10 Running time: Randomization versus LOF . . . . . . . . . . . . . . . 40
11 Running time of the randomized method . . . . . . . . . . . . . . . . 41
12 Convergence rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
13 kth nearest neighbor of p and q . . . . . . . . . . . . . . . . . . . . . 45
14 Adaptive nearest neighbors of p and q . . . . . . . . . . . . . . . . . 45
15 Constructing adaptive nearest neighbors . . . . . . . . . . . . . . . . 50
16 Outcast pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
17 The average daily sales in January for all items . . . . . . . . . . . . 52
18 The average daily sale for all items and top outliers . . . . . . . . . . 54
19 The average sale volumne for each day in January . . . . . . . . . . . 56
20 The 2D outlier p . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
21 The outlier p is suppressed in the 3D space . . . . . . . . . . . . . . . 62
22 Clustering pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . 78
23 Points p ,p ,p and cluster C are generated outliers . . . . . . . . . 80
1 2 3 1
24 Detection curve of the filtering method on KDD Cup ’99 . . . . . . . 82
25 Detection rate for the algorithm with/without using the filter . . . . 85
26 Point p is not an outlier in this 2d-subspace . . . . . . . . . . . . . 86
36
27 Point p is an outlier in the 1st and 26th dimensions . . . . . . . . . . 87
36
x