Robust and Adversarial Data Mining

136 Pages·2015·2.1 MB·English

Copyright and use of this thesis
This thesis must be used in accordance with the
provisions of the Copyright Act 1968.
Reproduction of material protected by copyright
may be an infringement of copyright and
copyright owners may be entitled to take
legal action against persons who infringe their
copyright.
Section 51 (2) of the Copyright Act permits
an authorized officer of a university library or
archives to provide a copy (by communication
or otherwise) of an unpublished thesis kept in
the library or archives, to a person who satisfies
the authorized officer that he or she requires
the reproduction for the purposes of research
or study.
The Copyright Act grants the creator of a work
a number of moral rights, specifically the right of
attribution, the right against false attribution and
the right of integrity.
You may infringe the author’s moral rights if you:
- fail to acknowledge the author of this thesis if
you quote sections from the work
- attribute this thesis to another author
- subject this thesis to derogatory treatment
which may prejudice the author’s reputation
For further information contact the
University’s Copyright Service.
sydney.edu.au/copyright
ROBUST AND ADVERSARIAL DATA
MINING
Athesissubmittedinfulfilmentoftherequirementsforthe
degreeofDoctorofPhilosophyintheSchoolofInformationTechnologiesat
TheUniversityofSydney
FeiWang
August2015
(cid:13)c Copyright by Fei Wang 2015
All Rights Reserved
ii
Abstract
In the domain of data mining and machine learning, researchers have made significant
contributionsindevelopingalgorithmshandlingclusteringandclassificationproblems.
We develop algorithms under assumptions that are not met by previous works. (i) In
adversarial learning, which is the study of machine learning techniques deployed in
non-benign environments. We design an algorithm to show how a classifier should be
designedtoberobustagainstsparseadversarialattacks. Ourmaininsightisthatsparse
featureattacks arebest defendedby designingclassifiers whichuse(cid:96) regularizers. (ii)
1
The different properties between (cid:96) (Lasso) and (cid:96) (Tikhonov or Ridge) regularization
1 2
has been studied extensively. However, given a data set, principle to follow in terms of
choosingthesuitableregularizerisyettobedeveloped. Weusemathematicalproperties
of the two regularization methods followed by detailed experimentation to understand
their impact based on four characteristics. (iii) The identification of anomalies is an
inherent component of knowledge discovery. In lots of cases, the number of features
of a data set can be traced to a much smaller set of features. We claim that algorithms
applied in a latent space are more robust. This can lead to more accurate results, and
potentially provide a natural medium to explain and describe outliers. (iv) We also
apply data mining techniques on health care industry. In a lot cases, health insurance
companies cover unnecessary costs carried out by healthcare providers. The potential
adversarial behaviors of surgeon physicians are addressed. We describe a specific con-
text of private healthcare in Australia and describe our social network based approach
(applied to health insurance claims) to understand the nature of collaboration among
doctors treating hospital inpatients and explore the impact of collaboration on cost and
quality of care. (v) We further develop models that predict the behaviors of orthopedic
surgeonsinregardtosurgerytypeanduseofprostheticdevice. Animportantfeatureof
thesemodelsisthattheycannotonlypredictthebehaviorsofsurgeonsbutalsoprovide
explanationforthepredictions.
iii
Statement of Originality
Iherebycertifythatthisthesiscontainsnomaterialthathasbeenacceptedfortheaward
ofanyotherdegreeinanyuniversityorotherinstitution.
————————————————————-
FeiWang
March31,2015
iv
Acknowledgements
Firstly,Iwouldliketogivethankstomydearsupervisor,ProfessorSanjayChawla,for
his essential support and guidance during my research . The achievements during my
threeyearsresearchhasbeenanoutcomebyhisinvaluableandindispensableexpertise,
stimulationandencouragement. IwouldalsoliketothankmyindustrysupervisorUma
SrinivasanandBavaniArunasalamfortheirinvaluableadviceandprovisionofdomain
knowledge, and my associate supervisor, Dr. Anastasios Viglas, for his purport to my
comfortableresearchenvironment.
IwouldliketothankCapitalMarketsCRCforprovidingmewithaPhDscholarship.
Finally,Iwanttothankmyparentsfortheirinfinitesupportthroughoutmylifeoverseas.
v
Publications
Thisthesishasledtothefollowingpublications:
1. Wang, Fei, Sanjay Chawla, and Didi Surian. “Latent outlier detection and the
lowprecisionproblem.”ProceedingsoftheACMSIGKDDWorkshoponOutlier
DetectionandDescription. ACM,2013.
2. Wang, Fei, Sanjay Chawla, and Wei Liu. “Tikhonov or Lasso Regularization:
Which Is Better and When.” Tools with Artificial Intelligence (ICTAI), 2013
IEEE25thInternationalConferenceon. IEEE,2013.
3. Wang, Fei, Uma Srinivasan, Shahadat Uddin, and Sanjay Chawla. “Application
ofNetworkAnalysisonHealthcare.”AdvancesinSocialNetworksAnalysisand
Mining(ASONAM),2014IEEE/ACMInternationalConferenceon. IEEE,2014.
4. Wang, Fei, Wei Liu, and Sanjay Chawla. “On Sparse Feature Attacks in Adver-
sarial Learning.” IEEE International Conference on Data Mining series (ICDM),
2014.
vi
Contents
Abstract iii
StatementofOriginality iv
Acknowledgements v
Publications vi
ListofTables xiii
ListofFigures xvi
1 Introduction 1
1.1 ContributionsofthisThesis . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Background 5
2.1 RelatedLiterature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 AlgorithmOriented . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 HealthcareOriented . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 KeyConceptsandEvaluationMetrics . . . . . . . . . . . . . . . . . . 9
2.2.1 NonZero-sumGame . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.2 Logisticregression . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.3 Regularizations . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.4 Networkconcepts . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.5 ConfusionMatrix . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.6 ROC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
vii
3 AdversarialLearning 20
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2 SolvingTheNonZero-sumGame . . . . . . . . . . . . . . . . . . . . 23
3.2.1 Lassoandrobustregression . . . . . . . . . . . . . . . . . . . 24
3.2.2 Robustregressionandminimumbudgetofadversary . . . . . . 26
3.2.3 Evaluationofregularizer . . . . . . . . . . . . . . . . . . . . . 27
3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3.1.1 (USPS)DigitImagedataset . . . . . . . . . . . . . . 28
3.3.1.2 (Malinglist)Mailinglistdataset . . . . . . . . . . . . 28
3.3.1.3 Spambasedataset . . . . . . . . . . . . . . . . . . . 28
3.3.2 Whysparsefeatureattackfortheadversary? . . . . . . . . . . 29
3.3.2.1 Amorerealisticbehavior . . . . . . . . . . . . . . . 29
3.3.2.2 Leadstobetterclassifier . . . . . . . . . . . . . . . . 29
3.3.2.3 The game converges faster with less cost and feature
modifications . . . . . . . . . . . . . . . . . . . . . 30
3.3.3 Why(cid:96) regularizerfordataminer? . . . . . . . . . . . . . . . . 32
1
3.3.4 EvaluationofthebudgetMB . . . . . . . . . . . . . . . . . . . 34
3.3.5 Evaluatinglogisticlosswith(cid:96)d . . . . . . . . . . . . . . . . . . 35
2
3.3.6 Evaluatinglogisticlosswith(cid:96)d . . . . . . . . . . . . . . . . . . 36
1
3.3.7 Comparisonof(cid:96)d and(cid:96)d onlogisticloss . . . . . . . . . . . . . 37
1 2
3.3.8 Comparisonwithothermethods . . . . . . . . . . . . . . . . . 38
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4 TikhonovorLassoRegularization 40
4.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.3 NotationandSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.4 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.5 DecisionMap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.5.1 Algorithmicstability . . . . . . . . . . . . . . . . . . . . . . . 44
4.5.2 Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.5.3 Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.5.4 Shape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
viii
4.5.5 Decisionmap . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.6.1 DataSets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.6.2 StabilityandRobustness . . . . . . . . . . . . . . . . . . . . . 50
4.6.3 Shape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.6.4 CorrelationandShape . . . . . . . . . . . . . . . . . . . . . . 53
4.6.4.1 AllWeak . . . . . . . . . . . . . . . . . . . . . . . . 54
4.6.4.2 FewStrong . . . . . . . . . . . . . . . . . . . . . . . 54
4.6.4.3 SpamBase . . . . . . . . . . . . . . . . . . . . . . . 55
4.6.4.4 WebSpam . . . . . . . . . . . . . . . . . . . . . . . 55
4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5 LatentOutlierDetection 57
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.2 Themultiplesubspaceview . . . . . . . . . . . . . . . . . . . . . . . . 58
5.3 High-DimensionalAnomalies . . . . . . . . . . . . . . . . . . . . . . 59
5.4 MatrixFactorization . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.4.1 TheimpactofProjections . . . . . . . . . . . . . . . . . . . . 61
5.4.2 SensitivitytoOutliers . . . . . . . . . . . . . . . . . . . . . . 61
5.5 ExperimentsandResults . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.5.1 DataSets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6 NetworkAnalysisonHealthcare 73
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6.2 Collaborationinhealthcare . . . . . . . . . . . . . . . . . . . . . . . 75
6.3 SurgeonCollaborationnetworkDesign . . . . . . . . . . . . . . . . . . 76
6.3.1 DesignofCollaborationNetwork(CN) . . . . . . . . . . . . . 76
6.3.2 DesignofSurgeonCentricCollaborationNetwork(SCCN) . . . 78
6.4 Dataanalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.4.1 Datapreparation . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.4.1.1 Selectionofadmission-relatedvariables . . . . . . . 79
6.4.1.2 Selectionofnetworkvariables . . . . . . . . . . . . 79
ix