Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond

638 Pages·2002·9.036 MB·English

by  Bernhard Schlkopf, Alexander J. Smola

Learning with Kernels
SupportVectorMachines,Regularization,Optimization,andBeyond
BernhardScho¨lkopf
AlexanderJ.Smola
TheMITPress
Cambridge,Massachusetts
London,England
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30
c2002MassachusettsInstituteofTechnology
(cid:1)
PrintedandboundintheUnitedStatesofAmerica
LibraryofCongressCataloging-in-PublicationData
LearningwithKernels—SupportVectorMachines,
Regularization,OptimizationandBeyond/byBernhardScho¨lkopf,
AlexanderJ.Smola.
p. cm.
Includesbibliographicalreferencesandindex.
ISBN0-262-19475-9(alk.paper)
1.Machinelearning.2.Algorithms.3.Kernelfunctions
I.Scho¨lkopf,Bernhard.II.Smola,AlexanderJ.
Contents
SeriesForeword xiii
Preface xv
1 ATutorialIntroduction 1
1.1 DataRepresentationandSimilarity . . . . . . . . . . . . . . . . . . . 1
1.2 ASimplePatternRecognitionAlgorithm . . . . . . . . . . . . . . . 4
1.3 SomeInsightsFromStatisticalLearningTheory . . . . . . . . . . . 6
1.4 HyperplaneClassifiers . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5 SupportVectorClassification . . . . . . . . . . . . . . . . . . . . . . 15
1.6 SupportVectorRegression . . . . . . . . . . . . . . . . . . . . . . . . 17
1.7 KernelPrincipalComponentAnalysis . . . . . . . . . . . . . . . . . 19
1.8 EmpiricalResultsandImplementations . . . . . . . . . . . . . . . . 21
I CONCEPTSANDTOOLS 23
2 Kernels 25
2.1 ProductFeatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2 TheRepresentationofSimilaritiesinLinearSpaces. . . . . . . . . . 29
2.3 ExamplesandPropertiesofKernels . . . . . . . . . . . . . . . . . . 45
2.4 TheRepresentationofDissimilaritiesinLinearSpaces . . . . . . . . 48
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3 RiskandLossFunctions 61
3.1 LossFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.2 TestErrorandExpectedRisk . . . . . . . . . . . . . . . . . . . . . . 65
3.3 AStatisticalPerspective . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4 RobustEstimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4 Regularization 87
4.1 TheRegularizedRiskFunctional . . . . . . . . . . . . . . . . . . . . 88
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30
4.2 TheRepresenterTheorem . . . . . . . . . . . . . . . . . . . . . . . . 89
4.3 RegularizationOperators . . . . . . . . . . . . . . . . . . . . . . . . 92
4.4 TranslationInvariantKernels . . . . . . . . . . . . . . . . . . . . . . 96
4.5 TranslationInvariantKernelsinHigherDimensions . . . . . . . . . 105
4.6 DotProductKernels . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.7 Multi-OutputRegularization . . . . . . . . . . . . . . . . . . . . . . 113
4.8 SemiparametricRegularization . . . . . . . . . . . . . . . . . . . . . 115
4.9 CoefficientBasedRegularization . . . . . . . . . . . . . . . . . . . . 118
4.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.11 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5 ElementsofStatisticalLearningTheory 125
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.2 TheLawofLargeNumbers . . . . . . . . . . . . . . . . . . . . . . . 128
5.3 WhenDoesLearningWork:theQuestionofConsistency . . . . . . 131
5.4 UniformConvergenceandConsistency . . . . . . . . . . . . . . . . 131
5.5 HowtoDeriveaVCBound . . . . . . . . . . . . . . . . . . . . . . . 134
5.6 AModelSelectionExample . . . . . . . . . . . . . . . . . . . . . . . 144
5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6 Optimization 149
6.1 ConvexOptimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
6.2 UnconstrainedProblems . . . . . . . . . . . . . . . . . . . . . . . . . 154
6.3 ConstrainedProblems . . . . . . . . . . . . . . . . . . . . . . . . . . 165
6.4 InteriorPointMethods . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.5 MaximumSearchProblems . . . . . . . . . . . . . . . . . . . . . . . 179
6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
6.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
II SUPPORTVECTORMACHINES 187
7 PatternRecognition 189
7.1 SeparatingHyperplanes . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.2 TheRoleoftheMargin . . . . . . . . . . . . . . . . . . . . . . . . . . 192
7.3 OptimalMarginHyperplanes . . . . . . . . . . . . . . . . . . . . . . 196
7.4 NonlinearSupportVectorClassifiers . . . . . . . . . . . . . . . . . . 200
7.5 SoftMarginHyperplanes . . . . . . . . . . . . . . . . . . . . . . . . 204
7.6 Multi-ClassClassification . . . . . . . . . . . . . . . . . . . . . . . . 211
7.7 VariationsonaTheme . . . . . . . . . . . . . . . . . . . . . . . . . . 214
7.8 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
7.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.10 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30
8 Single-ClassProblems:QuantileEstimationandNoveltyDetection 227
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
8.2 ADistribution’sSupportandQuantiles . . . . . . . . . . . . . . . . 229
8.3 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
8.4 Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
8.5 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
8.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
8.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
8.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
8.9 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
9 RegressionEstimation 251
9.1 LinearRegressionwithInsensitiveLossFunction. . . . . . . . . . . 251
9.2 DualProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
9.3 -SVRegression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
(cid:1)
9.4 ConvexCombinationsand -Norms . . . . . . . . . . . . . . . . . . 266
1
(cid:2)
9.5 ParametricInsensitivityModels . . . . . . . . . . . . . . . . . . . . . 269
9.6 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
9.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
9.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
10 Implementation 279
10.1 TricksoftheTrade. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
10.2 SparseGreedyMatrixApproximation . . . . . . . . . . . . . . . . . 288
10.3 InteriorPointAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . 295
10.4 SubsetSelectionMethods . . . . . . . . . . . . . . . . . . . . . . . . 300
10.5 SequentialMinimalOptimization . . . . . . . . . . . . . . . . . . . . 305
10.6 IterativeMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
10.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
10.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
11 IncorporatingInvariances 333
11.1 PriorKnowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
11.2 TransformationInvariance . . . . . . . . . . . . . . . . . . . . . . . . 335
11.3 TheVirtualSVMethod . . . . . . . . . . . . . . . . . . . . . . . . . . 337
11.4 ConstructingInvarianceKernels . . . . . . . . . . . . . . . . . . . . 343
11.5 TheJitteredSVMethod . . . . . . . . . . . . . . . . . . . . . . . . . . 354
11.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
11.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
12 LearningTheoryRevisited 359
12.1 ConcentrationofMeasureInequalities . . . . . . . . . . . . . . . . . 360
12.2 Leave-One-OutEstimates . . . . . . . . . . . . . . . . . . . . . . . . 366
12.3 PAC-BayesianBounds . . . . . . . . . . . . . . . . . . . . . . . . . . 381
12.4 Operator-TheoreticMethodsinLearningTheory . . . . . . . . . . . 391
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30
12.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
12.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
III KERNELMETHODS 405
13 DesigningKernels 407
13.1 TricksforConstructingKernels . . . . . . . . . . . . . . . . . . . . . 408
13.2 StringKernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
13.3 Locality-ImprovedKernels. . . . . . . . . . . . . . . . . . . . . . . . 414
13.4 NaturalKernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
13.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
13.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
14 KernelFeatureExtraction 427
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
14.2 KernelPCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
14.3 KernelPCAExperiments . . . . . . . . . . . . . . . . . . . . . . . . . 437
14.4 AFrameworkforFeatureExtraction . . . . . . . . . . . . . . . . . . 442
14.5 AlgorithmsforSparseKFA . . . . . . . . . . . . . . . . . . . . . . . 447
14.6 KFAExperiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450
14.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
14.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
15 KernelFisherDiscriminant 457
15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
15.2 Fisher’sDiscriminantinFeatureSpace . . . . . . . . . . . . . . . . . 458
15.3 EfficientTrainingofKernelFisherDiscriminants . . . . . . . . . . . 460
15.4 ProbabilisticOutputs . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
15.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466
15.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
15.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
16 BayesianKernelMethods 469
16.1 Bayesics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
16.2 InferenceMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
16.3 GaussianProcesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
16.4 ImplementationofGaussianProcesses . . . . . . . . . . . . . . . . . 488
16.5 LaplacianProcesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
16.6 RelevanceVectorMachines . . . . . . . . . . . . . . . . . . . . . . . 506
16.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
16.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
17 RegularizedPrincipalManifolds 517
17.1 ACodingFramework . . . . . . . . . . . . . . . . . . . . . . . . . . 518
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30
17.2 ARegularizedQuantizationFunctional . . . . . . . . . . . . . . . . 522
17.3 AnAlgorithmforMinimizing R [f] . . . . . . . . . . . . . . . . . 526
reg
17.4 ConnectionstoOtherAlgorithms . . . . . . . . . . . . . . . . . . . . 529
17.5 UniformConvergenceBounds . . . . . . . . . . . . . . . . . . . . . 533
17.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
17.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
17.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540
18 Pre-ImagesandReducedSetMethods 543
18.1 ThePre-ImageProblem . . . . . . . . . . . . . . . . . . . . . . . . . 544
18.2 FindingApproximatePre-Images . . . . . . . . . . . . . . . . . . . . 547
18.3 ReducedSetMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
18.4 ReducedSetSelectionMethods . . . . . . . . . . . . . . . . . . . . . 554
18.5 ReducedSetConstructionMethods. . . . . . . . . . . . . . . . . . . 561
18.6 SequentialEvaluationofReducedSetExpansions . . . . . . . . . . 564
18.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
18.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
A Addenda 569
A.1 DataSets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569
A.2 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
B MathematicalPrerequisites 575
B.1 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
B.2 LinearAlgebra. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
B.3 FunctionalAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
References 591
Index 617
NotationandSymbols 625
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30
Series Foreword
Thegoalofbuildingsystemsthatcanadapttotheirenvironmentsandlearnfrom
their experience has attracted researchers from many fields, including computer
science, engineering, mathematics, physics, neuroscience, and cognitive science.
Out of this research has come a wide variety of learning techniques that have
the potential to transform many scientific and industrial fields. Recently, several
research communities have converged on a common set of issues surrounding
supervised, unsupervised, and reinforcement learning problems. The MIT Press
seriesonAdaptiveComputationandMachineLearningseekstounifythemany
diverse strands of machine learning research and to foster high quality research
andinnovativeapplications.
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and
Beyond is an excellent illustration of this convergence of ideas from many fields.
The development of kernel-based learning methods has resulted from a combi-
nation of machine learning theory, optimization algorithms from operations re-
search,andkerneltechniquesfrommathematicalanalysis.Thesethreeideashave
spread far beyond the original support-vector machine algorithm: Virtually ev-
erylearningalgorithmhasbeenredesignedtoexploitthepowerofkernelmeth-
ods.BernhardScho¨lkopfandAlexanderSmolahavewrittenacomprehensive,yet
accessible,accountofthesedevelopments.This volumeincludesall ofthemath-
ematical and algorithmic background needed not only to obtain a basic under-
standingofthematerialbuttomasterit.Studentsandresearcherswhostudythis
bookwillbeabletoapplykernelmethodsincreativewaystosolveawiderange
ofproblemsinscienceandengineering.
ThomasDietterich
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30
Preface
Oneofthemostfortunatesituationsascientistcanencounteristoenterafieldin
its infancy. There is a large choice of topics to work on, and many of the issues
are conceptual rather than merely technical. Over the last seven years, we have
hadtheprivilegetobeinthis positionwith regardtothefield ofSupportVector
Machines (SVMs). We began working on our respectivedoctoral dissertations in
1994 and 1996. Upon completion, we decided to combine our efforts and write
abookaboutSVMs. Sincethen,thefield hasdevelopedimpressively,and hasto
anextentbeentransformed.Wesetupawebsitethatquickly becamethecentral
repository for the new community, and a number of workshops were organized
byvariousresearchers.Thescopeofthefieldhasnowwidenedsignificantly,both
in terms of new algorithms, such as kernel methods different to SVMs, and in
terms of a deeper theoretical understanding being gained. It has become clear
that kernel methods provide a framework for tackling some rather profound
issuesinmachinelearningtheory.Atthesametime,successfulapplicationshave
demonstrated that SVMs not only have a more solid foundation than artificial
neuralnetworks, but are able to serveas a replacementfor neuralnetworks that
performaswellorbetter,inawidevarietyoffields.Standardneuralnetworkand
patternrecognitiontextbookshavenowstartedincludingchaptersonSVMsand
kernelPCA(forinstance,[235,153]).
While these developments took place, we were trying to strike a balance be-
tweenpursuingexcitingnewresearch,andmakingprogresswiththeslowlygrow-
ing manuscript of this book. In the two and a half years that we worked on the
book,wefacedanumberoflessonsthatwesuspecteveryonewritingascientific
monograph—oranyotherbook—willencounter.First,writingabookismore
workthanyouthink,evenwithtwoauthorssharingtheworkinequalparts.Sec-
ond, our book got longer than planned. Once we exceeded the initially planned
length of 500 pages, we got worried. In fact, the manuscript kept growing even
afterwestoppedwritingnewchapters,andbeganpolishingthingsandincorpo-
ratingcorrectionssuggestedbycolleagues.Thiswasmainlyduetothefactthatthe
bookdealswithafascinatingnewarea,andresearcherskeepaddingfreshmaterial
tothebodyofknowledge.Welearnedthatthereisnoasymptoticregimeinwrit-
ingsuchabook—ifonedoesnotstop,itwillgrowbeyondanybound—unless
onestartscutting.Wethereforehadtotakepainfuldecisionstoleaveoutmaterial
thatweoriginallythoughtshouldbeinthebook.Sadly,andthisisthethirdpoint,
thebookthuscontainslessmaterialthanoriginallyplanned,especiallyonthesub-
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30
jectoftheoreticaldevelopments.Wesincerelyapologizetoallresearcherswhofeel
thattheircontributionsshouldhavebeenincluded—thebookiscertainlybiased
towardsourownwork,anddoesnotprovideafullycomprehensiveoverviewof
thefield.Wedid,however,aimtoprovideallthenecessaryconceptsandideasto
enable a reader equipped with some basic mathematical knowledge to enter the
engaging world of machine learning, using theoretically well-founded kernel al-
gorithms, and to understand and apply the powerful algorithms that have been
developedoverthelastfewyears.
Thebookis divided intothreelogical parts.Eachpartconsists ofabriefintro-
duction and a numberof technical chapters.In addition, we include two appen-
dicescontainingaddenda,technicaldetails,andmathematicalprerequisites.Each
chapterbeginswithashortdiscussionoutliningthecontentsandprerequisites;for
someofthelongerchapters,weincludeagraphthatsketchesthelogicalstructure
anddependenciesbetweenthesections.Attheendofmostchapters,weincludea
setofproblems,rangingfromsimpleexercises(markedby )tohardones( );in
addition,wedescribeopenproblemsandquestionsforfutureresearch( ).1The
(cid:1) (cid:1)(cid:1)(cid:1)
latteroftenrepresentworthwhileprojectsforaresearchpublication,orevenathe-
ÆÆÆ
sis.Referencesarealsoincludedinsomeoftheproblems.Thesereferencescontain
thesolutionstotheassociatedproblems,oratleastsignificantpartsthereof.
The overall structure of the book is perhaps somewhat unusual. Rather than
presenting a logical progression of chapters building upon each other, we occa-
sionallytouchonasubjectbriefly,onlytorevisititlaterinmoredetail.Forreaders
whoareusedtoreadingscientificmonographsandtextbooksfromcovertocover,
thiswillamounttosomeredundancy.Wehope,however,thatsomereaders,who
are more selective in their reading habits (or less generous with their time), and
only look at those chapters that they are interested in, will benefit. Indeed, no-
body is expected to read every chapter. Some chapters are fairly technical, and
cover material included for reasons of completeness. Other chapters, which are
morerelevanttothecentralsubjectsofthebook,arekeptsimpler,andshouldbe
accessibletoundergraduatestudents.
In a way, this book thus contains several books in one. For instance, the first
chaptercanbereadasastandalone“executivesummary”ofSupportVectorand
kernelmethods.Thischaptershouldalsoprovideafastentrypointforpractition-
ers.SomeoneinterestedinapplyingSVMstoapatternrecognitionproblemmight
wanttoreadChapters1and7only.AreaderthinkingofbuildingtheirownSVM
implementationcouldadditionallyreadChapter10,andpartsofChapter6.Those
whowouldliketogetactivelyinvolvedinresearchaspectsofkernelmethods,for
exampleby“kernelizing”anewalgorithm,shouldprobablyreadatleastChapters
1and2.Aone-semesterundergraduatecourseonlearningwithkernelscouldin-
cludethematerialofChapters1,2.1–2.3,3.1–3.2,5.1–5.2,6.1–6.3,7.Ifthereismore
1. We suggest that authors post their solutions on the book website www.learning-with-
kernels.org.
Scho¨lkopfandSmola:LearningwithKernels 2001/09/2410:30