Table Of Content

Learning with Kernels AdaptiveComputationandMachineLearning ThomasDietterich,Editor ChristopherBishop,DavidHeckerman,MichaelJordan,andMichaelKearns,As- sociateEditors Bioinformatics:TheMachineLearningApproach,PierreBaldiandSørenBrunak ReinforcementLearning:AnIntroduction,RichardS.SuttonandAndrewG.Barto GraphicalModelsforMachineLearningandDigitalCommunication,BrendanJ.Frey LearninginGraphicalModels,MichaelI.Jordan Causation,Prediction,andSearch,secondedition,PeterSpirtes,ClarkGlymour,and RichardScheines PrinciplesofDataMining,DavidHand,HeikkiMannila,andPadhraicSmyth Bioinformatics: The Machine Learning Approach, second edition, Pierre Baldi and SørenBrunak LearningKernelClassifiers:TheoryandAlgorithms,RalfHerbrich Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond,BernhardScho¨lkopfandAlexanderJ.Smola Learning with Kernels SupportVectorMachines,Regularization,Optimization,andBeyond BernhardScho¨lkopf AlexanderJ.Smola TheMITPress Cambridge,Massachusetts London,England c2002MassachusettsInstituteofTechnology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanicalmeans(includingphotocopying,recording,orinformationstorageandretrieval)without permissioninwritingfromthepublisher. TypesetbytheauthorsusingLATEX2 PrintedandboundintheUnitedStatesofAmerica LibraryofCongressCataloging-in-PublicationData LearningwithKernels—SupportVectorMachines, Regularization,OptimizationandBeyond/byBernhardScho¨lkopf, AlexanderJ.Smola. p. cm. Includesbibliographicalreferencesandindex. ISBN0-262-19475-9(alk.paper) 1.Machinelearning.2.Algorithms.3.Kernelfunctions I.Scho¨lkopf,Bernhard.II.Smola,AlexanderJ. Toourparents Contents SeriesForeword xiii Preface xv 1 ATutorialIntroduction 1 1.1 DataRepresentationandSimilarity . . . . . . . . . . . . . . . . . . . 1 1.2 ASimplePatternRecognitionAlgorithm . . . . . . . . . . . . . . . 4 1.3 SomeInsightsFromStatisticalLearningTheory . . . . . . . . . . . 6 1.4 HyperplaneClassifiers . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.5 SupportVectorClassification . . . . . . . . . . . . . . . . . . . . . . 15 1.6 SupportVectorRegression . . . . . . . . . . . . . . . . . . . . . . . . 17 1.7 KernelPrincipalComponentAnalysis . . . . . . . . . . . . . . . . . 19 1.8 EmpiricalResultsandImplementations . . . . . . . . . . . . . . . . 21 I CONCEPTSANDTOOLS 23 2 Kernels 25 2.1 ProductFeatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.2 TheRepresentationofSimilaritiesinLinearSpaces. . . . . . . . . . 29 2.3 ExamplesandPropertiesofKernels . . . . . . . . . . . . . . . . . . 45 2.4 TheRepresentationofDissimilaritiesinLinearSpaces . . . . . . . . 48 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3 RiskandLossFunctions 61 3.1 LossFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.2 TestErrorandExpectedRisk . . . . . . . . . . . . . . . . . . . . . . 65 3.3 AStatisticalPerspective . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.4 RobustEstimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 3.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4 Regularization 87 4.1 TheRegularizedRiskFunctional . . . . . . . . . . . . . . . . . . . . 88 viii Contents 4.2 TheRepresenterTheorem . . . . . . . . . . . . . . . . . . . . . . . . 89 4.3 RegularizationOperators . . . . . . . . . . . . . . . . . . . . . . . . 92 4.4 TranslationInvariantKernels . . . . . . . . . . . . . . . . . . . . . . 96 4.5 TranslationInvariantKernelsinHigherDimensions . . . . . . . . . 105 4.6 DotProductKernels . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 4.7 Multi-OutputRegularization . . . . . . . . . . . . . . . . . . . . . . 113 4.8 SemiparametricRegularization . . . . . . . . . . . . . . . . . . . . . 115 4.9 CoefficientBasedRegularization . . . . . . . . . . . . . . . . . . . . 118 4.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 4.11 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5 ElementsofStatisticalLearningTheory 125 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.2 TheLawofLargeNumbers . . . . . . . . . . . . . . . . . . . . . . . 128 5.3 WhenDoesLearningWork:theQuestionofConsistency . . . . . . 131 5.4 UniformConvergenceandConsistency . . . . . . . . . . . . . . . . 131 5.5 HowtoDeriveaVCBound . . . . . . . . . . . . . . . . . . . . . . . 134 5.6 AModelSelectionExample . . . . . . . . . . . . . . . . . . . . . . . 144 5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 5.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 6 Optimization 149 6.1 ConvexOptimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 6.2 UnconstrainedProblems . . . . . . . . . . . . . . . . . . . . . . . . . 154 6.3 ConstrainedProblems . . . . . . . . . . . . . . . . . . . . . . . . . . 165 6.4 InteriorPointMethods . . . . . . . . . . . . . . . . . . . . . . . . . . 175 6.5 MaximumSearchProblems . . . . . . . . . . . . . . . . . . . . . . . 179 6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 6.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 II SUPPORTVECTORMACHINES 187 7 PatternRecognition 189 7.1 SeparatingHyperplanes . . . . . . . . . . . . . . . . . . . . . . . . . 189 7.2 TheRoleoftheMargin . . . . . . . . . . . . . . . . . . . . . . . . . . 192 7.3 OptimalMarginHyperplanes . . . . . . . . . . . . . . . . . . . . . . 196 7.4 NonlinearSupportVectorClassifiers . . . . . . . . . . . . . . . . . . 200 7.5 SoftMarginHyperplanes . . . . . . . . . . . . . . . . . . . . . . . . 204 7.6 Multi-ClassClassification . . . . . . . . . . . . . . . . . . . . . . . . 211 7.7 VariationsonaTheme . . . . . . . . . . . . . . . . . . . . . . . . . . 214 7.8 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 7.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 7.10 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 Contents ix 8 Single-ClassProblems:QuantileEstimationandNoveltyDetection 227 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 8.2 ADistribution’sSupportandQuantiles . . . . . . . . . . . . . . . . 229 8.3 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 8.4 Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 8.5 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 8.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 8.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 8.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 8.9 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 9 RegressionEstimation 251 9.1 LinearRegressionwithInsensitiveLossFunction. . . . . . . . . . . 251 9.2 DualProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 9.3 -SVRegression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 9.4 ConvexCombinationsand -Norms . . . . . . . . . . . . . . . . . . 266 1 9.5 ParametricInsensitivityModels . . . . . . . . . . . . . . . . . . . . . 269 9.6 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 9.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 9.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 10 Implementation 279 10.1 TricksoftheTrade. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 10.2 SparseGreedyMatrixApproximation . . . . . . . . . . . . . . . . . 288 10.3 InteriorPointAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . 295 10.4 SubsetSelectionMethods . . . . . . . . . . . . . . . . . . . . . . . . 300 10.5 SequentialMinimalOptimization . . . . . . . . . . . . . . . . . . . . 305 10.6 IterativeMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 10.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 10.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 11 IncorporatingInvariances 333 11.1 PriorKnowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 11.2 TransformationInvariance . . . . . . . . . . . . . . . . . . . . . . . . 335 11.3 TheVirtualSVMethod . . . . . . . . . . . . . . . . . . . . . . . . . . 337 11.4 ConstructingInvarianceKernels . . . . . . . . . . . . . . . . . . . . 343 11.5 TheJitteredSVMethod . . . . . . . . . . . . . . . . . . . . . . . . . . 354 11.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 11.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 12 LearningTheoryRevisited 359 12.1 ConcentrationofMeasureInequalities . . . . . . . . . . . . . . . . . 360 12.2 Leave-One-OutEstimates . . . . . . . . . . . . . . . . . . . . . . . . 366 12.3 PAC-BayesianBounds . . . . . . . . . . . . . . . . . . . . . . . . . . 381 12.4 Operator-TheoreticMethodsinLearningTheory . . . . . . . . . . . 391 x Contents 12.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 12.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 III KERNELMETHODS 405 13 DesigningKernels 407 13.1 TricksforConstructingKernels . . . . . . . . . . . . . . . . . . . . . 408 13.2 StringKernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 13.3 Locality-ImprovedKernels. . . . . . . . . . . . . . . . . . . . . . . . 414 13.4 NaturalKernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 13.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 13.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 14 KernelFeatureExtraction 427 14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 14.2 KernelPCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 14.3 KernelPCAExperiments . . . . . . . . . . . . . . . . . . . . . . . . . 437 14.4 AFrameworkforFeatureExtraction . . . . . . . . . . . . . . . . . . 442 14.5 AlgorithmsforSparseKFA . . . . . . . . . . . . . . . . . . . . . . . 447 14.6 KFAExperiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 14.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 14.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452 15 KernelFisherDiscriminant 457 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457 15.2 Fisher’sDiscriminantinFeatureSpace . . . . . . . . . . . . . . . . . 458 15.3 EfficientTrainingofKernelFisherDiscriminants . . . . . . . . . . . 460 15.4 ProbabilisticOutputs . . . . . . . . . . . . . . . . . . . . . . . . . . . 464 15.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 15.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 15.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468 16 BayesianKernelMethods 469 16.1 Bayesics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470 16.2 InferenceMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475 16.3 GaussianProcesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480 16.4 ImplementationofGaussianProcesses . . . . . . . . . . . . . . . . . 488 16.5 LaplacianProcesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499 16.6 RelevanceVectorMachines . . . . . . . . . . . . . . . . . . . . . . . 506 16.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511 16.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513 17 RegularizedPrincipalManifolds 517 17.1 ACodingFramework . . . . . . . . . . . . . . . . . . . . . . . . . . 518 Contents xi 17.2 ARegularizedQuantizationFunctional . . . . . . . . . . . . . . . . 522 17.3 AnAlgorithmforMinimizing R [f] . . . . . . . . . . . . . . . . . 526 reg 17.4 ConnectionstoOtherAlgorithms . . . . . . . . . . . . . . . . . . . . 529 17.5 UniformConvergenceBounds . . . . . . . . . . . . . . . . . . . . . 533 17.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 17.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539 17.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540 18 Pre-ImagesandReducedSetMethods 543 18.1 ThePre-ImageProblem . . . . . . . . . . . . . . . . . . . . . . . . . 544 18.2 FindingApproximatePre-Images . . . . . . . . . . . . . . . . . . . . 547 18.3 ReducedSetMethods. . . . . . . . . . . . . . . . . . . . . . . . . . . 552 18.4 ReducedSetSelectionMethods . . . . . . . . . . . . . . . . . . . . . 554 18.5 ReducedSetConstructionMethods. . . . . . . . . . . . . . . . . . . 561 18.6 SequentialEvaluationofReducedSetExpansions . . . . . . . . . . 564 18.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566 18.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 A Addenda 569 A.1 DataSets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569 A.2 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572 B MathematicalPrerequisites 575 B.1 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575 B.2 LinearAlgebra. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580 B.3 FunctionalAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586 References 591 Index 617 NotationandSymbols 625

Learning with kernels : support vector machines, regularization, optimization, and beyond PDF

640 Pages·2002·10.325 MB·English

by Bernhard Schölkopf, Alexander J Smola

Checking for file health...

Save to my drive

Quick download

Download

Download Learning with kernels : support vector machines, regularization, optimization, and beyond PDF Free - Full Version

by Bernhard Schölkopf; Alexander J Smola| 2002| 640 pages| 10.325| English

Download Learning with kernels : support vector machines, regularization, optimization, and beyond by Bernhard Schölkopf; Alexander J Smola in PDF format completely FREE. No registration required, no payment needed. Get instant access to this valuable resource on PDFdrive.to!

Free Download PDF

About Learning with kernels : support vector machines, regularization, optimization, and beyond

No description available for this book.

Detailed Information

Author:	Bernhard Schölkopf; Alexander J Smola
Publication Year:	2002
ISBN:	9780585477596
Pages:	640
Language:	English
File Size:	10.325
Format:	PDF
Price:	FREE

Download Free PDF

Safe & Secure Download - No registration required

Why Choose PDFdrive for Your Free Learning with kernels : support vector machines, regularization, optimization, and beyond Download?

100% Free: No hidden fees or subscriptions required for one book every day.
No Registration: Immediate access is available without creating accounts for one book every day.
Safe and Secure: Clean downloads without malware or viruses
Multiple Formats: PDF, MOBI, Mpub,... optimized for all devices
Educational Resource: Supporting knowledge sharing and learning

Frequently Asked Questions

Is it really free to download Learning with kernels : support vector machines, regularization, optimization, and beyond PDF?

Yes, on https://PDFdrive.to you can download Learning with kernels : support vector machines, regularization, optimization, and beyond by Bernhard Schölkopf; Alexander J Smola completely free. We don't require any payment, subscription, or registration to access this PDF file. For 3 books every day.

How can I read Learning with kernels : support vector machines, regularization, optimization, and beyond on my mobile device?

After downloading Learning with kernels : support vector machines, regularization, optimization, and beyond PDF, you can open it with any PDF reader app on your phone or tablet. We recommend using Adobe Acrobat Reader, Apple Books, or Google Play Books for the best reading experience.

Is this the full version of Learning with kernels : support vector machines, regularization, optimization, and beyond?

Yes, this is the complete PDF version of Learning with kernels : support vector machines, regularization, optimization, and beyond by Bernhard Schölkopf; Alexander J Smola. You will be able to read the entire content as in the printed version without missing any pages.

Is it legal to download Learning with kernels : support vector machines, regularization, optimization, and beyond PDF for free?

https://PDFdrive.to provides links to free educational resources available online. We do not store any files on our servers. Please be aware of copyright laws in your country before downloading.

The materials shared are intended for research, educational, and personal use in accordance with fair use principles.