Mathematical Pictures at a Data Science Exhibition

339 Pages·2022·3.466 MB·English

by  Simon Foucart

MathematicalPicturesataDataScienceExhibition
Inthepastfewdecades,heuristicmethodsadoptedbybigtechcompanieshave
complementedexistingscientificdisciplinestoformthenewfieldofDataScience.
Thistextprovidesdeepandcomprehensivecoverageofthemathematicaltheory
supportingthefield.Composedof27lecture-lengthchapterswithexercises,it
embarksthereadersonanengagingitinerarythroughkeysubjectsindatascience,
includingmachinelearning,optimalrecovery,compressivesensing(alsoknownas
compressedsensing),optimization,andneuralnetworks.Whilestandardmaterialis
covered,thebookalsoincludesdistinctivepresentationsoftopicssuchasreproducing
kernelHilbertspaces,spectralclustering,optimalrecovery,compressivesensing,
grouptesting,andapplicationsofsemidefiniteprogramming.Studentsanddata
scientistswithlessmathematicalbackgroundwillappreciatetheappendicesthat
supplymoredetailsonsomeoftheabstractconcepts.
simon foucartisProfessorofMathematicsatTexasA&MUniversity,where
hewasnamedPresidentialImpactFellowin2019.Hehaspreviouslywritten,together
withHolgerRauhut,theinfluentialbookAMathematicalIntroductiontoCompressive
Sensing(2013).

Mathematical Pictures
at a
Data Science Exhibition
SIMON FOUCART
TexasA&MUniversity
UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom
OneLibertyPlaza,20thFloor,NewYork,NY10006,USA
477WilliamstownRoad,PortMelbourne,VIC3207,Australia
314–321,3rdFloor,Plot3,SplendorForum,JasolaDistrictCentre,
NewDelhi–110025,India
103PenangRoad,#05–06/07,VisioncrestCommercial,Singapore238467
CambridgeUniversityPressispartoftheUniversityofCambridge.
ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof
education,learning,andresearchatthehighestinternationallevelsofexcellence.
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781316518885
DOI:10.1017/9781009003933
©SimonFoucart2022
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2022
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary.
ISBN978-1-316-51888-5Hardback
ISBN978-1-009-00185-4Paperback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyof
URLsforexternalorthird-partyinternetwebsitesreferredtointhispublication
anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain,
accurateorappropriate.
PourJeanne,a` nouveau,
andnowalsoforE´mileandLe´opold

Contents
Preface pagexiii
Notation xvii
PARTONE MACHINELEARNING 1
ExecutiveSummary 3
1 RudimentsofStatisticalLearningTheory 4
1.1 TrueandEmpiricalRisks 4
1.2 PAC-Learnability 6
1.3 Validation 8
Exercises 9
2 Vapnik–ChervonenkisDimension 10
2.1 Definitions 10
2.2 Examples 11
2.3 SauerLemma 13
Exercises 15
3 LearnabilityforBinaryClassification 16
3.1 UniformConvergenceProperty 16
3.2 FiniteVC-DimensionImpliesPAC-Learnability 17
3.3 No-Free-LunchTheorem 20
Exercises 22
4 SupportVectorMachines 23
4.1 LinearSeparability 23
4.2 HardandSoftSVM 25
4.3 KernelTrick 27
Exercises 29
vii
viii Contents
5 ReproducingKernelHilbertSpaces 31
5.1 AbstractDefinition 31
5.2 Moore–AronszajnTheorem 33
5.3 MercerTheorem 37
Exercises 39
6 RegressionandRegularization 41
6.1 EmpiricalRiskMinimization 41
6.2 Regularization 43
6.3 ClassificationviaRegression 44
Exercises 45
7 Clustering 47
7.1 Single-LinkageClustering 48
7.2 Center-BasedClustering 49
7.3 SpectralClustering 51
Exercises 54
8 DimensionReduction 56
8.1 PrincipalComponentAnalysis 56
8.2 Johnson–LindenstraussLemma 58
8.3 LocallyLinearEmbedding 61
Exercises 64
PARTTWO OPTIMALRECOVERY 65
ExecutiveSummary 67
9 FoundationalResultsofOptimalRecovery 68
9.1 Models,Errors,andOptimality 69
9.2 LinearityofOptimalRecoveryMaps 71
9.3 AnExtremalPropertyofSplines 73
Exercises 75
10 ApproximabilityModels 76
10.1 TheModelSet 76
10.2 OptimalityinaHilbertSetting 78
10.3 OptimalityforLinearFunctionals 82
Exercises 85
11 IdealSelectionofObservationSchemes 86
11.1 HilbertSetting 86
11.2 IntegrationofLipschitzFunctions 88
11.3 AdaptivityDoesNotHelpMuch 91
Exercises 92
Contents ix
12 CurseofDimensionality 94
12.1 NotionsofTractability 94
12.2 IntegrationofTrigonometricPolynomials 96
12.3 IntegrationinWeightedSobolevSpaces 98
Exercises 100
13 Quasi-MonteCarloIntegration 102
13.1 VariationandDiscrepancy 103
13.2 Koksma–HlawkaInequality 105
13.3 Low-DiscrepancySets 107
Exercises 111
PARTTHREE COMPRESSIVESENSING 113
ExecutiveSummary 115
14 SparseRecoveryfromLinearObservations 116
14.1 (cid:2) -Minimization 117
0
14.2 (cid:2) -Minimization 118
1
14.3 (cid:2) -RestrictedIsometryProperty 119
1
Exercises 122
15 TheComplexityofSparseRecovery 123
15.1 LimitationsImposedbyStabilityandRobustness 123
15.2 GelfandWidthofthe(cid:2) -Ball 127
1
15.3 Irrelevanceof(cid:2) -Stability 130
2
Exercises 130
16 Low-RankRecoveryfromLinearObservations 132
16.1 NuclearNormMinimization 132
16.2 (cid:2) -RankRestrictedIsometryProperty 134
1
16.3 SemidefiniteProgrammingFormulation 136
Exercises 137
17 SparseRecoveryfromOne-BitObservations 139
17.1 EstimatingtheDirectionviaHardThresholding 139
17.2 EstimatingtheDirectionviaLinearProgramming 141
17.3 EstimatingBoththeDirectionandtheMagnitude 145
Exercises 147
18 GroupTesting 149
18.1 PropertiesoftheTestMatrix 149
18.2 SatisfyingtheSeparabityCondition 151
18.3 RecoveryviaaLinearFeasibilityProgram 154
Exercises 155