Table Of ContentCONCENTRATION INEQUALITIES
This page intentionally left blank
Concentration Inequalities
ANonasymptoticTheoryofIndependence
STÉPHANE BOUCHERON
GÁBOR LUGOSI
PASCAL MASSART
3
3
GreatClarendonStreet,Oxford,OX26DP,
UnitedKingdom
OxfordUniversityPressisadepartmentoftheUniversityofOxford.
ItfurtherstheUniversity’sobjectiveofexcellenceinresearch,scholarship,
andeducationbypublishingworldwide.Oxfordisaregisteredtrademarkof
OxfordUniversityPressintheUKandincertainothercountries
©StéphaneBoucheron,GáborLugosi,PascalMassart2013
Themoralrightsoftheauthorshavebeenasserted
FirstEditionpublishedin2013
Impression: 1
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedin
aretrievalsystem,ortransmitted,inanyformorbyanymeans,withoutthe
priorpermissioninwritingofOxfordUniversityPress,orasexpresslypermitted
bylaw,bylicenceorundertermsagreedwiththeappropriatereprographics
rightsorganization.Enquiriesconcerningreproductionoutsidethescopeofthe
aboveshouldbesenttotheRightsDepartment,OxfordUniversityPress,atthe
addressabove
Youmustnotcirculatethisworkinanyotherform
andyoumustimposethissameconditiononanyacquirer
BritishLibraryCataloguinginPublicationData
Dataavailable
ISBN978–0–19–953525–5
Printedandboundby
CPIGroup(UK)Ltd,Croydon,CR04YY
FOREWORD
Measure concentration ideas developed during the last century in various parts of
mathematics including functional analysis, probability theory, and statistical mechanics,
areastypicallydealingwithmodelsinvolvinganinfinitenumberofvariables.Afterearly
observations,andinparticularageometricinterpretationofthelawoflargenumbersby
E.Borel,therealbirthofmeasureconcentrationtookplaceintheearly1970swiththenew
proofbyV.Milman,relyingonLévy’sinequality(ofisoperimetricnature),ofDvoretzky’s
theoremonsphericalsectionsofconvexbodiesinhighdimension.Theinherentconceptof
measureconcentrationemphasizedbyV.Milmanthroughthisproofturnedouttobeone
ofthemainachievementsofanalysisofthesecondpartofthelastcentury.Itopenedapos-
terioricompletelynewperspectivesanddevelopmentswithapplicationstovariousfieldsof
mathematics.Inparticular,promptedbytheconceptandresults,inthe1980sand1990s
M. Talagrand undertook a deep investigation of concentration inequalities for product
measures, emphasizing a revolutionary new look at independence. Viewing namely ran-
domvariablesdepending(inasmoothway)ontheinfluenceofmanyindependentrandom
variables(butnottoomuchonanyofthem)asessentiallyconstantledhimtogroundbreak-
ingachievementsandstrikingapplications.Particularlywiththetoolofcelebratedconvex
distanceinequality,M.Talagranddevelopedapplicationstocombinatorialprobability,stat-
isticalmechanics,andempiricalprocesses.Simultaneously,theentropicmethod,relying
onanearlyobservationbyI.HerbstinthecontextoflogarithmicSobolevinequalitiesand
developinginformationtheoreticideas,becameapowerfuladditionalandflexiblemethod
intheinvestigationofnewconcentrationproperties.
Sincethen,theconcentration-of-measurephenomenonhasspreadouttoanimpressively
widerangeofillustrationsandapplications,andbecameacentraltoolandviewpointinthe
quantitativeanalysisofanumberofasymptoticpropertiesinnumeroustopicsofinterest
includinggeometricanalysis,probabilitytheory,statisticalmechanics,mathematicalstat-
isticsandlearningtheory,randommatrixtheoryorquantuminformationtheory,stochastic
dynamics,randomizedalgorithms,complexity,andsoon.
ThisbookbyS.Boucheron,G.Lugosi,andP.Massartisamostwelcomeandcomplete
accountofthemoderndevelopmentsofconcentrationinequalitiesinthecontextofthe
probabilisticmethod.Themonographcoversmostoftheimportantandrecentdevelop-
ments,withconstantattentiontoillustrationsandapplicationswhichmakethetheoryso
fruitfulandattractive.Theemphasisputoninformationtheoreticmethodsisonemainfea-
tureoftheexpositionandthereisconsiderablebenefitinthisapproachforanumberof
fundamentalresultsandtools,forexampletheconvexdistanceinequalityorsharpbounds
onempiricalprocessesoffundamentalimportanceinstatisticalapplications.Themono-
graphcoversfurtherbasicandmostillustrativeexamplesofthecurrentresearch,including
dimensionreduction,randommatrices,Booleananalysis,transportationinequalities,and
vi | FOREWORD
isoperimetric-type bounds. The style adopted by the authors is a perfect balance from
thebasicandclassicalmaterialuptothemostsophisticatedandpowerfulresults,always
accessible and clearly reachable. Young and confirmed scientists, independently of their
background, will find with this book the ideal path toward the powerful ideas and tools
ofconcentrationinequalities,suggestedandillustratedwiththemostrelevantapplications
anddevelopments.
Itisanhonourandapleasuretowritethisprefacetothiswonderfulbook,whichissure
tobeahugesuccess.
MichelLedoux
UniversitédeToulouse
CONTENTS
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 SumsofIndependentRandomVariablesandtheMartingaleMethod 2
1.2 TheConcentration-of-MeasurePhenomenon 4
1.3 TheEntropyMethod 9
1.4 TheTransportationMethod 12
1.5 ReadingGuide 13
1.6 Acknowledgments 16
2 BasicInequalities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.1 FromMomentstoTails 19
2.2 TheCramér–Chernoff Method 21
2.3 Sub-GaussianRandomVariables 24
2.4 Sub-GammaRandomVariables 27
2.5 AMaximalInequality 31
2.6 Hoeffding’sInequality 34
2.7 Bennett’sInequality 35
2.8 Bernstein’sInequality 36
2.9 RandomProjectionsandtheJohnson–LindenstraussLemma 39
2.10 AssociationInequalities 43
2.11 Minkowski’sInequality 44
2.12 BibliographicalRemarks 45
2.13 Exercises 46
3 BoundingtheVariance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.1 TheEfron–SteinInequality 53
3.2 FunctionswithBoundedDifferences 56
3.3 Self-BoundingFunctions 60
3.4 MoreExamplesandApplications 63
3.5 AConvexPoincaréInequality 66
3.6 ExponentialTailBoundsviatheEfron–SteinInequality 68
3.7 TheGaussianPoincaréInequality 72
3.8 AProofoftheEfron–SteinInequalityBasedonDuality 73
3.9 BibliographicalRemarks 76
3.10 Exercises 78
4 BasicInformationInequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.1 ShannonEntropyandRelativeEntropy 84
4.2 EntropyonProductSpacesandtheChainRule 85
4.3 Han’sInequality 86
viii | CONTENTS
4.4 EdgeIsoperimetricInequalityontheBinaryHypercube 87
4.5 CombinatorialEntropies 89
4.6 Han’sInequalityforRelativeEntropies 91
4.7 Sub-AdditivityoftheEntropy 93
4.8 EntropyofGeneralRandomVariables 96
4.9 DualityandVariationalFormulas 97
4.10 ATransportationLemma 101
4.11 Pinsker’sInequality 102
4.12 Birgé’sInequality 103
4.13 Sub-AdditivityofEntropy:TheGeneralCase 105
4.14 TheBrunn–MinkowskiInequality 107
4.15 BibliographicalRemarks 110
4.16 Exercises 112
5 LogarithmicSobolevInequalities . . . . . . . . . . . . . . . . . . . . . . . . 117
5.1 SymmetricBernoulliDistributions 118
5.2 Herbst’sArgument:ConcentrationontheHypercube 121
5.3 AGaussianLogarithmicSobolevInequality 124
5.4 GaussianConcentration:TheTsirelson–Ibragimov–Sudakov
Inequality 125
5.5 AConcentrationInequalityforSupremaofGaussianProcesses 127
5.6 GaussianRandomProjections 128
5.7 APerformanceBoundfortheLasso 132
5.8 Hypercontractivity:TheBonami–BecknerInequality 139
5.9 GaussianHypercontractivity 146
5.10 TheLargestEigenvalueofRandomMatrices 147
5.11 BibliographicalRemarks 152
5.12 Exercises 154
6 TheEntropyMethod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
6.1 TheBoundedDifferencesInequality 170
6.2 MoreonBoundedDifferences 174
6.3 ModifiedLogarithmicSobolevInequalities 175
6.4 BeyondBoundedDifferences 176
6.5 InequalitiesfortheLowerTail 178
6.6 ConcentrationofConvexLipschitzFunctions 180
6.7 ExponentialInequalitiesforSelf-BoundingFunctions 181
6.8 SymmetrizedModifiedLogarithmicSobolevInequalities 184
6.9 ExponentialEfron–SteinInequalities 185
6.10 AModifiedLogarithmicSobolevInequalityforthePoissonDistribution 188
6.11 WeaklySelf-BoundingFunctions 189
6.12 ProofofLemma6.22 196
6.13 SomeVariations 199
6.14 Janson’sInequality 204
6.15 BibliographicalRemarks 207
6.16 Exercises 209
CONTENTS | ix
7 ConcentrationandIsoperimetry. . . . . . . . . . . . . . . . . . . . . . . . . 215
7.1 Lévy’sInequalities 215
7.2 TheClassicalIsoperimetricTheorem 218
7.3 VertexIsoperimetricInequalityintheHypercube 222
7.4 ConvexDistanceInequality 224
7.5 ConvexLipschitzFunctionsRevisited 229
7.6 BinPacking 230
7.7 BibliographicalRemarks 232
7.8 Exercises 233
8 TheTransportationMethod . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
8.1 TheBoundedDifferencesInequalityRevisited 239
8.2 BoundedDifferencesinQuadraticMean 241
8.3 ApplicationsofMarton’sConditionalTransportationInequality 247
8.4 TheConvexDistanceInequalityRevisited 249
8.5 Talagrand’sGaussianTransportationInequality 251
8.6 Appendix:AGeneralInductionLemma 256
8.7 BibliographicalRemarks 259
8.8 Exercises 260
9 InfluencesandThresholdPhenomena. . . . . . . . . . . . . . . . . . . . . 262
9.1 Influences 263
9.2 SomeFundamentalInequalitiesforInfluences 264
9.3 LocalConcentration 271
9.4 DiscreteFourierAnalysisandaVarianceInequality 273
9.5 MonotoneSets 277
9.6 ThresholdPhenomena 279
9.7 BibliographicalRemarks 286
9.8 Exercises 287
10 IsoperimetryontheHypercubeandGaussianSpaces . . . . . . . . . . . 290
10.1 Bobkov’sInequalityforFunctionsontheHypercube 291
10.2 AnIsoperimetricInequalityontheBinaryHypercube 297
10.3 AsymmetricBernoulliDistributionsandThresholdPhenomena 298
10.4 TheGaussianIsoperimetricTheorem 303
10.5 LipschitzFunctionsofGaussianRandomVariables 307
10.6 BibliographicalRemarks 308
10.7 Exercises 309
11 TheVarianceofSupremaofEmpiricalProcesses . . . . . . . . . . . . . . 312
11.1 GeneralUpperBoundsfortheVariance 315
11.2 Nemirovski’sInequality 317
11.3 TheSymmetrizationandContractionPrinciples 322
11.4 WeakandWimpyVariances 327
11.5 UnboundedSummands 330
11.6 BibliographicalRemarks 335
11.7 Exercises 336
