Table Of ContentReproducing Kernel Hilbert Spaces
In
Probability and Statistics
REPRODUCING KERNEL HILBERT SPACES
IN
PROBABILITY AND STATISTICS
ALAIN BERLINET
Department of Mathematics,
UMR CNRS 5030,
University of Montpellier II,
place Bataillon
34095 Montpellier cedex 05, FRANCE
CHRISTINE THOMAS-AGNAN
GREMAQ,
UMR CNRS 5604,
University of Toulouse 1,
allees de Brienne,
31 000 Toulouse, FRANCE
Springer Scence+Business Media, LLC
Reprodueing Kemel Hilbert Spaees in Probability and Statisties
Berlinet, A. and Thomas-Agnan, C.
p.em.
lnc\udes index.
ISBN 978-1-4613-4792-7 ISBN 978-1-4419-9096-9 (eBook)
DOI 10.1007/978-1-4419-9096-9
Copyright © 2004 by Springer Science+Business Media New York
Origina11y published by Kluwer Academic Publishers in 2001
Softcover reprint of the hardcover 1s t edition 2001
AII rights reserved. No part ofthis work may be reproduced, stored in a retrieval
systern, or transmitted in any forrn or by any means, electronic, mechanical,
photocopying, microfilming, recording, or otherwise, without the written permission
from the Publisher, with the exception of any material supplied specifieally for the
purpose ofbeing entered and executed on a computer systern, for exclusive use by
the purchaser of the work.
Printed on acid-free pap er.
To Marie-France,
Emilie, Cecilia and Amael.
To Pascal,
Azalals and Maiti.
To our parents,
Emelia and Francois,
Lucette and Raymond.
Contents
Preface xiii
Acknowledgments xvii
Introduction XIX
1. THEORY 1
1.1 Introduction 1
1.2 Notation and basic definitions 3
1.3 Reproducing kernels and positive type functions 13
1.4 Basic properties of reproducing kernels 24
1.4.1 Sum of reproducing kernels 24
1.4.2 Restriction of the index set 25
1.4.3 Support of a reproducing kernel 26
1.4.4 Kernel of an operator 27
1.4.5 Condition for }{K C }{R. 30
1.4.6 Tensor products of RKHS 30
1.5 Separability. Continuity 31
1.6 Extensions 37
1.6.1 Schwartz kernels 37
1.6.2 Semi-kernels 40
1.7 Positive type operators 42
1.7.1 Continuous functions of positive type 42
1.7.2 Schwartzdistributions ofpositive typeor conditionally
of positive type 44
1.8 Exercises 48
VIII RKHS INPROBABILITY AND STATISTICS
2. RKHS AND STOCHASTIC PROCESSES 55
2.1 Introduction 55
2.2 Covariance function of a second order stochastic process 55
2.2.1 Case of ordinary stochastic processes 55
2.2.1.1 Case of generalized stochastic processes 56
2.2.2 Positivity and covariance 57
2.2.2.1 Positive type functions and covariance functions 57
2.2.2.2 Generalized covariances and conditionally ofpositive
type functions 59
2.2.3 Hilbert space generated by a process 62
2.3 Representation theorems 64
2.3.1 The Loeve representation theorem 65
2.3.2 The Mercer representation theorem 68
2.3.3 The Karhunen representation theorem 70
2.3.4 Applications 72
2.4 Applications to stochastic filtering 75
2.4.1 Best Prediction 76
2.4.1.1 Best prediction and best linear prediction 76
2.4.1.2 Best linear unbiased prediction 79
2.4.2 Filtering and spline functions 80
2.4.2.1 No drift-no noise model and interpolating splines 82
2.4.2.2 Noise without drift model and smoothing splines 83
2.4.2.3 Complete model and partial smoothing splines 84
2.4.2.4 Case of gaussian processes 86
2.4.2.5 The Kriging models 88
2.4.2.6 Directions of generalization 94
2.5 Uniform Minimum Variance Unbiased Estimation 95
2.6 Density functional of a gaussian process and applications
to extraction and detection problems 97
2.6.1 Density functional of a gaussian process 97
2.6.2 Minimum variance unbiased estimation ofthe mean
value of a gaussian process with known covariance 100
2.6.3 Applications to extraction problems 102
2.6.4 Applications to detection problems 104
2.7 Exercises 105
3. NONPARAMETRIC CURVE ESTIMATION 109
3.1 Introduction 109
3.2 A brief introduction to splines 110
3.2.1 Abstract Interpolating splines 111
3.2.2 Abstract smoothing splines 116
3.2.3 Partial and mixed splines 118
3.2.4 Some concrete splines 121
3.2.4.1 tr: splines 121
Contents IX
3.2.4.2 Periodic Dm splines 122
3.2.4.3 L splines 123
3.2.4.4 a-splines, thin plate splines and Duchon's rotation
invariant splines 123
3.2.4.5 Other splines 124
3.3 Random interpolating splines 125
3.4 Spline regression estimation 125
3.4.1 Least squares spline estimators 126
3.4.2 Smoothing spline estimators 127
3.4.3 Hybrid splines 128
3.4.4 Bayesian models 129
3.5 Spline density estimation 132
3.6 Shape restrictions in curve estimation 134
3.7 Unbiased density estimation 135
3.8 Kernels and higher order kernels 136
3.9 Local approximation of functions 143
3.10 Local polynomial smoothing of statistical functionals 148
3.10.1 Density estimation in selection bias models. 150
3.10.2 Hazard functions 152
3.10.3 Reliability and econometric functions. 154
3.11 Kernels of order (m,p) 155
3.11.1 Definition of Ko-based hierarchies 158
3.11.2 Computational aspects 160
3.11.3 Sequences of hierarchies 165
3.11.4 Optimality properties of higher order kernels 167
3.11.5 The multiple kernel method 171
3.11.6 The estimation procedure for the density and its
derivatives 172
3.12 Exercises 175
4. MEASURES AND RANDOM MEASURES 185
4.1 Introduction 185
4.1.1 Dirac measures 186
4.1.2 General approach 190
4.1.3 The example of moments 192
4.2 Measurability of RKHS-valued variables 194
4.3 Gaussian measure on RKHS 196
4.3.1 Gaussian measure and gaussian process 196
4.3.2 Construction of gaussian measures 198
4.4 Weak convergence in Pr(1l) 199
4.4.1 Weak convergence criterion 202
4.5 Integration of 1l-valued random variables 202
4.5.1 Notation. Definitions 203
x
RKHS IN PROBABILITY AND STATISTICS
4.5.2 Integrability of X and of {Xt : tEE}. 205
4.6 Inner products on sets of measures 210
4.7 Inner product and weak topology 214
4.8 Application to normal approximation 218
4.9 Random measures 220
4.9.1 The empirical measure as ?i-valued variable 223
4.9.1.1 Integrable kernels 224
4.9.1.2 Estimation of IIJ. 228
4.9.2 Convergence of random measures 232
4.10 Exercises 234
5. MISCELLANEOUS APPLICATIONS 241
5.1 Introduction 241
5.2 Law of Iterated Logarithm 241
5.3 Learning and decision theory 245
5.3.1 Binary classification with RKHS 245
5.3.2 Support Vector Machine 248
5.4 ANOVA in function spaces 249
5.4.1 ANOVA decomposition ofa function on a product
domain 249
5.4.2 Tensor product smoothing splines 252
5.4.3 Regression with tensor product splines 254
5.5 Strong approximation in RKHS 255
5.6 Generalized method of moments 259
5.7 Exercises 262
6. COMPUTATIONAL ASPECTS 265
6.1 Kernel of a given normed space 266
6.1.1 Kernel of a finite dimensional space 266
6.1.2 Kernel of some subspaces 266
6.1.3 Decomposition principle 267
6.1.4 Kernel of a class of periodic functions 268
6.1.5 A family of Beppo-Levi spaces 270
6.1.6 Sobolev spaces endowed with a variety of norms 276
6.1.6.1 First family of norms 277
6.1.6.2 Second family of norms 285
6.2 Norm and space corresponding to a given reproducing
kernel 288
6.3 Exercises 289
Contents XI
7. A COLLECTION OF EXAMPLES 293
7.1 Introduction 293
7.2 Using the characterization theorem 293
7.2.1 Case offinite X 294
7.2.2 Case of countably infinite X 294
7.2.3 Using any mapping from E into some pre-Hilbert
space 295
7.3 Factorizable kernels 295
7.4 Examples of spaces, norms and kernels 299
Appendix 344
Introduction to Sobolev spaces 345
A.l Schwartz-distributions or generalized functions 345
A.1.1 Spaces and their topology 345
A.1.2 Weak-derivative or derivative inthesense ofdistributions
346
A.1.3 Facts about Fourier transforms 346
A.2 Sobolev spaces 346
A.2.1 Absolute continuity of functions of one variable 346
A.2.2 Sobolev space with non negative integer exponent 347
A.2.3 Sobolev space with real exponent 348
A.2.4 Periodic Sobolev space 349
A.3 Beppo-Levi spaces 349
Index
353
Preface
The reproducing kernel Hilbert space construction is a bijection or
transform theory which associates a positive definite kernel (gaussian
processes) with a Hilbert space offunctions. Like all transform theories
(think Fourier), problems in one space may become transparent in the
other, and optimal solutions in one space are often usefully optimal in
the other.
The theory was born in complex function theory, abstracted and then
accidently injected into Statistics; Manny Parzen as a graduate student
at Berkeley was given a strip of paper containing his qualifying exam
problem- It read "reproducing kernel Hilbert space"- In the 1950's this
was a truly obscure topic. Parzen tracked it down and internalized the
subject. Soon after, he applied it to problems with the following fla
vor: consider estimating the mean functions of a gaussian process. The
mean functions which cannot be distinguished with probability one are
precisely the functions in the Hilbert space associated to the covariance
kernel of the processes. Parzen's own lively account of his work on re
producing kernels is charmingly told in his interview with H. Joseph
Newton in Statistical Science, 17, 2002, p. 364-366.
Parzen moved to Stanford and his infectious enthusiasm caught Jerry
Sacks,Don Ylvisaker and Grace Wahba among others. Sacks and Ylvis
aker applied the ideas to design problems such as the following. Sup
pose (XdO<t<l is a mean zero stationary gaussian process. We are to
choose observation times to,tll ...,tt« to minimize the L error of esti
2
mate fo1X, Qt dt with Qt a known function. They proved that the min
f
imum error equals the minimax quadrature error in estimating Q L
with L ranging over the unit ball in the associated kernel space. This
equivalence between an L2error and a minimaxerrorisatypicalexample
of the transform theory in action. The Sacks-Ylvisaker work (see Sec
tion 6.3 in Chapter 2) can be found in "Statistical designs and integral
