Computing Functions of Random Variables via Reproducing Kernel Hilbert Space Representations

0.34 MB·English

by  Bernhard Schölkopf

Computing Functions of Random Variables via
Reproducing Kernel Hilbert Space Representations
5 BernhardScho¨lkopf KrikamolMuandet
∗ ∗
1 KenjiFukumizu† JonasPeters‡
0
2
January28,2015

n
a
J

7
2 Abstract

Wedescribeamethodtoperformfunctionaloperationsonprobabilitydistri-
]
L butionsofrandomvariables. ThemethodusesreproducingkernelHilbertspace
M representations of probability distributions, and it is applicable to all operations
whichcanbeappliedtopointsdrawnfromtherespectivedistributions. Werefer
. toourapproachaskernelprobabilisticprogramming. Weillustrateitonsynthetic
t
a data,andshowhowitcanbeusedfornonparametricstructuralequationmodels,
t
s withanapplicationtocausalinference.
[

1
1 Introduction
v
4
9 Data types, derived structures, and associated operations play a crucial role for pro-
7 gramminglanguagesandthecomputationswecarryoutusingthem. Choosingadata
6 type, such as Integer, Float, or String, determines the possible values, as well as the
0
operationsthatanobjectpermits. Operationstypicallyreturntheirresultsintheform
.
1 ofdatatypes. Compositeorderiveddatatypesmaybeconstructedfromsimplerones,
0
alongwithspecialisedoperationsapplicabletothem.
5
Thegoalofthepresentpaperistoproposeawaytorepresentdistributionsoverdata
1
: types,andtogeneralizeoperationsbuiltoriginallyforthedatatypestooperationsap-
v
plicabletothosedistributions. Ourapproachisnonparametricandthusnotconcerned
i
X withwhatdistributionmodelsmakesenseonwhichstatisticaldatatype(e.g.,binary,
r ordinal, categorical). It is also general, in the sense that in principle, it applies to all
a
datatypesandfunctionaloperations. Thepricetopayforthisgeneralityisthat
ourapproachwill,inmostcases,provideapproximateresultsonly;however,we
•
includeastatisticalanalysisoftheconvergencepropertiesofourapproximations,
and
MaxPlanckInstituteforIntelligentSystems,72076Tu¨bingen,Germany
∗
†InstituteforStatisticalMathematics,10-3Midori-cho,Tachikawa,Tokyo,Japan
‡ETHZu¨rich,Seminarfu¨rStatistik,8092Zu¨rich,Switzerland
1
for each data type involved (as either input or output), we require a positive
•
definitekernelcapturinganotionofsimilaritybetweentwoobjectsofthattype.
Some of our results require, moreover, that the kernels be characteristic in the
sensethattheyleadtoinjectivemappingsintoassociatedHilbertspaces.
Inanutshell,ourapproachrepresentsdistributionsoverobjectsaselementsofaHilbert
spacegeneratedbyakernel,anddescribeshowthoseelementsareupdatedbyopera-
tionsavailableforsamplepoints. Ifthekernelistrivialinthesensethateachdistinct
objectisonlysimilartoitself,themethodreducestoaMonteCarloapproachwherethe
operations are applied to sample points which are propagated to the next step. Com-
putationally,werepresenttheHilbertspaceelementsasfiniteweightedsetsofobjects,
and all operations reduce to finite expansions in terms of kernel functions between
objects.
The remainder of the present article is organized as follows. After describing the
necessary preliminaries, we provide an exposition of our approach (Section 3). Sec-
tion4analysesanapplicationtotheproblemofcause-effectinferenceusingstructural
equationmodels. Weconcludewithalimitedsetofexperimentalresults.
2 Kernel Maps
2.1 Positivedefinitekernels
The concept of representing probability distributions in a reproducing kernel Hilbert
space (RKHS) has recently attracted attention in statistical inference and machine
learning [2, 36]. One of the advantages of this approach is that it allows us to apply
RKHS methods to probability distributions, often with strong theoretical guarantees
[44,45]. Ithasbeenappliedsuccessfullyinmanydomainssuchasgraphicalmodels
[38,39],two-sampletesting[11],domainadaptation[16,13,23],andsupervisedlearn-
ingonprobabilitydistributions[24,48]. Webeginbybrieflyreviewingthesemethods,
startingwithsomeprerequisites.
Weassumethatourinputdata x ,...,x liveinanonemptysetX andaregen-
1 m
{ }
erated i.i.d. by a random experiment with Borel probability distribution p. By k, we
denoteapositivedefinitekernelonX X,i.e.,asymmetricfunction
×
k:X X R (1)
× →
(x,x) k(x,x) (2)
(cid:48) (cid:48)
(cid:55)→
satisfyingthefollowingnonnegativitycondition: foranym N,anda ,...,a R,
1 m
∈ ∈
m
∑ aa k(x,x ) 0. (3)
i j i j
≥
i,j=1
If equality in (3) implies that a = =a =0, the kernel is called strictly positive
1 m
···
definite.
2
2.2 Kernelmapsforpoints
Kernelmethodsinmachinelearning,suchasSupportVectorMachinesorKernelPCA,
are based on mapping the data into a reproducing kernel Hilbert space (RKHS) H
[3,34,35,15,47],
Φ:X H (4)
→
x Φ(x), (5)
(cid:55)→
wherethefeaturemap(orkernelmap)Φsatisfies
k(x,x)= Φ(x),Φ(x) (6)
(cid:48) (cid:48)
(cid:104) (cid:105)
for all x,x X. One can show that every k taking the form (6) is positive definite,
(cid:48)
andeveryp∈ositivedefinitek allowstheconstructionofH andΦsatisfying(6). The
canonicalfeaturemap,whichiswhatbydefaultwethinkofwheneverwewriteΦ,is
Φ:X RX (7)
→
x k(x,.), (8)
(cid:55)→
withaninnerproductsatisfyingthereproducingkernelproperty
k(x,x)= k(x,.),k(x,.) . (9)
(cid:48) (cid:48)
(cid:104) (cid:105)
Mappingobservationsx X intoaHilbertspaceisratherconvenientinsomecases.
IftheoriginaldomainX∈hasnolinearstructuretobeginwith(e.g.,ifthexarestrings
or graphs), then the Hilbert space representation provides us with the possibility to
construct geometric algorithms by using the inner product of H. Moreover, even if
X is a linear space in its own right, it can be helpful to use a nonlinear feature map
inordertoconstructalgorithmsthatarelinearinH whilecorrespondingtononlinear
methodsinX.
2.3 Kernelmapsforsetsanddistributions
OnecangeneralizethemapΦtoacceptasinputsnotonlysinglepoints,butalsosets
of points or distributions. It was pointed out that the kernel map of a set of points
X:= x ,...,x ,
1 m
{ } 1 m
µ[X]:= ∑Φ(x), (10)
i
m
i=1
corresponds to a kernel density estimator in the input domain [33, 32], provided the
kernel is nonnegative and integrates to 1. However, the map (10) can be applied for
all positive definite kernels, including ones that take negative values or that are not
normalized. Moreover,thefactthat µ[X]livesinanRKHSandtheuseoftheassoci-
atedinnerproductandnormwillhaveanumberofsubsequentadvantages. Forthese
reasons,itwouldbemisleadingtothinkof(10)simplyasakerneldensityestimator.
Thekernelmapofadistribution pcanbedefinedastheexpectationofthefeature
map[2,36,11],
µ[p]:=E [Φ(x)], (11)
x p
∼
3
k(x,x)= x,x meanofX
(cid:48) (cid:48)
k(x,x)=(cid:104)( x,x(cid:105) +1)n momentsofXuptoordern N
(cid:48) (cid:48)
(cid:104) (cid:105) ∈
k(x,x)strictlyp.d. allofX(i.e.,µ injective)
(cid:48)
Table 1: What information about a sample X does the kernel map µ[X] (see (10))
contain?
k(x,x)= x,x expectationof p
(cid:48) (cid:48)
k(x,x)=(cid:104)( x,x(cid:105) +1)n momentsof puptoordern N
(cid:48) (cid:48)
(cid:104) (cid:105) ∈
k(x,x)characteristic/universal allof p(i.e.,µ injective)
(cid:48)
Table2: Whatinformationabout pdoesthekernelmap µ[p](see(11))contain? For
thenotionsofcharacteristic/universalkernels,see[46,8,7];anexamplethereofisthe
Gaussiankernel(26).
whereweoverloadthesymbolµ andassume,hereandbelow,that pisaBorelproba-
bilitymeasure,and
E [k(x,x)]<∞. (12)
x,x p (cid:48)
(cid:48)∼
AsufficientconditionforthistoholdistheassumptionthatthereexistsanM Rsuch
∈
that
k(x,.) M<∞, (13)
(cid:107) (cid:107)≤
or equivalently k(x,x) M2, on the support of p. Kernel maps for sets of points or
≤
distributionsaresometimesreferredtoaskernelmeanmapstodistinguishthemfrom
theoriginalkernelmap. Note,however,thattheyincludethekernelmapofapointasa
specialcase,sothereissomejustificationinusingthesameterm. If p=p isthelaw
X
ofarandomvariableX,wesometimeswriteµ[X]insteadofµ[p].
Inallcasesitisimportanttounderstandwhatinformationweretain,andwhatwe
lose,whenrepresentinganobjectbyitskernelmap. Wesummarizetheknownresults
[47,8,36,11,45]inTables1and2.
Weconcludethissectionwithadiscussionofhowtousekernelmeanmaps.Tothis
end,firstassumethatΦisinjective,whichisthecaseifkisstrictlypositivedefinite(see
Table1)orcharacteristic/universal(seeTable2). Particularcasesincludethemoment
generatingfunctionofaRVwithdistribution p,
(cid:104) (cid:105)
M (.)=E ex, , (14)
p x p (cid:104) ·(cid:105)
∼
whichequals(11)fork(x,x(cid:48))=e(cid:104)x,x(cid:48)(cid:105)using(8).
Wecanusethemaptotestforequalityofdatasets,
µ[X] µ[X] =0 X=X, (15)
(cid:48) (cid:48)
(cid:107) − (cid:107) ⇐⇒
ordistributions,
µ[p] µ[p] =0 p=p. (16)
(cid:48) (cid:48)
(cid:107) − (cid:107) ⇐⇒
4
Two applications of this idea lead to tests for homogeneity and independence. In the
latter case, we estimate µ[p p ] µ[p ] [1, 12]; in the former case, we estimate
x y xy
(cid:107) − (cid:107)
µ[p] µ[p] [11].
(cid:48)
(cid:107) − (cid:107)
Estimatorsfortheseapplicationscanbeconstructedintermsoftheempiricalmean
estimator(thekernelmeanoftheempiricaldistribution)
1 m
µ[pˆ ]= ∑Φ(x)=µ[X], (17)
m i
m
i=1
where X= x ,...,x is an i.i.d. sample from p (cf. (10)). As an aside, note that
1 m
{ }
using ideas from James-Stein estimation [17], we can construct shrinkage estimators
thatimproveuponthestandardempiricalestimator(seee.g.,[25,26]).
Onecanshowthatµ[pˆ ]convergesatratem 1/2(cf. [36]and[37,Theorem27]):
m −
Theorem1 Assumethat f ∞ 1forall f H with f H 1. Thenwithproba-
(cid:107) (cid:107) ≤ ∈ (cid:107) (cid:107) ≤
bilityatleast1 δ,
−
(cid:114)
2 (cid:104) (cid:105) 2ln(2/δ)
µ[pˆm] µ[p] H E √trK + , (18)
(cid:107) − (cid:107) ≤ m m
whereK :=k(x,x ).
ij i j
Independent of the requirement of injectivity, µ can be used to compute expecta-
tionsofarbitraryfunctions f livingintheRKHS,usingtheidentity
E [f(x)]= µ[p],f , (19)
x p
∼ (cid:104) (cid:105)
whichfollowsfromthefactthatkrepresentspointevaluationintheRKHS,
f(x)= k(x,.),f . (20)
(cid:104) (cid:105)
AsmallRKHS,suchastheonespannedbythelinearkernel
k(x,x)= x,x , (21)
(cid:48) (cid:48)
(cid:104) (cid:105)
may not contain the functions we are interested in. If, on the other hand, our RKHS
is sufficiently rich (e.g., associated with a universal kernel [46]), we can use (19) to
approximate,forinstance,theprobabilityofanyinterval(a,b)onaboundeddomain,
byapproximatingtheindicatorfunctionI(a,b) asakernelexpansion∑ni=1aik(xi,.),and
substitutingthelatterinto(19).See[19]forfurtherdiscussion. Alternatively,ifphasa
density,wecanestimateitusingmethodssuchasreproducingkernelmomentmatching
andcombinationswithkerneldensityestimation[41,19].
Thisshowsthatthemapisnotaone-wayroad: wecanmapourobjectsofinterest
into the RKHS, perform linear algebra and geometry on them [34], and at the end
answerquestionsofinterest. Inthenextsection,weshalltakethisastepfurther,and
discusshowtoimplementrathergeneraloperationsintheRKHS.
Beforedoingso, wementiontwoadditionalapplicationsofkernelmaps. Wecan
map conditional distributions and perform Bayesian updates [8, 51, 9], and we can
connectkernelmapstoFourieroptics,leadingtoaphysicalrealizationasFraunhofer
diffraction[14].
5
3 Computing Functions of Independent Random Vari-
ables
3.1 IntroductionandEarlierWork
Arandomvariable(RV)isameasurablefunctionmappingpossibleoutcomesofanun-
derlyingrandomexperimenttoasetE (often,E Rd,butourapproachwillbemore
⊂
general).Theprobabilitymeasureoftherandomexperimentinducesthedistributionof
therandomvariable. Wewillbelownotdealwiththeunderlyingprobabilityspaceex-
plicitly,andinsteaddirectlystartfromrandomvariablesX,Y withdistributions p ,p
X Y
andvaluesinX,Y. Supposewehaveaccessto(datafrom) p and p ,andwewant
X Y
to compute the distribution of the random variable f(X,Y), where f is a measurable
functiondefinedonX Y.
×
For instance, if our operation is addition f(X,Y)=X+Y, and the distributions
p and p have densities, we can compute the density of the distribution of f(X,Y)
X Y
by convolving those densities. If the distributions of X andY belong to some para-
metric class, such as a class of distributions with Gaussian density functions, and if
the arithmetic expression is elementary, then closed form solutions for certain favor-
ablecombinationsexist. Attheotherendofthespectrum,wecanresorttonumerical
integrationorsamplingtoapproximate f(X,Y).
Arithmetic operations on random variables are abundant in science and engineer-
ing. Measurements in real-world systems are subject to uncertainty, and thus subse-
quentarithmeticoperationsonthesemeasurementsareoperationsonrandomvariables.
Anexampledueto[42]issignalamplification.Considerasetofnamplifiersconnected
togetherinaserialfashion. Iftheamplificationofthei-thamplifierisdenotedbyX,
i
then the total amplification, denoted by Y, is Y =X X X , i.e., a product of n
1 2 n
· ···
randomvariables.
A well-established framework for arithmetic operation on independent random
variables (iRVs) relies on integral transform methods [42]. The above example of
addition already suggests that Fourier transforms may help, and indeed, people have
used transforms such as the ones due to Fourier and Mellin to derive the distribution
functionofeitherthesum,difference,product,orquotientofiRVs[5,43,30,42]. [49]
proposes an approximation using Laguerre polynomials, and a notion of envelopes
bounding the cumulative distribution function. This framework also allows for the
treatmentofdependentrandomvariables,buttheboundscanbecomeverylooseafter
repeatedoperations.[21]approximatetheprobabilitydistributionsoftheinputrandom
variablesasmixturemodels(usinguniformandGaussiandistributions),andapplythe
computationstoallmixturecomponents.
[18]considersanumericalapproachtoimplementarithmeticoperationsoniRVs,
representingthedistributionsusingpiecewiseChebyshevapproximations. Thislends
itselfwelltotheuseofapproximationmethodsthatperformwellaslongasthefunc-
tionsarewell-behaved. Finally,MonteCarloapproachescanbeusedaswell,andare
popularinscientificapplications(seee.g.,[6]).
Thegoalofthepresentpaperistodevelopaderiveddatatyperepresentingadistri-
butionoveranotherdatatype,andtogeneralizetheavailablecomputationaloperations
6
to this data type, at least approximately. This would allow us to conveniently handle
errorpropagationasintheexamplediscussedearlier. Itwouldalsohelpusperformin-
ferenceinvolvingconditionaldistributionsofsuchvariablesgivenobserveddata. The
latter is the main topic of a subject area that has recently begun to attract attention,
probabilistic programming [10]. A variety of probabilistic programming languages
hasbeenproposed[50,27,4]. Toemphasizethecentralrolethatkernelmapsplayin
ourapproach,werefertoitaskernelprobabilisticprogramming(KPP).
3.2 ComputingFunctionsofIndependentRandomVariablesusing
KernelMaps
The key idea of KPP is to provide a consistent estimator of the kernel map of an ex-
pression involving operations on random variables. This is done by applying the ex-
pressiontothesamplepoints,andshowingthattheresultingkernelexpansionhasthe
desiredproperty. OperationsinvolvingmorethanoneRVwillincreasethesizeofthe
expansion, but we can resort to existing RKHS approximation methods to keep the
complexity of the resulting expansion limited, which is advisable in particular if we
want to use it as a basis for further operations. The benefits of KPP are three-fold.
First,wedonotmakeparametricassumptionsonthedistributionsassociatedwiththe
randomvariables. Second, ourapproachappliesnotonlytoreal-valuedrandomvari-
ables, but also to multivariate random variables, structured data, functional data, and
otherdomains,aslongaspositivedefinitekernelscanbedefinedonthedata. Finally,
itdoesnotrequireexplicitdensityestimationasanintermediatestep,whichisdifficult
inhighdimensions.
We begin by describing the basic idea. Let f be a function of two independent
RVs X,Y taking values in the sets X,Y, and suppose we are given i.i.d. m-samples
x ,...,x andy ,...,y . Weareinterestedinthedistributionof f(X,Y), andseekto
1 m 1 m
estimateitsrepresentationµ[f(X,Y)]:=E[Φ(f(X,Y))]intheRKHSas
1 m
∑ Φ(f(x,y )). (22)
m2 i j
i,j=1
Althoughx ,...,x p andy ,...,y p arei.i.d.observations,thisdoesnotim-
1 m X 1 m Y
plythatthe f(x,y∼)i,j=1,...,m for∼mani.i.d.m2-samplefrom f(X,Y),since—
i j
{ | }
looselyspeaking—eachx (andeachy )leavesafootprintinmoftheobservations,
i j
leading to a (possibly weak) dependency. Therefore, Theorem 1 does not imply that
(22)isconsistent. Weneedtodosomeadditionalwork:
Theorem2 GiventwoindependentrandomvariablesX,Y withvaluesinX,Y,mu-
tually independent i.i.d. samples x ,...,x and y ,...,y , a measurable function f :
1 m 1 n
X Y Z,andapositivedefinitekernelonZ Z withRKHSmapΦ,then
× → ×
1 m n
∑∑Φ(f(x,y )) (23)
i j
mn
i=1j=1
isanunbiasedandconsistentestimatorofµ[f(X,Y)].
7
Moreover,wehaveconvergenceinprobability
(cid:13) (cid:13)
(cid:13) 1 m n (cid:13) (cid:18) 1 1 (cid:19)
(cid:13) ∑∑Φ(f(x,y )) E[Φ(f(X,Y))](cid:13)=O + , (m,n ∞). (24)
(cid:13)(cid:13)mni=1j=1 i j − (cid:13)(cid:13) p √m √n →
Asanaside,notethat(23)isanRKHSvaluedtwo-sampleU-statistic.
Proof Foranyi,j,wehaveE[Φ(f(x,y ))]=E[Φ(f(X,Y))],hence(23)isunbiased.
i j
Theconvergence(24)canbeobtainedasacorollarytoTheorem3,andtheproofis
omittedhere.
3.3 ApproximatingExpansions
Tokeepcomputationalcostlimited,weneedtouseapproximationswhenperforming
multi-step operations. If for instance, the outcome of the first step takes the form
(23),thenwealreadyhavem nterms,andsubsequentstepswouldfurtherincreasethe
·
numberofterms,thusquicklybecomingcomputationallyprohibitive.
WecandosobyusingthemethodsdescribedinChapter18of[34].Theyfallintwo
categories.Inreducedsetselectionmethods,weprovideasetofexpansionpoints(e.g.,
allpoints f(x,y )in(23)),andtheapproximationmethodsparsifiesthevectorofex-
i j
pansioncoefficients. Thiscanbeforinstancedonebysolvingeigenvalueproblemsor
linearprograms. Reducedsetconstructionmethods,ontheotherhand,constructnew
expansionpoints. Inthesimplestcase, theyproceedbysequentiallyfindingapproxi-
matepre-imagesofRKHSelements.Theytendtobecomputationallymoredemanding
andsufferfromlocalminima;however,theycanleadtosparserexpansions.
Eitherway,wewillendupwithanapproximation
p
∑γ Φ(z ) (25)
k k
k=1
of (23), where usually p m n. Here, the z are either a subset of the f(x,y ), or
k i j
otherpointsfromZ. (cid:28) ·
Itisinstructivetoconsidersomespecialcases.Forsimplicity,assumethatZ =Rd.
IfweuseaGaussiankernel
k(x,x)=exp( x x 2/(2σ2)) (26)
(cid:48) (cid:48)
−(cid:107) − (cid:107)
whose bandwidth σ is much smaller than the closest pair of sample points, then the
pointsmappedintotheRKHSwillbealmostorthogonalandthereisnowaytosparsify
akernelexpansionsuchas(23)withoutincurringalargeRKHSerror. Inthiscase,we
can identify the estimator with the sample itself, and KPP reduces to a Monte Carlo
method. If, on the other hand, we use a linear kernel k(z,z)= z,z on Z =Rd,
(cid:48) (cid:48)
(cid:104) (cid:105)
then Φ is the identity map and the expansion (23) collapses to one real number, i.e.,
we would effectively represent f(X,Y) by its mean for any further processing. By
choosingkernelsthatlie‘inbetween’thesetwoextremes,weretainavaryingamount
ofinformationwhichwecanthustunetoourwishes,seeTable1.
8
3.4 ComputingFunctionsofRKHSApproximations
Moregenerally,considerapproximationsofkernelmeansµ[X]andµ[Y]
m n
(cid:48) (cid:48)
µˆ[X]:=∑αΦ (x), µˆ[Y]:= ∑β Φ (y ). (27)
i x i(cid:48) j y (cid:48)j
i=1 j=1
In our case, we think of (27) as RKHS-norm approximations of the outcome of pre-
viousoperationsperformedonrandomvariables. Suchapproximationstypicallyhave
coefficients α Rn(cid:48) and β Rm(cid:48) that arenot uniform, that may notsum toone, and
∈ ∈
thatmaytakenegativevalues[34],e.g.,forconditionalmeanmaps[40,9].
Weproposetoapproximatethekernelmeanµ[f(X,Y)]bytheestimator
1 m(cid:48) n(cid:48)
µˆ[f(X,Y)]:= ∑∑αβ Φ (f(x,y )), (28)
∑mi=(cid:48)1αi∑nj=(cid:48) 1βji=1j=1 i j z i(cid:48) (cid:48)j
where the feature map Φ defined on Z, the range of f, may be different from both
z
Φ andΦ . Theexpansionhasm n terms,whichwecansubsequentlyapproximate
x y (cid:48) (cid:48)
·
morecompactlyintheform(25),readyforthenextoperation. Notethat(28)contains
(23)asaspecialcase.
Oneoftheadvantagesofourapproachisthat(23)and(28)applyforgeneraldata
types. In other words, X,Y,Z need not be vector spaces — they may be arbitrary
nonemptysets,aslongaspositivedefinitekernelscanbedefinedonthem.
Convergence analysis in an idealized setting We analyze the convergence of (28)
under the assumption that the expansion points are actually samples x ,...,x from
1 m
X and y ,...,y fromY, which is for instance the case if the expansions (27) are the
1 n
resultofreducedsetselectionmethods(cf.Section3.3). Moreover,weassumethatthe
expansioncoefficientsα ,...,α andβ ,...,β areconstants,i.e.,independentofthe
1 m 1 n
samples.
The following proposition gives a sufficient condition for the approximations in
(27) to converge. Note that below, the coefficients α ,...,α depend on the sample
1 m
sizem,butforsimplicitywerefrainfromwritingthemasα ,...,α ;likewise,for
1,m m,m
β ,...,β . We make this remark to ensure that readers are not puzzled by the below
1 n
statementthat∑m α2 0asm ∞.
i=1 i → →
Proposition1 Letx1,...,xmbeani.i.d.sampleand(αi)mi=1beconstantswith∑mi=1αi=
1.AssumeE[k(X,X)]>E[k(X,X˜)],whereX andX˜ areindependentcopiesofx.Then,
i
theconvergence
(cid:13) (cid:13)2
(cid:13) m (cid:13)
E(cid:13)∑αΦ(x) µ[X](cid:13) 0 (m ∞)
(cid:13) i i (cid:13)
(cid:13)i=1 − (cid:13) → →
holdstrueifandonlyif∑m α2 0asm ∞.
i=1 i → →
Proof Fromtheexpansion
(cid:13) (cid:13)2
(cid:13) m (cid:13)
E(cid:13)∑αΦ(x) µ[X](cid:13)
(cid:13) i i (cid:13)
(cid:13)i=1 − (cid:13)
9
m m
= ∑ αα E[k(x,x )] 2∑αE[k(x,X)]+E[k(X,X˜)]
i s i s i i
−
i,s=1 i=1
(cid:16) (cid:17)2 (cid:16) (cid:17)(cid:110) (cid:111)
= 1 ∑α E[k(X,X˜)]+ ∑α2 E[k(X,X)] E[k(X,X˜)] ,
i i
− −
i i
theassertionisstraightforward.
The next result shows that if our approximations (27) converge in the sense of
Proposition 1, then the estimator (28) (with expansion coefficients summing to 1) is
consistent.
Theorem3 Let x ,...,x and y ,...,y be mutually independent i.i.d. samples, and
1 m 1 n
the constants (αi)mi=1,(βj)nj=1 satisfy ∑mi=1αi = ∑nj=1βj = 1. Assume ∑mi=1αi2 and
∑n β2convergetozeroasn,m ∞. Then
j=1 j →
(cid:13) (cid:13) (cid:32) (cid:33)
(cid:13) m n (cid:13) (cid:114) (cid:114)
(cid:13)∑∑αβ Φ(f(x,y )) µ[f(X,Y)](cid:13)=O ∑α2+ ∑β2
(cid:13)(cid:13)i=1j=1 i j i j − (cid:13)(cid:13) p i i j j
asm,n ∞.
→
Proof Byexpandingandtakingexpectations,onecanseethat
(cid:13) (cid:13)2
(cid:13) m n (cid:13)
E(cid:13)∑∑αβ Φ(f(x,y )) E[Φ(f(X,Y))](cid:13)
(cid:13) i j i j (cid:13)
(cid:13)i=1j=1 − (cid:13)
equals
m n
∑∑α2β2E[k(f(X,Y),f(X,Y))]+∑∑αα β2E[k(f(X,Y),f(X˜,Y))]
i j i s j
i=1j=1 s=i j
(cid:54)
+∑∑α2β βE[k(f(X,Y),f(X,Y˜))]
i j t
i t=j
(cid:54)
+∑∑αα β βE[k(f(X,Y),f(X˜,Y˜))]
i s j t
s=it=j
(cid:54) (cid:54)
2∑∑αβ E[k(f(X,Y),f(X˜,Y˜))]+E[k(f(X,Y),f(X˜,Y˜))]
i j
−
i j
(cid:16) (cid:17)(cid:16) (cid:17)
= ∑α2 ∑β2 E[k(f(X,Y),f(X,Y))]
i j
i j
(cid:40)
(cid:16) (cid:17)2
+ 1 ∑α ∑β +∑α2∑β2
i j i j
−
i j i j
(cid:41)
∑α2(∑β )2 (∑α)2∑β2 E[k(f(X,Y),f(X˜,Y˜))]
i j i j
− −
i j i j
(cid:16) (cid:17)(cid:16) (cid:17)
+ (∑α)2 ∑α2 ∑β2 E[k(f(X,Y),f(X˜,Y))]
i i j
−
i i j
10