Consistency of the group Lasso and multiple kernel learning

0.78 MB·English

by  Francis Bach

Consistency of the Group Lasso
and Multiple Kernel Learning
FrancisR.Bach [email protected]
INRIA-Willowproject
De´partementd’Informatique,EcoleNormaleSupe´rieure
45,Rued’Ulm
8
0 75230Paris,France
0 Abstract
2
We consider the least-square regression problem with regularization by a block ℓ1-norm, i.e., a
n
sum ofEuclideannormsoverspacesofdimensionslargerthanone. Thisproblem,referredto as
a
J thegroupLasso,extendstheusualregularizationbytheℓ1-normwhereallspaceshavedimension
one,whereitiscommonlyreferredtoastheLasso. Inthispaper,westudytheasymptoticmodel
8
2 consistencyofthegroupLasso. Wederivenecessaryandsufficientconditionsfortheconsistency
of group Lasso under practical assumptions, such as model misspecification. When the linear

] predictorsand Euclideannormsare replacedby functionsand reproducingkernelHilbertnorms,
G
the problemis usually referredto as multiple kernellearningand is commonlyused for learning
L
fromheterogeneousdatasourcesandfornonlinearvariableselection. Usingtoolsfromfunctional
.
s analysis, and in particular covarianceoperators, we extend the consistency results to this infinite
c
dimensionalcaseandalsoproposeanadaptiveschemetoobtainaconsistentmodelestimate,even
[
whenthenecessaryconditionrequiredforthenonadaptiveschemeisnotsatisfied.

2
Keywords: Sparsity,regularization,consistency,convexoptimization,covarianceoperators
v
0
9
1. Introduction
3
3
Regularization hasemerged as adominant theme inmachine learning and statistics. Itprovides an
.
7
intuitive and principled tool for learning from high-dimensional data. Regularization by squared
0
7 EuclideannormsorsquaredHilbertiannormshasbeenthoroughly studiedinvarioussettings,from
0 approximation theory to statistics, leading to efficient practical algorithms based on linear alge-
:
v bra and very general theoretical consistency results (TikhonovandArsenin, 1997, Wahba, 1990,
i
X Hastieetal.,2001,Steinwart,2001,CuckerandSmale,2002).
r In recent years, regularization by non Hilbertian norms has generated considerable interest in
a
linearsupervised learning, wherethegoalistopredictaresponse asalinearfunction ofcovariates;
in particular, regularization by the ℓ -norm (equal to the sum of absolute values), a method com-
1
monly referred to as the Lasso (Tibshirani, 1994, Osborneetal., 2000), allows toperform variable
selection. However,regularization bynonHilbertiannormscannotbesolvedempiricallybysimple
linear algebra and instead leads to general convex optimization problems and much of the early
effort has been dedicated to algorithms to solve the optimization problem efficiently. In particular,
theLarsalgorithm ofEfronetal.(2004)allowstofindtheentireregularization path(i.e.,thesetof
solutions forallvaluesoftheregularization parameters) atthecostofasinglematrixinversion.
Asthe consequence ofthe optimality conditions, regularization by the ℓ -norm leads to sparse
1
solutions, i.e., loading vectors with many zeros. Recent works (ZhaoandYu, 2006, YuanandLin,
2007, Zou, 2006, Wainwright, 2006) have looked precisely at the model consistency of the Lasso,
i.e., if we know that the data were generated from a sparse loading vector, does the Lasso actually
recover it when the number of observed data points grows? In the case of a fixed number of co-
1
variates, theLassodoes recover thesparsity pattern ifandonly ifacertain simplecondition onthe
generating covariance matrices is verified (YuanandLin, 2007). In particular, in low correlation
settings, theLassoisindeedconsistent. However,inpresenceofstrongcorrelations, theLassocan-
not be consistent, shedding light on potential problems of such procedures for variable selection.
Adaptive versions where data-dependent weights are added to the ℓ -norm then allow to keep the
1
consistency inallsituations (Zou,2006).
A related Lasso-type procedure is the group Lasso, where the covariates are assumed to be
clustered in groups, and instead of summing the absolute values of each individual loading, the
sum ofEuclidean norms of the loadings in each group is used. Intuitively, this should drive all the
weights in one group to zero together, and thus lead to group selection (YuanandLin, 2006). In
Section 2, we extend the consistency results of the Lasso to the group Lasso, showing that similar
correlation conditions are necessary and sufficient conditions for consistency. The passage from
groups of size one to groups of larger sizes leads however to a slightly weaker result as we can
not get asingle necessary and sufficient condition (in Section 2.4, weshow that the stronger result
similar to the Lasso is not true as soon as one group has dimension larger than one). Also, in our
proofs, we relax the assumptions usually made for such consistency results, i.e., that the model is
completely well-specified (conditional expectation oftheresponse whichislinear inthecovariates
andconstant conditional variance). Inthecontext ofmisspecification, whichisacommonsituation
when applying methods such as the ones presented in this paper, we simply prove convergence
to the best linear predictor (which is assumed to be sparse), both in terms of loading vectors and
sparsity patterns.
The group Lasso essentially replaces groups of size one by groups of size larger than one. It
is natural in this context to allow the size of each group to grow unbounded, i.e., to replace the
sum of Euclidean norms by a sum of appropriate Hilbertian norms. When the Hilbert spaces are
reproducing kernel Hilbert spaces (RKHS), this procedure turns out to be equivalent to learn the
best convex combination of a set of basis kernels, where each kernel corresponds to one Hilber-
tian norm used for regularization (Bachetal., 2004a). This framework, referred to as multiple
kernel learning (Bachetal., 2004a), has applications in kernel selection, data fusion from het-
erogeneous data sources and non linear variable selection (Lanckrietetal., 2004a). In this latter
case, multiple kernel learning can exactly be seen as variable selection in a generalized additive
model (HastieandTibshirani, 1990). We extend the consistency results of the group Lasso to this
non parametric case, by using covariance operators and appropriate notions of functional analysis.
These notions allow tocarry out theanalysis entirely in“primal/input” space, whilethealgorithm
has to work in “dual/feature” space to avoid infinite dimensional optimization. Throughout the
paper, wewillalwaysgobackandforthbetween primalanddualformulations, primalformulation
foranalysisanddualformulation foralgorithms.
Thepaperisorganizedasfollows: inSection2,wepresenttheconsistencyresultsforthegroup
Lasso, while in Section 3, we extend these to Hilbert spaces. Finally, we present the adaptive
schemes in Section 4 and illustrate our set of results with simulations on synthetic examples in
Section5.
2. Consistency oftheGroup Lasso
We consider the problem of predicting a response Y ∈ R from covariates X ∈ Rp, where X has
a block structure with m blocks, i.e., X = (X1⊤,...,Xm⊤)⊤ with each Xj ∈ Rpj, j = 1,...,1m,
2
and m p = p. Unlessotherwise specified, kXkwilldenote theEuclidean normofavector X.
j=1 j
Theonlyassumptions thatwemakeonthejointdistribution P of(X,Y)arethefollowing:
XY
P
(A1) X andY havefinitefourthordermoments: EkXk4 < ∞andEkYk4 < ∞.
(A2) Thejointcovariance matrixΣ = EXX⊤ −(EX)(EX)⊤ ∈ Rp×p isinvertible.
XX
(A3) We let (w,b) ∈ Rp × R denote any minimizer of E(Y − X⊤w − b)2. We assume that
E((Y −w⊤X−b)2|X)isalmostsurelygreaterthanσ2 > 0. WeletdenoteJ = {j,w 6=
min j
0}thesparsitypatternofw.1
Theassumption (A3)doesnotstatethatE(Y|X)isanaffinefunctionofX andthattheconditional
variance is constant, as it is commonly done in most works dealing with consistency for linear
supervised learning. We simply assume that given the best affine predictor of Y given X (defined
by w ∈ Rp and b ∈ R), there is still a strictly positive amount of variance in Y. If (A2) is
satisfied, then the full loading vector w is uniquely defined and is equal to w = (Σ⊤ )−1Σ ,
XX XY
where Σ = E(XY)−(EX)(EY) ∈ Rp. Notethat throughout this paper, wedo include a non
XY
regularized constant term b but since we use a square loss it will optimized out in closed form by
centering thedata. Thusallourconsistency statementswillbestatedonlyfortheloadingvectorw;
corresponding resultsforbthenimmediatelyfollow.
Weoftenusethenotationε = Y −w⊤X−b. Intermsofcovariancematrices,ourassumption
(A3)leads to: Σ = E(εε|X) > σ2 and Σ = 0(but εmightnot ingeneral beindependent
εε|X min εX
fromX).
Applicationsofgroupedvariables Inthis paper, weassume thatthegroupings oftheunivariate
variablesisknownandfixed,i.e.,thegroupstructureisgivenandwewishtoachievesparsityatthe
levelofgroups. Thishasnumerousapplications,e.g.,inspeechandsignalprocessing,wheregroups
mayrepresentdifferentfrequencybands(McAuleyetal.,2005),orbioinformatics(Lanckriet etal.,
2004a) and computer vision (VarmaandRay, 2007, Harchaoui andBach, 2007)where each group
may correspond to different data sources or data types. Note that those different data sources are
sometimesreferredtoasviews(see,e.g.,ZhouandBurges,2007).
Moreover,wealwaysassumethatthenumbermofgroupsisfixedandfinite. Consideringcases
wheremisallowedtogrowwiththenumberofobserveddatapoints,inthelineofMeinshausen andYu
(2006),isoutside thescopeofthispaper.
Notations Throughout thispaper, weconsider theblock covariance matrixΣ withm2 blocks
XX
Σ , i,j = 1,...,m. Werefer to the submatrix composed of all blocks indexed by sets I, J as
XiXj
Σ . Similarly,ourloadingsarevectorsdefinedfollowingblockstructure,w = (w⊤,...,w⊤)⊤
XIXJ 1 m
andwedenotew theelementsindexedbyI. Moreoverwedenote1 thevectorinRq withconstant
I q
components equaltoone,andI theidentity matrixofsizeq.
q
2.1 GroupLasso
We consider independent and identically distributed (i.i.d.) data (x ,y ) ∈ Rp ×R, i = 1,...,n,
i i
sampled from P and the data are given in the form of matrices Y¯ ∈ Rn and X¯ ∈ Rn×p and
XY
wewriteX¯ = (X¯1,...,X¯m)whereeachX¯j ∈ Rn×pj represents thedataassociated withgroup j.
1. Notethatthroughoutthispaper,weuseboldfacefontsforpopulationquantities.
3
Throughoutthispaper,wemakethesamei.i.d. assumption;dealingwithnonidenticallydistributed
ordependent dataandextending ourresultsinthosesituations areleftforfutureresearch.
Weconsiderthefollowingoptimization problem:
m
1
min kY¯ −X¯w−b1 k2+λ d kw k,
n n j j
w∈Rp,b∈R 2n
j=1
X
where d ∈ Rm is a vector of strictly positive fixed weights. Note that considering weights in
the block ℓ -norm is important in practice as those have an influence regarding the consistency of
1
the estimator (see Section 4 for further details). Since b is not regularized, we can minimize in
closed form with respect to b, by setting b = 11⊤(Y¯ −X¯w). This leads to the following reduced
n n
optimization problem inw:
m
1 1
min Σˆ −Σˆ⊤ w+ w⊤Σˆ w+λ d kw k, (1)
w∈Rp 2 YY XY 2 XX n j j
j=1
X
where Σˆ = 1Y¯⊤Π Y¯, Σˆ = 1X¯⊤Π Y¯ and Σˆ = 1X¯⊤Π X¯ are empirical covariance
YY n n XY n n XX n n
matrices(withthecenteringmatrixΠ definedasΠ = I − 11 1⊤). Wedenotewˆanyminimizer
n n n n n n
ofEq.(1). Werefertowˆ asthegroupLassoestimate2. Notethatwithprobability tendingtoone,if
(A2)issatisfied(i.e.,ifΣ isinvertible), thereisauniqueminimum.
XX
Problem(1)isanon-differentiable convexoptimizationproblem,forwhichclassicaltoolsfrom
convex optimization (BoydandVandenberghe, 2003) lead to the following optimality conditions
(seeproofbyYuanandLin(2006)andinAppendix A.1):
Proposition 1 A vector w ∈ Rp with sparsity pattern J = J(w) = {j, w 6= 0} is optimal for
j
problem (1)ifandonlyif
∀j ∈ Jc, Σˆ w−Σˆ 6 λ d , (2)
XjX XjY n j
(cid:13) (cid:13) λ d
∀j ∈ J, Σ(cid:13)ˆ w−Σˆ =(cid:13)−w n j. (3)
(cid:13)XjX XjY (cid:13) jkw k
j
2.2 Algorithms
Efficient exact algorithms exist for the regular Lasso, i.e., for the case where all group dimensions
p are equal to one. They are based on the piecewise linearity of the set of solutions as a func-
j
tion of the regularization parameter λ (Efronetal., 2004). For the group Lasso, however, the
n
path is only piecewise differentiable, and following such a path is not as efficient as for the Lasso.
Other algorithms have been designed to solve problem (1) for a single value of λ , in the original
n
group Lasso setting (YuanandLin, 2006) and in the multiple kernel setting (Bachetal., 2004a,b,
Sonnenburg etal., 2006, Rakotomamonjy etal., 2007). In this paper, westudy path consistency of
the group Lasso and of multiple kernel learning, and in simulations we use the publicly available
code for the algorithm of Bachetal. (2004b), that computes an approximate but entire path, by
followingthepiecewisesmoothpathwithpredictor-corrector methods.
2.Weusetheconventionthatall“hat”notationscorrespondtodata-dependentandthusn-dependentquantities,sowe
donotneedtheexplicitdependenceonn.
4
2.3 ConsistencyResults
Weconsiderthefollowingtwoconditions:
1
mi∈aJxc di ΣXiXJΣ−X1JXJDiag(dj/kwjk)wJ <1, (4)
(cid:13) (cid:13)
1 (cid:13) (cid:13)
mi∈aJxc di (cid:13)ΣXiXJΣ−X1JXJDiag(dj/kwjk)wJ(cid:13) 61, (5)
(cid:13) (cid:13)
where Diag(d /kw k) denotes t(cid:13)he block-diagonal matrix (with blo(cid:13)ck sizes p ) in which each di-
j j (cid:13) (cid:13) j
agonal block is equal to kwdjjkIpj (with Ipj the identity matrix of size pj), and wJ denotes the
concatenation of theloading vectors indexed byJ. Notethat the conditions involve the covariance
betweenallactivegroups X ,j ∈ Jandallnonactivegroups X ,i ∈ Jc.
j i
These are conditions on both the input (through the joint covariance matrix Σ ) and on the
XX
weightvector w. Notethat, whenallblocks havesize1,thiscorresponds totheconditions derived
for the Lasso (ZhaoandYu, 2006, YuanandLin, 2007, Zou, 2006). Note also the difference be-
tween the strong condition (4) and the weak condition (5). For the Lasso, with our assumptions,
YuanandLin (2007) has shown that the strong condition (4) is necessary and sufficient for path
consistency of the Lasso; i.e., the path of solutions consistently contains an estimate which isboth
consistent for the 2-norm (regular consistency) and the ℓ -norm (consistency of patterns), if and
0
onlyifcondition (4)issatisfied.
Inthecase ofthegroup Lasso, even withafinitefixednumberofgroups, ourresults arenotas
strong, aswecanonly get thestrict condition assufficient and theweakcondition asnecessary. In
Section2.4,weshowthatthiscannotbeimprovedingeneral. Morepreciselythefollowingtheorem,
proved in Appendix B.1, shows that if the condition (4) is satisfied, any regularization parameter
thatsatisfiesacertaindecayconditions willleadtoaconsistent estimator;thusthestrongcondition
(4)issufficientforpathconsistency:
Theorem2 Assume(A1-3). Ifcondition(4)issatisfied,thenforanysequenceλ suchthatλ → 0
n n
and λ n1/2 → +∞, then the group Lasso estimate wˆ defined in Eq. (1) converges in probability
n
to w and the group sparsity pattern J(wˆ) = {j,wˆ 6= 0} converges in probability to J (i.e.,
j
P(J(wˆ) = J)→ 1).
The following theorem, proved in Appendix B.2, states that if there is a consistent solution on
thepath,thentheweakcondition (5)mustbesatisfied.
Theorem3 Assume (A1-3). If there exists a (possibly data-dependent) sequence λ such that wˆ
n
converges towandJ(wˆ)converges toJinprobability, thencondition (5)issatisfied.
On the one hand, Theorem 2 states that under the “low correlation between variables in J and
variables in Jc” condition (4), the group Lasso is indeed consistent. On the other hand, the re-
sult (and the similar one for the Lasso) is rather disappointing regarding the applicability of the
group Lassoasapractical groupselection method, asTheorem 3statesthatiftheweakcorrelation
condition (5)isnotsatisfied, wecannothaveconsistency.
Moreover, this is to be contrasted with a thresholding procedure of the joint least-square esti-
mator, whichisalsoconsistent withnoconditions (buttheinvertibility ofΣ ),ifthethreshold is
XX
properly chosen (smaller than the smallest norm kw k for j ∈ J or with appropriate decay condi-
j
tions). However, the Lasso and group Lasso do not have to set such athreshold; moreover, further
5
analysis show that the Lasso has additional advantages over regular regularized least-square pro-
cedure (Meinshausen andYu, 2006), and empirical evidence shows that in the finite sample case,
they do perform better (Tibshirani, 1994), in particular in the case where the number m of groups
is allowed to grow. In this paper wefocus on the extension from uni-dimensional groups to multi-
dimensional groups forfinitenumber ofgroups mandleave the possibility ofletting mgrow with
nforfutureresearch.
Finally, by looking carefully at condition (4) and (5), we can see that if we were to increase
the weight d for j ∈ Jc and decrease the weights otherwise, we could always be consistent: this
j
howeverrequires the (potentially empirical) knowledge ofJand this isexactly theidea behind the
adaptiveschemethatwepresent inSection4. Beforelooking attheseextensions, wediscussinthe
nextSection,qualitative differences betweenourresultsandthecorresponding onesfortheLasso.
2.4 RefinementsofConsistencyConditions
Our current results state that the strict condition (4) is sufficient for joint consistency of the group
Lasso, while the weak condition (5) is only necessary. When all groups have dimension one, then
thestrictcondition turnsouttobealsonecessary (YuanandLin,2007).
The main technical reason for those differences is that in dimension one, the set of vectors
of unit norm is finite (two possible values), and thus regular squared norm consistency leads to
estimatesofthesignsoftheloadings(i.e.,theirnormalizedversionswˆ /kwˆ k)whichareultimately
j j
constant. Whengroupshavesizelargerthanone,thenwˆ /kwˆ kwillnotbeultimatelyconstant(just
j j
consistent) andthisaddeddependence ondataleadstothefollowing refinementofTheorem 2(see
proofinAppendixB.3):
Theorem4 Assume(A1-3). Assumetheweakcondition (5)issatisfied andthatforalli ∈ Jc such
that d1i ΣXiXJΣ−X1JXJDiag(dj/kwjk)wJ = 1,wehave
(cid:13) (cid:13)
(cid:13) (cid:13)
(cid:13) (cid:13) w w⊤
∆⊤Σ Σ Σ−1 Diag d /kw k I − j j ∆ > 0, (6)
XJXi XiXJ XJXJ j j pj w⊤w
" j j!#
with∆ = −Σ−X1JXJDiag(dj/kwjk)wJ. Thenforanysequenceλnsuchthatλn → 0andλnn1/4 →
+∞,thenthegroupLassoestimatewˆdefinedinEq.(1)convergesinprobabilitytowandthegroup
sparsity patternJ(wˆ) = {j,wˆ 6= 0}converges inprobability toJ.
j
This theorem is of lower practical significance than Theorem 2 and Theorem 3. It merely shows
that the link between strict/weak conditions and sufficient/necessary conditions arein asense tight
(assoon asthereexists j ∈ Jsuch thatp > 1, itiseasy toexhibit examples whereEq.(6)isoris
j
not satisfied). Theprevious theorem does not contradict the fact that condition (4)is necessary for
w w⊤
path-consistency in the Lasso case: indeed, if w has dimension one, then I − j j is always
j pj w⊤w
j j
equal to zero, and thus Eq. (6) is never satisfied. Note that when condition (6) is an equality, we
could still refine the condition by using higher orders in the asymptotic expansions presented in
AppendixB.3.
WecanalsofurtherrefinedthenecessaryconditionresultsinTheorem3: asstatedinTheorem3,
thegroupLassoestimatormaybebothconsistent intermsofnormandsparsitypatterns onlyifthe
condition (5) is satisfied. However, if we require only the consistent sparsity pattern estimation,
6
thenwemayallowtheconvergenceoftheregularizationparameterλ toastrictlypositivelimitλ .
n 0
Inthissituation, wemayconsider thefollowingpopulation problem:
m
1
min (w−w)⊤Σ (w−w)+λ d kw k. (7)
XX 0 j j
w∈Rp 2
j=1
X
If there exists λ > 0 such that the solution has the correct sparsity pattern, then the group Lasso
0
estimate with λ → λ , will have a consistent sparsity pattern. The following proposition, which
n 0
canbeprovedwithstandard M-estimation arguments, makethisprecise:
Proposition 5 Assume(A1-3). Ifλ tends toλ > 0, thenthegroup Lasso estimate wˆ issparsity-
n 0
consistent ifandonlyifthesolution ofEq.(7)hasthecorrectsparsity pattern.
Thus,evenwhencondition(5)isnotsatisfied,wemayhaveconsistentestimationofthesparsity
patternbutinconsistent estimation oftheloading vectors. WeprovideinSection5suchexamples.
2.5 ProbabilityofCorrectPatternSelection
In this section, we focus on regularization parameters that tend to zero, at the rate n−1/2, i.e.,
λ = λ n−1/2 with λ > 0. For this particular setting, we can actually compute the limit of the
n 0 0
probability ofcorrect pattern selection (proposition proved inAppendix B.4). Notethat inorder to
obtainasimplerresult,weassumeconstant conditional varianceofY givenw⊤X:
Proposition 6 Assume (A1-3) and var(Y|w⊤x) = σ2 almost surely. Assume moreover λ =
n
λ n−1/2 with λ > 0. Then, the group Lasso wˆ converges in probability to w and the probability
0 0
ofcorrectsparsity patternselection hasthefollowinglimit:
1 σ d
P(cid:18)mi∈aJxc di (cid:13)λ0ti−ΣXiXJΣ−X1JXJDiag(kwjjk)wJ(cid:13)6 1(cid:19), (8)
(cid:13) (cid:13)
(cid:13) (cid:13)
where t is normally distributed(cid:13)with mean zero and covariance matrix(cid:13)ΣXJcXJc|XJ = ΣXJcXJc −
ΣXJcXJΣ−X1JXJΣXJXJc (whichistheconditional covariance matrixofXJc givenXJ).
The previous theorem states that the probability of correct selection tends to the mass under a non
degenerate multivariate distribution of the intersection of cylinders. Under our assumptions, this
setisneveremptyandthusthelimitingprobability isstrictlypositive, i.e.,thereis(asymptotically)
alwaysapositiveprobability ofestimatingthecorrectpatternofgroups.
Moreover,additional insightsmaybegainedfromProposition6,namelyintermsofthedepen-
denceonσ,λ andthetightnessoftheconsistency conditions. First,whenλ tendstoinfinity,then
0 0
thelimitdefinedinEq.(8)tends tooneifthestrict consistency condition (4)issatisfied, andtends
tozeroifoneoftheconditions isstrictlynotmet. Thiscorroborates theresultsofTheorem2and3.
Note however, that only an extension of Proposition 6 to λ that may deviate from a n−1/2 would
n
actually leadtoaproofofTheorem2,whichisasubjectofongoing research.
Finally, Eq. (8) shows that σ has a smoothing effect on the probability of correct pattern se-
lection, i.e., if condition (4) is satisfied, then this probability is a decreasing function of σ (and an
increasing functionofλ ). Finally,thestrictertheinequality inEq.(4),thelargertheprobability of
0
correctrankselection, whichisillustrated inSection5onsynthetic examples.
7
2.6 LoadingIndependentSufficientCondition
Condition(4)dependsontheloadingvectorwandonthesparsitypatternJ,whicharebothapriori
unknown. Inthissection,weconsidersufficientconditionsthatdonotdependontheloadingvector,
butonlyonthesparsitypatternJandofcourseonthecovariancematrices. Thefollowingcondition
is sufficient for consistency of the group Lasso, for all possible loading vectors w with sparsity
patternJ:
1
C(ΣXX,d,J) = mi∈aJxc ∀j∈Jm,kauxjk=1(cid:13)diΣXiXJΣ−X1JXJDiag(dj)uJ(cid:13) < 1. (9)
(cid:13) (cid:13)
As opposed to the Lasso case, C(ΣXX,d,J)(cid:13)cannot be readily computed in(cid:13)closed form, but
(cid:13) (cid:13)
wehavethefollowingupperbound:
1
C(Σ ,d,J) 6 max d Σ Σ−1 ,
XX i∈Jc di Xj∈J j(cid:13)(cid:13)Xk∈J XiXk (cid:16) XJXJ(cid:17)kj(cid:13)(cid:13)
(cid:13) (cid:13)
where for a matrix M, kMk denotes its maximal sin(cid:13)gular value (also known(cid:13)as its spectral norm).
(cid:13) (cid:13)
This leads to the following sufficient condition for consistency of the group Lasso (which extends
thecondition ofYuanandLin,2007):
1
max d Σ Σ−1 < 1. (10)
i∈Jc di Xj∈J j(cid:13)(cid:13)Xk∈J XiXk(cid:16) XJXJ(cid:17)kj(cid:13)(cid:13)
(cid:13) (cid:13)
Given a set of weights d, better sufficien(cid:13)t conditions than Eq. (10(cid:13)) may be obtained by solving a
(cid:13) (cid:13)
semidefiniteprogramming problem (BoydandVandenberghe,2003):
2
Proposition 7 Thequantity∀j∈Jm,kauxjk=1 ΣXiXJΣ−X1JXJDiag(dj)uJ isupperbounded by
(cid:13) (cid:13)
(cid:13) (cid:13)
max trM Diag(d(cid:13))Σ−1 Σ Σ Σ−1(cid:13) Diag(d ) , (11)
M<0,trMii=1 j XJXJ XJXi XiXJ XJXJ j
(cid:16) (cid:17)
whereM isamatrixdefinedbyblocksfollowingtheblockstructureofΣ . Moreover,thebound
XJXJ
isalsoequalto
m
min λ .
j
λ∈Rm,Diag(dj)Σ−X1JXJΣXJXiΣXiXJΣ−X1JXJDiag(dj)4Diag(λ)Xj=1
Proof WeletdenoteM = uu⊤ < 0. Thenifallu forj ∈ Jhavenorm1,thenwehavetrM = 1
j jj
for all j ∈ J. This implies the convex relaxation. The second problem is easily obtained as the
convexdualofthefirstproblem(BoydandVandenberghe,2003).
Note that for the Lasso, the convex bound in Eq. (11) is tight and leads to the bound given above
in Eq. (10) (YuanandLin, 2007, Wainwright, 2006). For the Lasso, ZhaoandYu (2006) consider
several particular patterns of dependencies using Eq. (10). Note that this condition (and not the
condition in Eq. (9)) is independent from the dimension and thus does not readily lead to rules of
thumbsallowingtosettheweightd asafunctionofthedimensionp ;severalrulesofthumbshave
j j
been suggested, that loosely depend on the dimension on the blocks, in the context of the linear
groupLasso(YuanandLin,2006)ormultiplekernellearning(Bachetal.,2004b);weargueinthis
paper, thatweightsshould alsodependontheresponse aswell(seeSection4).
8
2.7 AlternativeFormulationoftheGroupLasso
Following Bachetal. (2004a), we can instead consider regularization by the square of the block
ℓ -norm:
1
2
m
1 1
min kY¯ −X¯w−b1 k2+ µ d kw k .
n n j j
w∈Rp,b∈R 2n 2  
j=1
X
 
This leads to the same path of solutions, but it is better behaved because each variable which is
not zero is still regularized by the squared norm. The alternative version has also two advan-
tages: (a) it has very close links to more general frameworks for learning the kernel matrix from
data (Lanckrietetal., 2004b), and (b) it is essential in our proof of consistency in the functional
case. Wealso getthe equivalent formulation toEq.(1),byminimizing inclosed form withrespect
tob,toobtain:
2
m
1 1 1
min Σˆ −Σˆ w+ w⊤Σˆ w+ µ d kw k . (12)
YY YX XX n j j
w∈Rp 2 2 2  
j=1
X
 
Thefollowing proposition givesthe optimality conditions fortheconvex optimization problem de-
finedinEq.(12)(seeproofinAppendixA.2):
Proposition 8 Avectorw ∈ Rp withsparsitypatternJ = {j, w 6= 0}isoptimalforproblem(12)
j
ifandonlyif
∀j ∈ Jc, Σˆ w−Σˆ 6 µ d ( n d kw k), (13)
XjX XjY n j i=1 i i
∀j ∈ J, Σ(cid:13)(cid:13)ˆ w−Σˆ =(cid:13)(cid:13)−µ ( nPd kw k) djwj. (14)
(cid:13)XjX XjY (cid:13) n i=1 i i kw k
j
P
Note the correspondence at the optimum between optimal solutions of the two optimization prob-
lems in Eq. (1) and Eq. (12) through λ = µ ( n d kw k). As far as consistency results are
n n i=1 i i
concerned,Theorem3immediatelyappliestothealternativeformulationbecausetheregularization
P
paths are the same. For Theorem 2, it does not readily apply. But since the relationship between
λ and µ at optimum is λ = µ ( n d kw k) and that n d kwˆ k converges to a constant
n n n n i=1 i i i=1 i i
wheneverwˆ isconsistent, itdoesapplyaswellwithminormodifications(inparticular, todealwith
P P
thecasewhereJisempty,whichrequiresµ = ∞).
n
3. CovarianceOperators andMultipleKernelLearning
Wenowextendthepreviousconsistencyresultstothecaseofnon-parametricestimation,whereeach
group is a potentially infinite dimensional space of functions. Namely, the non parametric group
Lassoaimsatestimatingasparselinearcombinationoffunctionsofseparaterandomvariables,and
canthenbeseenasavariableselectionmethodinageneralizedadditivemodel(HastieandTibshirani,
1990). Moreover, as shown in Section 3.5, the non-parametric group Lasso may also be seen as
equivalent tolearning aconvex combination of kernels, aframework referred toas multiple kernel
learning(MKL).Inthiscontextitiscustomarytohaveasingleinputspacewithseveralkernels(and
hence Hilbertspaces) definedonthesameinput space(Lanckriet etal.,2004b,Bachetal.,2004a).
9
Our framework accomodates this case as well, but our assumption (A5) regarding the invertibility
ofthejointcorrelation operator statesthatthekernelscannotspanHilbertspaceswhichintersect.
Inthisnonparametriccontext,covarianceoperatorsconstituteappropriatetoolsforthestatistical
analysis and are becoming standard in the theoretical analysis of kernel methods (Fukumizuetal.,
2004, Grettonetal., 2005, Fukumizuetal., 2007, Caponnetto anddeVito, 2005). The following
sectionreviewsimportantconcepts. Formoredetails,seeBaker(1973)andFukumizuetal.(2004).
3.1 ReviewofCovarianceOperatorTheory
In this section, we first consider a single set X and a positive definite kernel k : X × X → R,
associated with the reproducing kernel Hilbert space (RKHS) F of functions from X to R (see,
e.g., Scho¨lkopfandSmola (2001) or BerlinetandThomas-Agnan (2003) for an introduction to
RKHS theory). The Hilbert space and its dot product h·,·i are such that for all x ∈ X, then
F
k(·,x) ∈ F and for all f ∈ F, hk(·,x),fi = f(x), which leads to the reproducing property
F
hk(·,x),k(·,y)i = k(x,y)forany(x,y) ∈ X ×X.
F
Covariance operator and norms Given a random variable X on X with bounded second order
moment, i.e., such that Ek(X,X) < ∞, we can define the covariance operator as the bounded
linearoperatorΣ fromF toF suchthatforall(f,g) ∈ F ×F,
XX
hf,Σ gi = cov(f(X),g(X)) =E(f(X)g(X))−(Ef(X))(Eg(X)).
XX F
TheoperatorΣ isauto-adjoint,non-negativeandHilbert-Schmidt,i.e.,foranyorthonormalbasis
XX
(e ) ofF,then ∞ kΣ e k2 isfinite;inthiscase,thevaluedoesnotdependonthechosen
p p>1 p=1 XX p F
basisandisreferredtoasthesquareoftheHilbert-Schmidtnorm. Thenormthatweusebydefault
P
inthis paper isthe operator norm kΣ k = sup kΣ fk , whichis dominated by
XX F f∈F,kfkF=1 XX F
the Hilbert-Schmidt norm. Note that in the finite dimensional case where X = Rp, p > 0 and the
kernel is linear, the covariance operator is exactly the covariance matrix, and the Hilbert-Schmidt
norm istheFrobenius norm,whiletheoperator norm isthemaximum singular value (alsoreferred
toasthespectralnorm).
Thenullspaceofthecovarianceoperatoristhespaceoffunctionsf ∈ F suchthatvarf(X)=
0,i.e.,suchthatf isconstant onthesupport ofX.
Empirical estimators Given data x ∈ X,i = 1,...,n sampled i.i.d. from P , then the empir-
i X
ical estimate Σˆ of Σ is defined such that hf,Σˆ gi is the empirical covariance between
XX XX XX F
f(X)andg(X),whichleadsto:
n n n
1 1 1
Σˆ = k(·,x )⊗k(·,x )− k(·,x )⊗ k(·,x ),
XX i i i i
n n n
i=1 i=1 i=1
X X X
whereu⊗vistheoperatordefinedbyhf,(u⊗v)gi = hf,ui hg,vi . Ifwefurtherassumethatthe
F F F
fourth order moment is finite, i.e., Ek(X,X)2 < ∞, then the estimate is uniformly consistent i.e.,
kΣˆ −Σ k = O (n−1/2)(seeFukumizuetal.(2007)andAppendixC.1),whichgeneralizes
XX XX F p
theusualresultoffinitedimension.3
3.ArandomvariableZ issaidtobeoforderO (a )ifforanyη >0,thereexistsM >0suchthatsup P(|Z |>
n p n n n
Ma )<η.SeeVanderVaart(1998)forfurtherdefinitionsandpropertiesofasymptoticsinprobability.
n
10