Degrees of Freedom for Piecewise Lipschitz Estimators

0.89 MB·

by  Frederik Riis Mikkelsen

DEGREES OF FREEDOM FOR PIECEWISE LIPSCHITZ
ESTIMATORS
FREDERIKRIISMIKKELSENANDNIELSRICHARDHANSEN
6
1
0 Abstract. ArepresentationofthedegreesoffreedomakintoStein’slemma
2 is given for a class of estimators of a mean value parameter in Rn. Contrary
topreviousresultsourrepresentationholdsforarangeofdiscontinuesestima-
n
tors. It shows that even though the discontinuities form a Lebesgue null set,
u
theycannotbeignoredwhencomputingdegreesoffreedom. Estimatorswith
J
discontinuitiesarisenaturallyinregressionifdatadrivenvariableselectionis

0 used. Two such examples, namely best subset selection and debiased lasso,
1 are considered in detail in this paper. For debiased lasso the general repre-
sentation leads to an estimate of the degrees of freedom based on the lasso

] solution path, which in turn can be used for estimating the risk of debiased
T
lasso. Asimilarestimateisproposedforbestsubsetselection. Theusefulness
S oftheriskestimatesforselectingthenumberofvariablesisdemonstratedvia
. simulationswithaparticularfocusondebiasedlasso.
h
t
a
m
[
1. Introduction
2
v Representations of the effective dimension of a statistical model have been stud-
4 ied extensively in many different frameworks. For classical model selection criteria
2
such as AIC and Mallows’s C the dimension of the parameter space is used to
5 p
adjust the empirical risk for its optimism so as to provide a fair model score across
3
0 different dimensions. A number of extensions to models or methods without a well
. defined dimension exist, such as the trace of the smoother matrix for scatter plot
1
0 smoothers, see e.g. Hastie & Tibshirani (1990), and the use of the divergence of a
6 sufficiently differentiable estimator based on Stein’s lemma as described in Efron
1 (2004). Stein’s lemma was used by Zou et al. (2007) and Tibshirani & Taylor
:
v (2012)todemonstratethatforthelassoestimatorinalinearregressionmodelwith
i Gaussian errors, the number of estimated non-zero parameters is an appropriate
X
estimate of the effective dimension.
r
ItiswellknownthatneitherMallows’sC norAICorrelatedinformationcriteria
a p
correctly adjust for the optimism that results from selecting one model among
a number of models of equal dimension. The usage of such methods for model
selectionwithoutadequateadjustmentswascalled“aquietscandalinthestatistical
community” by Breiman (1992), who proposed a bootstrap based method for risk
estimation as an alternative. Ye (1998) defined the notion of generalized degrees
of freedom for an estimator of the mean in a Gaussian model and showed how
to use this number for risk estimation. The results by Ye apply to discontinuous
2010 Mathematics Subject Classification. 62J05,62J07.
Key words and phrases. best subset selection, debiased lasso, degrees of freedom, Stein’s
Lemma.
1
2 F.R.MIKKELSENANDN.R.HANSEN
estimatorsthatinvolvemodelselection,buthisproposalforcomputingthedegrees
of freedomwassimilarly to Breiman’sbasedon refittingmodels to perturbed data.
If the estimator satisfies the differentiability requirements for Stein’s lemma,
Lemma 2 in Stein (1981), the divergence of the estimator w.r.t. the data is an
unbiased estimate of the degrees of freedom in the generalized sense of Ye (1998).
This was used by Meyer & Woodroofe (2000), Zou et al. (2007), Kato (2009) and
Tibshirani & Taylor (2012) among others to derive formulas for the degrees of
freedom of estimators that are Lipschitz continuous.
ForestimatorswithdiscontinuitiesStein’slemmagenerallybreaksdownandthe
divergence will not be an unbiased estimate of the degrees of freedom. Note that
an estimator can be continuous or even differentiable almost everywhere – it can
beaprojectionlocally–andstillbedefinedgloballyinsuchawaythatithasnon-
ignorable discontinuities. This is, in particular, the case in regression when data
adaptive variable selection is used to select among a number of projection estima-
tors. Bestsubsetselectionisonecentralexample,butvariableselectionprocedures
lead in general to non-ignorable discontinuities. A variable selection procedure ef-
fectively divides the sample space into a finite number of disjoint regions, with the
estimator being a projection, say, on each region. The resulting estimator consist-
ing of a selection step and a projection step will generally be discontinuous on the
boundary between two regions.
Estimators defined as solutions of non-convex optimization problems may have
non-ignorable discontinuties as well. In regression, non-convexity can arise from
nonlinear parametrizations of the mean value or from the use of data adaptive or
non-convexpenalties. ExamplesofthelatterincludeSCAD,MCPandk-normsfor
k [0,1). The non-convexity of the penalties is closely related to their abilities to
∈
perform variable selection. Penalization with the 0-norm yields, for instance, the
Lagrangian formulation of best subset selection.
Tibshirani (2015) recently made headway with the computation of the degrees
of freedom for some discontinuous estimators. Specifically, he considered a linear
regressionmodelwithanorthogonaldesignandshowedhowtocomputethedegrees
of freedom for hard thresholding, which for orthogonal designs is equivalent to the
Lagrangianformulationofbestsubsetselection. HealsogaveanextensionofStein’s
lemma tosome discontinuous estimators, though itwasnot shown ifthis extension
applies to subset selection estimators. Hansen & Sokol (2014) gave a different
generalizationofStein’slemmaforallestimatorsthataremetricprojectionsontoa
closedset. Thisgeneralizationappliestosubsetselectionandotherestimatorswith
non-convex constraints, but did not lead to a readily computable representation of
thecontributiontothedegreesoffreedomthatareduetothediscontinuitiesofthe
metric projection.
ThemainresultinthispaperisTheorem2.4,whichisaversionofStein’slemma
for estimators that are locally Lipschitz continuous on each of a finite number
of open sets, whose union makes up almost all of Rn. This is a general class
containing a number of regression estimators that include variable selection. As
a main example the debiased lasso estimator in a linear regression setup – closely
related to the LARS–OLS hybrid in Efron et al. (2004) and a limit case of the
relaxed lasso as considered in Meinshausen (2007) – is investigated in detail in
Section 3. For the debiased lasso an estimate of the degrees of freedom is derived,
whichonlyinvolvesthecomputationofthelassosolutionpath. Asimulationstudy
DEGREES OF FREEDOM FOR PIECEWISE LIPSCHITZ ESTIMATORS 3
reportedinSection4demonstratedthattheresultingriskestimateleadstoreliable
model selection across a range of different designs and parameter settings.
For the Lagrangian formulation of best subset selection the general representa-
tion of degrees of freedom holds, but the situation is more complicated than for
debiased lasso. However, it is possible to derive an approximation, which is exact
for orthogonal designs, as shown in Section 5. The proof of Theorem 2.4 and some
auxiliary technical results and arguments are in the appendix.
2. A general representation of degrees of freedom
Throughout the paper we consider the multivariate normal model (ξ,σ2I) on
N
Rn with ξ the unknown parameter, and we let ξˆ:Rn Rn denote an estimator of
ξ. A typical application is to linear regression estimat→ors of the form Xβˆ where X
denotesann pmatrixandβˆdenotesanestimatoroftheparametersinthelinear
regressionmo×del. Whentheestimatorβˆsetssomeoftheparameterstoexactlyzero
we say that the estimator does variable selection. The lasso, Tibshirani (1996), is
anexampleofagloballyLipschitzcontinuousestimatorthatdoesvariableselection,
whilebestsubsetselectionisadiscontinuousestimatorthatdoesvariableselection.
The debiased lasso – as studied intensively in Section 3 – is another example of a
discontinuousregressionestimatorthatdoesvariableselection. Thoughdiscontinu-
ous regression estimators that do variable selection constitute the main motivation
forthepresentpaper,thegeneralresultsaremoreconvenientlyformulatedinterms
of estimators of the mean ξ without reference to the regression setup.
Letting Y (ξ,σ2I) the risk of the estimator is defined as
∼N
Risk(ξˆ):=E ξ ξˆ(Y) 2,
k − k2
providedthatξˆ(Y)hasfinitesecondmoment,whichwillthusbeassumedthrough-
out. The risk is a quantification of the error of ξˆ, hence a good estimate of this
quantity is desirable. Our main interest is to estimate the risk under the normal
model. The following definition introduces two notions of degrees of freedom that
are useful when we want to estimate the risk. In the definition, ψ(y;ξ,σ2) denotes
thedensityforthe (ξ,σ2I)distributionand , denotesthestandardinnerprod-
N h· ·i
uct on Rn. The divergence operator is also needed. It is the differential operator
defined as
n
div(f)= ∂ f
i i
i=1
X
for f : Rn Rn Lebesgue almost everywhere differentiable and with ∂i denoting
→
the partial derivative w.r.t. the ith coordinate.
Definition 2.1. For a measurable map ξˆ : Rn Rn such that ξˆ(Y) has finite
→
second moment the degrees of freedom of ξˆis
n cov(Y ,ξˆ(Y) ) y ξ,ξˆ(y)
(1) df(ξˆ):= i i = h − iψ(y;ξ,σ2)dy.
σ2 σ2
i=1 Z
X
If ξˆis differentiable in Lebesgue almost all points and div(ξˆ) has finite first moment
Stein’s degrees of freedom of ξˆis
(2) df (ξˆ):=E(div(ξˆ(Y))).
S
4 F.R.MIKKELSENANDN.R.HANSEN
A simple expansion of the risk yields
(3) Risk=E Y ξˆ(Y) 2 nσ2+2σ2df(ξˆ).
k − k2−
Hence Y ξˆ(Y) 2 nσ2+2σ2df is an unbiased risk estimate if df is an unbiased
k − k2−
estimate of df(ξˆ). In practice, σ2 must be estimated as well and a bias of df can
also be preferable if it reduces tbhe variance. Hence exact unbiasbedness of a risk
estimate based on (3) is of secondary interest, but it is of interest to find adebquate
correctionsofthesquarederror Y ξˆ(Y) 2 thatcanbeusedformodelassessment
k − k2
and comparison.
If ξˆis almost differentiable then df(ξˆ) = df (ξˆ) due to Stein’s lemma (Lemma
S
2 in Stein (1981)), in which case div(ξˆ)(Y) is an unbiased estimate of df(ξˆ). How-
ever,mostestimatorswithdiscontinuitiesarenotalmostdifferentiableandforsuch
estimators it is not clear if div(ξˆ)(Y) is a useful estimate of the degrees of freedom.
Indeed, our main result, Theorem 2.4, provides a representation of df(ξˆ) df (ξˆ),
S
−
whichisnonzeroforarangeofestimators. Theresultprovidesthetheoreticalbasis
for establishing more adequate estimates of the degrees of freedom and thus the
risk. Furthermore, Theorem 3.2 provides a quite remarkable connection between
df(ξˆ) and df (ξˆ) for the debiased lasso estimator, which can be used to derive an
S
estimate of df(ξˆ). This result is directly applicable in practice and provides fast
and accurate risk estimation without the need for cross validation, say.
Our main result is derived under the assumptions on the estimator as stated
below. To fix notation we let B(x,r) denote the closed ball of radius r and center
x. Additionally, we let n 1 denote the n 1 dimensional Hausdorff measure – a
−
H −
generalisation of the surface measure of n 1 dimensional hypersurfaces in Rn (see
−
e.g. Evans & Gariepy (1992) for details).
Assumption 2.2. The estimator ξˆ can be written as ξˆ = N 1 ξˆ for a col-
i=1 Ui i
lection of open and disjoint sets {Ui}Ni=1 with Ni=1Ui = RPn and int(Ui) = Ui.
Additionally, for each i=1,...,n:
S
(a) The map ξˆi :Ui Rn is locally Lipschitz.
→
(b) The random variable 1 div(ξˆ)(Y) has finite first moment and ξˆ is polyno-
Ui i k ik
mially bounded on U .
i
(c) The function r n 1(∂U B(0,r)) is polynomially bounded.
− i
7→H ∩
Remark 2.3. The following points are worth noting:
a) Boundary values of the estimator. Assumption 2.2(c) implies that the
boundaries of the sets Ui are Lebesgue null sets, and thus that Rn\ iUi has
Lebesgue measure zero. The estimator ξˆ is here defined to be zero on this
S
null set, but with Y having an absolutely continuous distribution its value on a
null set is irrelevant. Note, however, that Assumption 2.2(a) ensures that ξˆ is
i
uniquelydefinedon∂U . Inaconcretecasetheremaybeanaturalwaytodefine
i
ξˆon the common boundary between U and U , say, but we make no abstract
i j
attempt to select between ξ and ξ on the boundary.
i j
b) Degrees of freedom. Assumption 2.2(a) implies by Rademacher’s theorem
(Theorem3.1.6and3.1.7inFederer(1969))thatdiv(ξˆ)isdefinedLebesguea.e..
i
CombiningthiswithAssumption2.2(b)weconcludethatunderAssumption2.2
both df(ξˆ) and df (ξˆ) are well defined.
S
DEGREES OF FREEDOM FOR PIECEWISE LIPSCHITZ ESTIMATORS 5
c) Existence of normal vectors. Assumption 2.2(c) implies that the sets U
i
have locally finite perimeter (see Theorem 5.11.1 in Evans & Gariepy (1992)),
thus a measure theoretic outer unit normal η is defined on a subset of ∂U
i i
(Lemma 5.8.1 in Evans & Gariepy (1992)). By convention, η is set equal to
i
zerowhenitisnotdefined. Whenever∂U issmooththemeasuretheoreticunit
i
normal coincides with the usual pointwise unit normal.
d) Regular open assumption. The sets U are assumed regular open, that is,
i
int(U ) = U , which among other things implies that ∂U U . For
i i i ⊆ j:j=i j
the details see the proof of Lemma A.2. It is a convenient though n6 ot strictly
S
necessary assumption. Given Assumptions 2.2(a) and 2.2(c) the estimator is
only changed on a Lebesgue null set if U is replaced by int(U ) on which ξˆ is
i i i
well defined by Assumption 2.2(a). This can only make the boundaries smaller
and Assumption 2.2(c) still holds after the replacement.
Estimators that involve data driven variable selection will generally fulfill As-
sumption 2.2 with each U corresponding to a set of selected variables. Example
i
2.5 and Appendix A.1 provide a thorough characterization of U in the debiased
i
lasso setup. Moreover, a similar characterization of U is given in Example 3.4 for
i
a class of estimators defined via minimization of a penalized loss function.
The conditions in Assumption 2.2 are typically easy to verify, except perhaps
the third condition, as it involves bounding Hausdorff measures. Appendix A.2
provides some results that can be helpful for verifying the third condition. For
estimators satisfying Assumption 2.2 we have the following representation of the
degrees of freedom.
Theorem 2.4. If ξˆsatisfies Assumption 2.2 then
1
(4) df(ξˆ)=df (ξˆ)+ ξˆ ξˆ,η ψ( ;ξ,σ2) d n 1,
S j i i −
2 h − i · H
Xi6=jZUi∩Uj
where η denotes the measure theoretic outer unit normal to ∂U .
i i
The proof is in Appendix A.3. The essential part is an application of a gener-
alized version of Gauss-Green’s formula combined with a dominated convergence
argument. Note that though U U is a Lebesgue null set – on which ξˆis defined
i j
∩
to be zero – ξˆ and ξˆ are uniquely defined by Assumption 2.2(a) and generally
j i
non-zero and different, cf. also Remark 2.3(a).
IfξˆsatisfiesAssumption2.2andiscontinuousthen(4)reducestodf(ξˆ)=df (ξˆ),
S
which is Stein’s lemma for a class of locally Lipschitz continuous estimators. The
boundary integrals therefore account for potential jumps of ξˆacross the boundary
of any two adjacent regions U and U .
i j
The following example illustrates how to verify Assumption 2.2 for the debiased
lasso estimator, which is the estimator that will also be the main focus of the
subsequent section.
Example2.5(Thedebiasedlassoestimator). Considerlinearregressionestimators
of the form ξˆ=Xβˆ, where X is an n p-matrix and βˆ:Rn Rp. We will assume
× →
throughout that X has rank p and thus p n. For any subset A 1,...,p , X
A
≤ ⊆ { }
denotes the matrix whose columns are those of X indexed by A, and similarly, β
A
denotes (β ) . We denote by
i i A
∈
Π =X (XTX ) 1XT
A A A A − A
6 F.R.MIKKELSENANDN.R.HANSEN
theorthogonalprojectionontocol(X )–thecolumnspaceofX . Welet denote
A A
A
the power set of 1,...,p and supp(β) = i β = 0 denote the support of
i
{ } { | 6 } ∈ A
β Rp.
∈
The lasso estimator of β with tuning parameter λ>0 is defined as
1
βˆλ(y)=argmin y Xβ 2+λ β .
2k − k2 k k1
β
It follows from the first order subgradient conditions for this convex minimization
problem that βˆλ(y) is the lasso estimate if and only if
X ,y Xβˆλ(y) =λsign(βˆλ(y)) for βˆλ(y)=0
h i − i i 6
X ,y Xβˆλ(y) λ for βˆλ(y)=0,
|h i − i|≤ i
see e.g. Section 3 in Tibshirani & Taylor (2012) or Lemma 2.1 in Bu¨hlmann
& van de Geer (2011). The lasso estimator of ξ is ξˆλ = Xβˆλ, and with
lasso
Sˆλ(y)=supp(βˆλ(y))denotingthesetofactivepredictorsitholdsthatdf (ξˆλ )=
S lasso
df(ξˆλ )=E Sˆλ(Y), see Zou et al. (2007). Note that since the columns of X are
lasso | |
assumed linearly independent the lasso estimate βˆλ(y) is unique, hence Sˆλ(y) is
well defined. The map y βˆλ(y) is well defined as a single valued map, and from
7→
Theorem 1 in Zou et al. (2007) it is continuous.
The debiased lasso estimator is the estimator ξˆλ = Π , which is the or-
d.lasso Sˆλ
thogonal projection onto the (data driven) subspace determined by the active set
Sˆλ for the lasso estimator. Introducing the sets
UAλ :=(y ∈Rn (cid:12) |βˆiλ(y)|>0 for i∈A and |hXi,y−Xβˆλ(y)i|<λ for i∈/ A)
(cid:12)
(cid:12)
forA itfollows(cid:12)fromthefirstorderconditionsthatify Uλthensupp(βˆλ(y))=
∈A (cid:12) ∈ A
A. It also follows that the sets Uλ for A are disjoint and from the definition
A ∈ A
thattheyareopen. Fromthefirstordersubgradientconditionsweget,inaddition,
that Rp = Uλ. Moreover, if y ∂Uλ then either βˆλ(y) = 0 for some i A or
A A ∈ A i ∈
X ,y Xβˆλ(y) =λ for some i / A. It is then possible to find A =A such that
|h i − S i| ∈ 0 6
y ∂Uλ as well. This shows that int(Uλ) = Uλ. Observe that ξˆλ = Π on
Uλ∈, henAc0e A A d.lasso A
A
(5) ξˆλ = 1 Π
d.lasso Uλ A
A
A
X∈A
on Uλ, and we will henceforth take (5) as the definition of ξˆλ on Rn.
A A d.lasso
Since Π is Lipschitz continuous – and in particular polynomially bounded – with
A
div(SΠ ) = A we conclude that ξˆλ is an estimator of the form required by
A | | d.lasso
Assumption 2.2, which fulfills Assumptions 2.2(a) and 2.2(b). Assumption 2.2(c)
remains to be demonstrated.
Since βλ(y) = 0 for y Uλ and i / A it follows by continuity that ξˆλ (y)
i ∈ A ∈ lasso ∈
col(X ) for y Uλ. Hence, for any pair A ,A , it holds that ξˆλ (y)
A ∈ A 1 2 ∈ A lasso ∈
λ λ
col(X ) col(X )=col(X )whenevery U U . Moreover,bydefining
A1 ∩ A2 A1∩A2 ∈ A1∩ A2
the affine subspaces
LA(s):= u Rn Xi,u =siλ for i A
{ ∈ |h i ∈ }
DEGREES OF FREEDOM FOR PIECEWISE LIPSCHITZ ESTIMATORS 7
Figure 1. Illustrations of the decomposition of R2 into the four
sets U1, U1 , U1 and U1 according to the lasso estimator
1 2 1,2
with λ∅= 1{.} Th{e}set U1 c{ons}ists of the points shrunk to zero,
the sets U1 and U1 t∅o the points where either the second or
1 2
the first co{o}rdinate,{re}spectively, is shrunk to zero and U1 to
1,2
the set where none of the coordinates are shrunk to zero{. }The
decomposition depends on the angle between the two columns in
X.
for s 1,1 A, it follows from the first order conditions that
| |
∈{− }
y ξˆλ (y) L (s)
− lasso ∈ A1∪A2
s
[
λ λ
for y U U . Thus we have that
∈ A1 ∩ A2
λ λ
(6) U U col(X )+ L (s),
A1 ∩ A2 ⊆ A1∩A2 A1∪A2
s
[
which by Remark 2.3(d) implies that
(7) ∂Uλ (col(X )+L (s)).
A1 ⊆ A1∩A2 A1∪A2
A2:A[26=A1[s
8 F.R.MIKKELSENANDN.R.HANSEN
The latter is a finite union of affine subspaces of dimensions
A A +n A A =n+2A A A A n 1.
1 2 1 2 1 2 1 2
| ∩ | −| ∪ | | ∩ |−| |−| |≤ −
Hence Assumption 2.2(c) holds. Figure 2 provides an illustration of the partition
of Rn for n=p=2 for different choices of angles between the columns in X. (cid:3)
TheargumentsinExample2.5aboveareelementaryandbasedonthefirstorder
conditionsforthelassoestimator. Itispossibletoexpresstheargumentsinamore
abstract but powerful way. We briefly outline these arguments in Appendix A.1.
3. Risk estimation for debiased lasso
It is not obvious how the general formula in Theorem 2.4 for df(ξˆ) can be used
for computing or estimating the degrees of freedom. The first term, df (ξˆ), may
S
be estimated by div(ξˆ)(Y), but the second term is more difficult. In this section
we show how this second term can be related to the derivative of λ df (ξˆλ )
7→ S d.lasso
for debiased lasso. First we recapitulate the computations in Tibshirani (2015) of
the degrees of freedom for debiased lasso with X orthogonal, which will reveal the
general formula shown below.
Example 3.1 (Continuation of Example 2.5). Assume that n=p and X =I. In
this case it is well known that the lasso and the debiased lasso estimators become
the soft and hard threshold estimators, respectively. That is,
Y λsign(Y ) if Y >λ Y if Y >λ
ξˆλ = i− i | i| and ξˆλ = i | i| .
lasso,i (0 otherwise d.lasso,i (0 otherwise
We can write up closed form expressions for df(ξˆλ ) and df (ξˆλ ):
d.lasso S d.lasso
df (ξˆλ )= ψ(y;ξ,σ2) 1 dy = ψ(y ;ξ ,σ2) dy
S d.lasso Z Xi (|yi|>λ) Xi Z(|yi|>λ) i i i
λ ξ λ ξ
i i
= Φ − − + 1 Φ − ,
σ − σ
i (cid:18) (cid:19) (cid:18) (cid:18) (cid:19)(cid:19)
X
and as in Tibshirani (2015)
df(ξˆλ )= ∞ψ(y ;ξ ,σ2)yi(yi−ξi) dy + −λψ(y ;ξ ,σ2)yi(yi−ξi) dy
d.lasso i i σ2 i i i σ2 i
Xi Zλ Z−∞
= −ψ(yi;ξi,σ2)yi ∞λ + ∞ψ(yi;ξi,σ2) dyi
i Zλ
X(cid:2) (cid:3)
λ
+ ψ(yi;ξi,σ2)yi −λ + − ψ(yi;ξi,σ2) dyi
−
=λ (cid:2)ψ(λ;ξ ,σ2)+ψ((cid:3)−λ∞;ξ ,Zσ−2)∞+df (ξˆλ ).
i − i S d.lasso
i
X(cid:0) (cid:1)
Letting ∂ denote the differential operator with respect to λ we observe that
λ
(8) df(ξˆλ )=df (ξˆλ ) λ∂ df (ξˆλ ),
d.lasso S d.lasso − λ S d.lasso
which is a striking identity. This is because the formula for df(ξˆλ ), though
d.lasso
explicit, involves the unknown parameter ξ and is not readily estimable. But we
have the divergence estimator, 1 , of df (ξˆλ ), and if we from this
i (|yi|>λ) S d.lasso
P
DEGREES OF FREEDOM FOR PIECEWISE LIPSCHITZ ESTIMATORS 9
can estimate its derivative as well, the formula above suggests how to estimate
df(ξˆdλ.lasso). (cid:3)
Theremarkablefactthatwewillshowisthat(8)holdswithouttheorthogonality
assumption on X.
Theorem 3.2. For the debiased lasso estimator defined in Example 2.5 it holds
that
(9) df(ξˆλ )=df (ξˆλ ) λ∂ df (ξˆλ )
d.lasso S d.lasso − λ S d.lasso
where ∂ denotes differentiation w.r.t. λ.
λ
Theorem 3.2 suggests that df(ξˆλ ) can be estimated by differentiation of an
d.lasso
estimate of df (ξˆλ ). The divergence estimate of Stein’s degrees of freedom is,
S d.lasso
however, not differentiable as a function of λ, and we need to somehow smooth it.
Tothisenditisconvenienttoreparametrizethepenalizationintermsofδ =log(λ),
so that with
h(δ):=df (ξˆexp(δ)),
S d.lasso
then
df(ξˆexp(δ))=h(δ) h(δ).
d.lasso − 0
Insimulationshwasfoundtobemonotonelydecreasing,andthush tobenegative,
0
but we cannot prove that this is generally the case. The integral representation of
h from Theorem 2.4 is not particularly helpful as the integrand can, in fact, be
0
negative. Basedonourcomputationalobservations–and toreducevariance ofthe
resulting estimate – our proposal is based on the assumption that h is negative. It
0
iseffectivelyakernelsmootherthatestimatestheintensityofjumpsforamonotone
jump process.
We note that Sˆexp(δ) is an unbiased estimate of h(δ) and that the function
| |
δ Sˆexp(δ) is a step function. The problem of estimating the derivative, h, of its
0
7→| |
meanisthusanalogoustoestimatingtheintensityforajumpprocesswithonemain
difference; the step function can have jumps of negative as well as positive sign,
though most jumps will be negative. Our proposed estimate ignores the positive
excursions of the step function and is computed as follows:
• Compute the jump points, λi and jump sizes, ∆i := infλ<λi|Sˆλ|−|Sˆλi+|,
of the decreasing function λ inf Sˆλ0 for i=1,...,M.
7→ λ0<λ| |
Applyakerneldensitysmoothertothepointsδ =log(λ )fori=1,...,M
i i
•
counted with the multiplicities ∆ . In the simulations presented in this
i
paper an adaptive Gaussian kernel density smoother was used (see Section
10.4.3.2 in Givens & Hoeting (2012)).
Rescale the density estimate by the total number of jumps, that is, by
• M ∆ .
i=1 i
As mentioned above, we can think of the proposed estimate of h as a non-
P 0
parametricestimateoftheintensityofthejumpsforamonotonelydecreasingjump
process. Alternatively, we can think of it as smoothing the jumps by a sigmoidal
function (the anti-derivative of the kernel) to obtain a smooth estimate of Stein’s
degrees of freedom, which can then be differentiated. Note that even if ∆ may
i
alwaysbe1intheory,thejumpsareinpracticecomputedonagridandmaythusbe
larger than 1, which the procedure accounts for. The estimate of λ∂ df (ξˆλ )
− λ S d.lasso
resulting from the procedure above is denoted by ∂.
b
10 F.R.MIKKELSENANDN.R.HANSEN
dfS df=dfS+¶ ¶ dfS E(df) df
100 100
75 75
50 50
25 25
0 0
−10.0 −7.5 −5.0 −2.5 −10.0 −7.5 −5.0 −2.5
log(l ) log(l )
( ) ( )
Risk Risk E Risk E Risk Risk
dfS df dfS df
30 30
20 20
10 10
0 0
−10.0 −7.5 −5.0 −2.5 −10.0 −7.5 −5.0 −2.5
log(l ) log(l )
Figure2. Left: Realizationoftheestimatesofdegreesoffreedom
dˆf = Sˆλ and dˆf = Sˆλ +∂ as well as the correction term ∂
S
| | | |
as a function of log(λ) (top) and corresponding estimates of the
risk (bottom). Right: Similarbto the left but mean values of thbe
estimates obtained by averaging over 1000 samples along with the
degreesoffreedomdf =df(ξˆλ )obtainedfromthe1000samples
d.lasso
using the covariance definition (1). The design parameters were:
σ =0.5, n=p=100, γ =1, α=0.1 and the design type was (S)
with constant correlation of ρ=0.1 (see Section 4).
Using Sˆλ +∂ as an estimate of degrees of freedom leads to the risk estimate
| |
(10) bRiskdf :=kY −ξˆdλ.lassok22−nσ2+2σ2 |Sˆλ|+∂ .
(cid:16) (cid:17)
For an example odf the above estimate see Figure 2, where ∂ anbd Risk are applied
df
to a single realization of Y along with an average over 1000 replications.
ToproveTheorem3.2weproveamoregeneralintermedibateresdultforestimators
that are parametrized in a similar way by a tuning parameter. We use in the
following D to denote the differential operator w.r.t. y.
Proposition 3.3. Let q >0 and suppose that ξˆλ = 1 ξˆ where
i Uλ i
i
(11) Uλ =λqU1, for all i=1,P...,N.
i i