Model selection for density estimation with L2-loss

0.36 MB·English

by  Lucien Birgé

Model selection for density
L
estimation with -loss
2
8
0
0 Lucien Birg´e
2

v Universit´e Paris VI
o
Laboratoire de Probabilit´es et Mod`eles Al´eatoires
N
U.M.R. C.N.R.S. 7599
9
1

]
T 20/10/2008
S
.
h
t
a
m Abstract
[ We consider here estimation of an unknown probability density s belonging

2 toL2(µ)whereµisaprobabilitymeasure. We haveathandn i.i.d.observations
v withdensitysandusethe squaredL2-normasourlossfunction. Thepurposeof
6 thispaperistoprovideanabstractbutcompletelygeneralmethodforestimating
1 s by model selection, allowing to handle arbitrary families of finite-dimensional
4 (possiblynon-linear)modelsandanys L2(µ). Weshall,inparticular,consider
.1 thecasesofunboundeddensitiesandbo∈undeddensitieswithunknownL∞-norm
8 and investigate how the L∞-norm of s may influence the risk. We shall also
0
provide applications to adaptive estimation and aggregation of preliminary es-
8
timators. Although of a purely theoretical nature, our method leads to results
0
: that cannot presently be reached by more concrete ones.
v
i
X
1 Introduction
r
a
1.1 Histograms and partition selection
Suppose we have at hand n i.i.d. observations X ,...,X with values in the mea-
1 n
surable space ( , ) and they have an unknown density s with respect to some
X W
probability measure µ on . The simplest method for finding an estimator of s is to
X
buildan histogram. Given afinite partition = I ,...,I of with µ(I ) = l >0
1 k j j
I { } X
for 1 j k, the histogram sˆ based on this partition is defined by
I
≤ ≤
k n
1
sˆ (X ,...,X ) = N 1l , with N = 1l (X ). (1.1)
I 1 n nl j Ij j Ij i
j
j=1 i=1
X X
0AMS 1991 subject classifications. Primary 62G07; secondary 62G10.
Key words and phrases. Density estimation, L2-loss, model selection, estimator selection, his-
tograms.
1
Let
k k
p
p = sdµ, s = j1l and S = β 1l β R for 1 j k .
j I l Ij I  j Ij (cid:12) j ∈ ≤ ≤ 
ZIj Xj=1 j Xj=1 (cid:12)(cid:12) 
(cid:12)
If s L2(µ), then sI is the orthogonal projection of s ont(cid:12)(cid:12)o the k-dimensional linear
∈
space S spanned by the functions 1l . Choosing the squared L -distance induced
I Ij 2
by the norm of L (µ) as our loss function leads to the following quadratic risk
2
k·k
for the estimator sˆ :
I
k
1 p (1 p )
E sˆ s 2 = s s 2+ j − j . (1.2)
I I
k − k k − k n l
j
j=1
(cid:2) (cid:3) X
Hence, if s L (µ), with norm s , the quadratic risk of sˆ can be bounded by
∞ ∞ I
∈ k k
(k 1) s
E sˆ s 2 s s 2+ − k k∞, (1.3)
I I
k − k ≤ k − k n
h i
and, as we shall see below, this bound is essentially unimprovable without additional
assumptions.
The histogram estimator sˆ is probably the simplest example of a model-based
I
estimator with model S , i.e. an estimator of s with values in S . It may acually be
I I
viewed as the empirical counterpart of the projection s of s onto S .
I I
Suppose now that we are given a finite (although possibly very large) family
,m of finite partitions of with respective cardinalities , hence the
m m
{I ∈ M} X |I |
correspondingfamilies of models S ,m and histogram estimators sˆ ,m
{ Im ∈ M} { Im ∈
. It is natural to try to find one estimator in the family which leads, at least ap-
pMro}ximately, to the minimal risk inf E sˆ s 2 . But one cannot select such
m∈M k Im − k
an estimator from (1.2) or (1.3) since the risk depends on the unknown density s via
(cid:2) (cid:3)
s . Methods of model or estimator selection base the choice of a suitable partition
Im
with mˆ = mˆ(X ,...,X ) on the observations. When s L (µ) one would like
mˆ 1 n ∞
I ∈
to know whether it is possible to design a selection procedure mˆ(X ,...,X ) leading
1 n
(at least approximately), in view of (1.3), to a risk bound of the form
E sˆ s 2 C inf s s 2+n−1 s ,
k Imˆ − k ≤ m∈M k Im − k k k∞|Im|
h i
(cid:8) (cid:9)
for some universal constant C, even when s is unknown.
∞
k k
1.2 What is presently known
There exists a considerable amount of litterature dealing with problems of model or
estimator selection. Most of it is actually devoted to the analysis of Gaussian prob-
lems, or regression problems, or density estimation with either Hellinger or Kullback
loss and it is not our aim here to review this litterature. Only a few papers are
actually devoted to our subject, namely model or estimator selection for estimating
densities with L -loss, and we shall therefore concentrate on these only. Thesepapers
2
can roughly be divided into three groups: the ones dealing with penalized projection
estimators, the ones that study aggregation by selection of preliminary estimators
2
and the ones which use methods based on the thresholding of empirical coefficients
within a given basis. The last ones are typically not advertised as dealing with model
selection but, as explained for instance in Section 5.1.2 of Birg´e and Massart (2001),
they can be viewed as special instances of model selection methods for models that
are spanned by some finite subsets of an orthonormal basis. All these papers have in
common the fact that they require more or less severe restrictions on the families of
models and, apart from some special cases, typically assume that s L (µ) with a
∞
∈
known or estimated bound for s .
∞
k k
In order to see how such methods apply to our problem of partition selection,
let us be more specific and assume that = [0,1], µ is the Lebesgue measure and
X
= j/(N + 1),1 j N for some (possibly very large) positive integer N.
N { ≤ ≤ }
For any subset m of , we denote by the partition of generated by the in-
m
N I X
tervals with set of endpoints m 0,1 and we set S = S and sˆ = sˆ . This
∪{ } m Im m Im
leads to a set with cardinality 2N and the corresponding families of linear models
M
S ,m and related histogram estimators sˆ ,m . Then all models S
m m m
∈ M { ∈ M}
are linear subspaces of the largest one S . Of particular interest is the dyadic case
N
(cid:8) (cid:9)
with N = 2K 1 for which S is the linear span of the 2K first coefficients of the
N
−
Haar basis. There is, nevertheless, a difference between expansions in the Haar basis
and projections on our family of models. Let us, for instance, consider the function
1l[0,2−K). It belongs to the two-dimensional model S{1} but its expansion in the Haar
basis has K non-zero coefficients.
Given a sample X ,...,X with unknown density s, which partition should we
1 n m
I
choose to estimate s and what bound could we derive for the resulting estimator?
Penalized projection estimators have been considered by Birg´e and Massart (1997)
and an improved version is to be found in Chapter 7 of Massart (2007). The method
either deals with polynomial collections of models (which does not apply to our case)
orwithsubsetselection withinagiven basiswhichapplies hereonlywhenN = 2K 1
−
and we use the Haar basis. Moreover, it requires that N < n/logn and a bound on
s beknownor estimated, as inSection 4.4.4 of Birg´e andMassart (1997), since
k INk∞
the penalty depends on it.
Methodsbasedonwaveletthresholding,asdescribedinDonoho,Johnstone,Kerky-
acharian and Picard (1996) or Kerkyacharian and Picard (2000) (see also the numer-
ous references therein) require the same type of restrictions and, in particular, a
control on s in order to properly calibrate the threshold. Also, as mentioned
∞
k k
above, restricting to subsets of the Haar basis may result in expansions that use
many more coefficients (K instead of 2, for instance) than needed with the partition
selection approach.
Aggregation of estimators by selection assumes that preliminary estimators (one
for each model in our case) are given in advance (we should here use the histograms)
and typically leads to a risk bound including a term of the form n−1 s log =
∞
k k |M|
n−1N s log2sothatallsuchresultsareuselessforN n. Moreover, mostofthem
∞
k k ≥
also require that an upper bound for s be known since it enters the construction
∞
k k
of the aggregate estimator. This is the case in Rigollet (2006) (see for instance his
Corollary 2.7) and Juditsky, Rigollet and Tsybakov (2007, Corollary 5.7) since the
parameter β that governs their mirror averaging method depends crucially on an
upper bound for s . As to Samarov and Tsybakov (2005), their Assumption 1
∞
k k
requires that N be not larger than Clogn. Similar restrictions are to be found in
3
Yang (2000) in his developments for mixing strategies and in Rigollet and Tsybakov
(2007) for linear aggregation of estimators. Lounici (2008) does not assume that
s L but, instead, that all preliminary estimators are uniformly bounded. Onecan
∞
∈
always truncate the estimators to get this, but to be efficient, the truncation should
be adapted to the unknown parameter s, and therefore chosen from the data in a
suitable way. We do not know of any paper that allows such a data driven choice.
Consequently, none of these results can solve our partition selection problem in a
satisfactory way when N is at least of size n and s is unknown. This fact was
∞
k k
one motivation for our study of model selection for density estimation with L -loss.
2
Results about partition selection will be a consequence of our general treatment of
model selection. This treatment allows to consider arbitrary countable families of
finite-dimensional models (possibly nonlinear) and does not put any assumption on
the density s, apart from the fact that it belongs to L (µ); it may, in particular, be
2
unbounded. We do not know of any result that applies to such a situation. There is
a counterpart to this level of generality: our procedure is of a purely abstract nature
and not constructive, only indicating what is theoretically feasible. Unfortunately,
we are unable to design a practical procedure with similar properties.
2 Model based estimation and model selection
To begin with, let us fix our framework and notations. We want to estimate an
unknown density s, with respect to some probability measure µ on the measurable
space ( , ), from an i.i.d. sample X = (X ,...,X ) of random variables X
1 n i
X W ∈ X
with distribution P = s µ. Throughout the paper we denote by P the probability
s s
·
thatgives X thedistributionP⊗n, byE thecorrespondingexpectation operatorand
s s
by the norm in L (µ), omitting the subscript when q = 2 for simplicity. We
q q
k·k
denote by d the distance in L (µ): d (t,u) = t u . For 1 q + and Γ > 1,
2 2 2
k − k ≤ ≤ ∞
we set
L = t L (µ) t 0 and tdµ = 1 ; LΓ = t L t Γ . (2.1)
q ∈ q ≥ ∞ ∈ ∞ k k∞ ≤
(cid:26) (cid:12) Z (cid:27)
(cid:12) (cid:8) (cid:12) (cid:9)
Wemeasuretheper(cid:12)(cid:12)formanceats L2ofanestimatorsˆ(X) L2(cid:12)byitsquadraticrisk
∈ ∈
E d2(sˆ(X),s) . More generally, if (M,d) is a metric space of measurable functions
s 2
on suchthatM L = , thequadraticriskofsomeestimator sˆ M ats M L
1 1
(cid:2)X (cid:3) ∩ 6 ∅ ∈ ∈ ∩
is defined as E d2(sˆ(X),s) . We denote by the cardinality of the set and set
s
|I| I
a banda bforthemaximumandtheminimumofaandb,respectively. Throughout
∨ ∧ (cid:2) (cid:3)
the paper C (or C′, ...) will denote a universal (numerical) constant and C(a,b,...)
or C a fonction of the parameters a,b,... or q. Both may vary from line to line.
q
Finally, from now on, countable will always mean “finite or countable”.
2.1 Model based estimation
A common method for estimating s consists in choosing a particular subset S of
(M,d) that we shall call a model for s and design an estimator with values in S. Of
this type are the maximum likelihood estimator over S or the projection estimator
onto S. Let us set M = L and choose for d either the Hellinger distance h or the
1
4
variation distance v given respectively by
1 2 1
h2(t,u) = √t √u dµ and v(t,u) = t u dµ.
2 − 2 | − |
Z (cid:16) (cid:17) Z
It follows from Le Cam (1973, 1975, 1986) and subsequent results by Birg´e (1983,
2006a) that the risk of suitably designed estimators with values in S is the sum of
two terms, an approximation term depending on the distance from s to S and an
estimation term depending on the dimension of the model S which can be defined as
follows.
Definition 1 Let S be a subset of some metric space (M,d) and let (t,r) denote
d
B
the open ball of center t and radius r with respect to the metric d. Given η > 0, a
subset S of M is called an η-net for S if, for each t S, one can find t′ S with
η η
∈ ∈
d(t,t′) η.
≤
We say that S has a metric dimension bounded by D 0 if, for every η > 0, there
≥
exists an η-net S for S such that
η
S (t,xη) exp Dx2 for all x 2 and t M. (2.2)
η d
| ∩B | ≤ ≥ ∈
(cid:2) (cid:3)
Remark: One can always assume that S S at the price of replacing D by 25D/4
η
⊂
according to Proposition 7 of Birg´e (2006a).
Typical examples of sets with metric dimension bounded by D when (M,d) is a
normed linear space are subsets of 2D-dimensional linear subspaces of as shown
M
in Birg´e (2006a). If d is either h or v and S L has a metric dimension bounded
1
⊂
by D 1/2, there exists a universal constant C and an estimator sˆ(X) with values
≥
in S such that, for any s L ,
1
∈
E d2(sˆ(X),s) C inf d2(s,t)+n−1D . (2.3)
s
≤
(cid:20)t∈S (cid:21)
(cid:2) (cid:3)
In particular, sup E d2(sˆ(X),s) Cn−1D. This results from the following
s∈S s ≤
theoremaboutmodelselection ofBirg´e (2006a) bysetting = 0 , S = S,D = D
(cid:2) (cid:3) M { } 0 0
and ∆ = 1/2.
0
Theorem 1 Let X ,...,X be an i.i.d. sample with unknown density s belonging
1 n
to L and S ,m a finite or countable family of subsets of L with metric
1 m 1
∈ M
dimensions bounded by D 1/2 respectively. Let the nonegative weights ∆ ,m
m m
(cid:8) (cid:9) ≥ ∈
satisfy
M
exp[ ∆ ] = Σ <+ . (2.4)
m
− ∞
m∈M
X
Then there exists a universal constant C and an estimator s˜(X ,...,X ) such that,
1 n
for any s L ,
1
∈
E d2(s˜,s) C(1+Σ) inf inf d2(s,t)+n−1(D ∆ ) . (2.5)
s m m
≤ m∈M(cid:20)t∈Sm ∨ (cid:21)
(cid:2) (cid:3)
5
Unfortunately, as we shall see below, (2.3) does not hold in general when (M,d) =
(L (µ),d ). In particular, whatever the estimator sˆ, sup E d2(sˆ(X),s) may
2 2 s∈S s
be infinite even if S L (µ) has a bounded metric dimension. This difference is
⊂ 2 (cid:2) (cid:3)
due to the following fact: h and v are actually distances defined on the set of all
probabilities on ( , ) and h(s,t) = h(P ,P ) is independent of the choice of the
s t
X W
underlyingdominating measure, the same property holding for the variation distance
v. This is not the case for d which is a distance in L (µ) depending on the choice of
2 2
µ and definitely not a distance between probabilities. Even the fact that s = dP /dµ
s
belong or not to L (µ) depends on µ. Further remarks on this subject can be found
2
in Devroye and Gy¨orfi (1985) and Devroye (1987).
Nevertheless, the L -distance has been much more popular in the past than either
2
the Hellinger or variation distances, mainly because of its simplicity due to the clas-
sical “squared bias plus variance” decomposition of the risk. Although hundreds of
papers have been devoted to the derivation of risk bounds for various specific esti-
mators, we do not know of any general bound for the risk similar to (2.3) based on
purely metric considerations for the distance d .
2
2.2 Projection and histogram estimators
To illustrate the specificity of the L -risk, let us turn to a quite classical family of
2
model-based estimators for densities, the projection estimators of Cencov (1962).
To estimate a density s L from an i.i.d. sample X ,...,X , we chose some k-
2 1 n
∈
dimensionallinearsubspaceS ofL (µ)togetherwithanorthonormalbasis(ϕ ,...,ϕ )
2 1 k
so that the projection s of s onto S can be written s = k β ϕ . Then we
j=1 j j
estimate each coefficient β = ϕ sdµ in this expansion by its empirical version
j j P
βˆ = n−1 n ϕ (X ). This results in the projection estimator sˆ = k βˆ ϕ
j i=1 j i R j=1 j j
(which in general does not belong to L ) with risk
P 1 P
k
E sˆ s 2 = s s 2+n−1 Var ϕ (X )
s s j 1
k − k k − k
j=1
(cid:2) (cid:3) X (cid:0) (cid:1)
k
s s 2+n−1 ϕ2(x) s(x)dµ(x)
≤ k − k  j 
Z j=1
X
 
k
s s 2+n−1min ϕ2 ;k s . (2.6)
≤ k − k (cid:13) j(cid:13) k k∞
(cid:13)(cid:13)Xj=1 (cid:13)(cid:13)∞ 
(cid:13) (cid:13)
A particular case occurs with the histogram sˆ give(cid:13)n by (1(cid:13).1) which corresponds to
I (cid:13) (cid:13)
choosing ϕ = l−1/21l , S = S and s = s . If l = k−1 for all j, we get a regular
j j Ij I I j
histogram and derive from (1.2) and a convexity argument that
E s sˆ 2 s s 2+(k 1)/n.
s I I
k − k ≤ k − k −
h i
But, for general partitions, thebound(1.3) clearly emphasizes thedifference with the
risk boundof the form (2.3) obtained in Birg´e and Rozenholc (2006) for the Hellinger
loss:
E h2(s,sˆ ) h2(s,s )+(k 1)/(2n).
s I I
≤ −
(cid:2) (cid:3)
6
Moreover, (1.3) is essentially unimprovable without further assumptions on s if the
partition is arbitrary, as shown by the following example. Define the partition
I
on = [0,1] by I = [(j 1)α,jα) for 1 j < k and I = [(k 1)α,1] with
j k
I X − ≤ −
0 < α < (k 1)−1. Set s = s = [(k 1)α]−1[1 1l ]. Then p = (k 1)−1 for
− I − − Ik j −
1 j < k, s = s and it follows from (1.2) that
I
≤
k 2 (k 2) s
E s sˆ 2 = − = − k k∞,
s I
k − k (k 1)αn n
h i −
which shows that there is little space for improvement in (1.3).
2.3 Some negative results
The fact that the L -norm of s comes into the risk is not due to the use of his-
∞
tograms or projection estimators as shown by another negative result provided by
Proposition 4 of Birg´e (2006b) that we recall below for the sake of completeness.
Proposition 1 For each L > 0 and each integer D with 1 D 3n, one can find
≤ ≤
a finite set S of densities with the following properties:
i)itisasubsetof some D-dimensional affine subspace ofL ([0,1],dx) witha metric
2
dimension bounded by D/2;
ii) sup s L+1;
s∈Sk k∞ ≤
iii)forany estimator sˆ(X ,...,X )belonging to L ([0,1],dx) and based on ani.i.d.
1 n 2
sample with density s S,
∈
supE sˆ s 2 > 0.0139DLn−1. (2.7)
s
k − k
s∈S
(cid:2) (cid:3)
It follows that there is no hope to get an analogue of (2.3), under the same assump-
tions, when d = d and the best one can expect in full generality, when S is a model
2
with metric dimension bounded by D and s L , is to design an estimator sˆwith a
∞
∈
risk bounded by
E d2(sˆ,s) C inf d2(s,t)+n−1D s . (2.8)
s 2 ≤ 2 k k∞
(cid:20)t∈S (cid:21)
(cid:2) (cid:3)
The situation becomes worse when s L (µ) or if sup s = + as shown by
6∈ ∞ s∈Sk k∞ ∞
the following lower bound to be proved in Section 7.1.
Proposition 2 Let S = s ,0 < θ 1/3 be the set of densities with respect to
θ
{ ≤ }
Lebesgue measure on [0,1] given by
s = θ−21l + θ2+θ+1 −11l .
θ [0,θ3] (θ3,1]
(cid:0) (cid:1)
If we have at disposal n i.i.d. observations with density s S, we can build an
θ
∈
estimator sˆ such that sup E nh2(s ,sˆ ) C for some C independent of
n 0<θ≤1/3 sθ θ n ≤
n. On the other hand, although the metric dimension of S with respect to the distance
(cid:2) (cid:3)
d is bounded by 2, sup E s s˜ 2 = + , whatever n and the estimator
2 0<θ≤1/3 sθ k θ − nk ∞
s˜ .
n
(cid:2) (cid:3)
7
2.4 About this paper
Ourpurposehereistwofold. Wefirstwanttoderiveestimatorsachievingariskbound
whichgeneralizes (2.8)in thesensethat it could also applyto thecase of s L . We
∞
6∈
knowfrom(2.6)thatprojection estimators dosatisfy(2.8)whenS isaD-dimensional
linear space, but do not have any result for non-linear models. Our second goal is to
handle many models simultaneously and design an estimator which performs as well
(or almost as well) as the estimator based on the “best model”, i.e. one leading to
the smallest risk bound (up to some universal constant). This is possible when the
distance d is either the Hellinger distance h or the variation distance v, as shown by
Theorem1. Ascomparedtothebound(2.3),wesimplypaythepriceofreplacingD
m
by ∆ when ∆ > D . This price is due to the complexity of the family of models
m m m
we use (there is nothing to pay in the simplest case of one model per dimension) and
this price is essentially unavoidable, as shown in a specific case by Birg´e and Massart
(2006).
It follows from the previous section that it is impossible to get an analogue of
Theorem 1 when d = d . We shall explain what kind of (necessarily weaker) results
2
can be obtained in this context and to what extent Theorem 1 can be rescued. For
this, we shall proceed in several steps. In the next section we shall explain how
to build estimators based on families of special models S , following the method
m
explained in Birg´e (2006a). These models need to be discrete subsets of LΓ (for
∞
some given Γ) with bounded metric dimension while there is no reason that our
initial models S be of this type (think of linear models). Section 4 will therefore
m
be devoted to the construction of such special models S from ordinary ones. This
m
construction will finally lead to an estimator sˆΓ belonging to LΓ , the performance
∞
of which strongly depends on our choice of Γ. In Section 5, we shall explain how,
given ageometrically increasingsequence(Γ ) of valuesofΓandthecorresponding
i i≥1
sequence of estimators sˆΓi, we can use the observations to choose a suitable value for
Γ. Since we have a single sample X to build the estimators sˆΓi and to choose Γ, we
shall proceed by sample splitting using one half of the sample for the construction of
the estimators and the second half to select a value of Γ. In particular, for the case
of a single model, this will lead to a generalized version of (2.8) that can also handle
the case of s L . When s L (with an unknown value of s ), the risk bounds
∞ ∞ ∞
6∈ ∈ k k
we get completely parallel (apart from some constants depending on s ) those
∞
k k
obtained for estimating s in the white noise model. We shall give a few applications
of these results, in particular to aggregation of preliminary estimators, in Section 6,
while the last section will be devoted to the most technical proofs.
3 T-estimators for L -loss
2
Inorder to defineestimators based on families of models with boundedmetric dimen-
sions, we shall follow the approach of Birg´e (2006a) based on what we have called
T-estimators. We refer to this paper for the definition of these estimators, recalling
that it relies on the existence of suitable tests between balls of the underlying metric
space L ,d . To derive such tests, we need a few specific technical tools to deal
2 2
with the L -distance.
2
(cid:0) (cid:1)
8
3.1 Tests between L -balls
2
3.1.1 Randomizing our sample
In the sequel we shall make use of randomized tests based on a randomization trick
duetoYangandBarron(1998, page106)whichhastheeffectofreplacingalldensities
involved in our problem by new ones which are uniformly bounded away from zero.
For this, we choose some number λ (0,1) and consider the mapping τ from L to
2
∈
L given by τ(u) = λu+1 λ. Note that τ is one-to-one and isometric, upto a factor
2
λ, i.e. d (τ(u),τ(v)) = λd−(u,v). If u LΓ , then τ(u) LΓ′ with Γ′ = λΓ+1 λ.
2 2 ∈ ∞ ∈ ∞ −
Let s′ = τ(s). Given our initial i.i.d. sample X, we want to build new i.i.d. vari-
ables X′,...,X′ with density s′. For this, we consider two independent n-samples,
1 n
Z ,...,Z and ε ,...,ε with respective distributions µ and Bernoulli with parame-
1 n 1 n
ter λ. Both samples are independent of X. We then set X′ = ε X +(1 ε )Z for
i i i − i i
1 i n. It follows that X′ has density s′ as required. We shall still denote by P
≤ ≤ i s
the probability on Ω that gives X′ = (X′,...,X′) the distribution P⊗n. Given two
1 n s′
distinct points t,u L we define a (randomized) test function ψ(X′) between t and
2
∈
u as a measurable function with values in t,u , ψ(X′) = t meaning deciding t and
{ }
ψ(X′) =u meaning deciding u.
Once we have used the randomization trick of Yang and Barron, for instance with
λ = 1/2, we deal with an i.i.d. sample X′ with a density s′ which is bounded from
below by 1/2 and we may therefore work within the set of densities that satisfy this
property.
3.1.2 Some minimax results
The main tool for the design of tests between L -balls of densities is the following
2
propositionwhich derives from theresults of Birg´e (1984) (keeping herethe notations
of that paper) and in particular from Corollary 3.2, specialized to the case of I = t
{ }
and c = 0.
Proposition 3 Let be some linear space of finite measures on some measurable
M
space (Ω, ) with a topology of a locally convex separated linear space. Let , be
A P Q
two disjoint sets of probabilities in and F a set of positive measurable functions
M
on Ω with the following properties (with respect to the given topology on ):
M
i) and are convex and compact;
P Q
ii) for any f F and 0 < z < 1 the function P fzdP is well-defined and
∈ 7→
upper semi-continuous on ;
P ∪Q R
iii) for any P , Q , t (0,1) and ε> 0, there exists an f F such that
∈ P ∈ Q ∈ ∈
(1 t) ftdP +t f1−tdQ < (dP)1−t(dQ)t +ε;
−
Z Z Z
iv) all probabilities in (respectively in ) are mutually absolutely continuous.
P Q
Then one can find P and Q such that
∈ P ∈ Q
t 1−t
Q P
sup dP = sup dQ = sup (dP)1−t(dQ)t
P Q
P∈PZ (cid:18) (cid:19) Q∈QZ (cid:18) (cid:19) P∈P,Q∈QZ
1−t t
= dP dQ .
Z
(cid:0) (cid:1) (cid:0) (cid:1)
9
In Birg´e (1984) we assumed that was the set of all finite measures on (Ω, ) but
M A
the proof actually only uses the fact that and are subsets of . Recalling that
P Q M
theHellingeraffinitybetweentwodensitiesuandvisdefinedbyρ(u,v) = √uvdµ =
1 h2(u,v), we get the following corollary.
− R
Corollary 1 Let µ be a probability measure on ( , ) and, for 1 i n, let
X W ≤ ≤
( , ) be a pair of disjoint convex and weakly compact subsets of L (µ) such that
i i 2
P Q
n
s > 0 µ-a.s. and sdµ = 1 for all s ( ). (3.1)
i i
∈ P ∪Q
Z i=1
[
For each i, one can find p and q such that
i i i i
∈ P ∈ Q
sup q /p udµ = sup p /q vdµ = sup ρ(u,v) = ρ(p ,q ).
i i i i i i
u∈PiZ v∈QiZ u∈Pi,v∈Qi
p p
Let X = (X ,...,X ) be a random vector on n with distribution n (s µ) with
1 n X i=1 i·
s for 1 i n and let x R. Then
i i
∈P ≤ ≤ ∈ N
n n n
P log(q /p )(X ) 2x e−x ρ(p ,q ) exp x h2(p ,q ) .
i i i i i i i
≥ ≤ ≤ − −
" # " #
i=1 i=1 i=1
X Y X
If X has distribution n (u µ) with u for 1 i n, then
i=1 i· i ∈ Qi ≤ ≤
n N n n
P log(q /p )(X ) 2x ex ρ(p ,q ) exp x h2(p ,q ) .
i i i i i i i
≤ ≤ ≤ −
" # " #
i=1 i=1 i=1
X Y X
Proof: We apply the previous proposition with t = 1/2, ( , ) = (Ω, ) and
X W A M
the set of measures of the form u µ, u L (µ) endowed with the weak L -topology.
2 2
· ∈
In view of (3.1), and can be identified with two sets of probabilities and we
i i
P Q
can take for F the set of all positive functions such that logf is bounded. As a
consequence, all four assumptions of Proposition 3 are satisfied. In order to get iii)
we simply take for f a suitably truncated version of s/u when P = s µ and Q = u µ.
· ·
As to the probability bounds they derive from classical exponential inequalities, as
for Lemma 7 of Birg´e (2006a).
3.1.3 Abstract tests between L -balls
2
The purpose of this section is to prove the following result.
Theorem 2 Let t,u LΓ for some Γ < + . For any x R, there exists a test
∈ ∞ ∞ ∈
ψ between t and u, based on the randomized sample X′ defined in Section 3.1.1
t,u,x
with a suitable value of λ, which satisfies
n t u 2 +x
sup P [ψ (X′)= u] exp k − k , (3.2)
s t,u,x
≤ − 65Γ
s∈L2|d2(s,t)≤d2(t,u)/4 " (cid:0) (cid:1)#
{ }
n t u 2 x
sup P [ψ (X′)= t] exp k − k − . (3.3)
s t,u,x
≤ − 65Γ
s∈L2|d2(s,u)≤d2(t,u)/4 " (cid:0) (cid:1)#
{ }
10