Table Of ContentCENTRAL LIMIT THEOREMS AND MULTIPLIER
BOOTSTRAP WHEN p IS MUCH LARGER THAN n
3
VICTOR CHERNOZHUKOV,DENISCHETVERIKOV,ANDKENGO KATO
1
0
2 Abstract. We derive a central limit theorem for the maximum of a
n sum ofhighdimensionalrandom vectors. Specifically,weestablish con-
a ditions under which the distribution of the maximum is approximated
J bythatofthemaximum ofasumoftheGaussian random vectorswith
3 the same covariance matrices as the original vectors. The key innova-
2 tion ofthisresult isthat it applies evenwhen thedimension ofrandom
vectors (p) is large compared to the sample size (n); in fact, p can be
] much larger than n. We also show that the distribution of the maxi-
T
mumofasumoftherandomvectorswithunknowncovariancematrices
S
can be consistently estimated by the distribution of the maximum of
.
h a sum of the conditional Gaussian random vectors obtained by mul-
t tiplying the original vectors with i.i.d. Gaussian multipliers. This is
a
m the multiplier bootstrap procedure. Here too, p can be large or even
muchlargerthann. Thesedistributionalapproximations,eitherGauss-
[
ian or conditional Gaussian, yield a high-quality approximation to the
3 distribution of the original maximum, often with approximation error
v decreasing polynomially in thesample size, and henceare of interest in
6 many applications. Wedemonstratehow ourcentrallimit theorem and
0 the multiplier bootstrap can be used for high dimensional estimation,
9 multiplehypothesistesting,andadaptivespecificationtesting. Allthese
6
results contain non-asymptoticboundson approximation errors.
.
2
1
2
1 1. Introduction
:
v Let x ,...,x be independent random vectors in Rp, with each x having
1 n i
i
X coordinates denoted by xij, i.e., xi = (xi1,...,xip)′. Suppose that each xi
is centered, namely E[x ] = 0, and has a finite covariance matrix E[x x ].
r i i ′i
a
Consider the rescaled average:
n
1
(1) X := (X ,...,X ) := x .
1 p ′ i
√n
i=1
X
Date: First version: June 2012. First Arxiv version: December, 2012. This version:
January 25, 2013.
Key words and phrases. Dantzig selector, Slepian, Stein method, maximum of vector
sums, high dimensionality, anti-concentration.
V.ChernozhukovandD.ChetverikovaresupportedbyaNationalScienceFoundation
grant. K.KatoissupportedbytheGrant-in-AidforYoungScientists(B)(22730179), the
Japan Society for thePromotion of Science.
1
2 CHERNOZHUKOV,CHETVERIKOV,ANDKATO
Our goal is to obtain a distributional approximation for the statistic T
0
defined as the maximum coordinate of vector X:
T := max X ,
0 j
16j6p
The distribution of T is of interest in many applications. When p is fixed,
0
this distribution can be approximated by the classical Central Limit The-
orem (CLT) applied to X. However, in modern applications, cf. [10], p
is often comparable or even larger than n, and the classical CLT does not
apply in such cases. This paper provides a tractable approximation to the
distribution of T when p is large and possibly much larger than n.
0
Thefirst mainresultofthepaperistheGaussianapproximationtheorem,
which bounds the Kolmogorov distance between the distributions of T and
0
its Gaussian analog Z . Specifically, let y ,...,y be independent centered
0 1 n
Gaussian random vectors in Rp such that each y has the same covariance
i
matrix as x , namely y N(0,E[x x ]). Consider the rescaled average of
i i ∼ i ′i
these vectors,
n
1
(2) Y := (Y ,...,Y ) := y .
1 p ′ i
√n
i=1
X
Vector Y is the Gaussian analog of X in the sense of sharing the same
mean and covariance matrix, namely E[X] = E[Y] = 0 and E[XX ] =
′
E[YY ]= n 1 n E[x x ]. We then define the Gaussian analog Z of T as
′ − i=1 i ′i 0 0
the maximum coordinate of vector Y:
P
(3) Z := max Y .
0 j
16j6p
Ourmain resultshowsthat, undersuitable momentassumptions, as n
→ ∞
and possibly p = p ,
n
→ ∞
(4) ρ := sup P(T 6 t) P(Z 6 t) 6 Cn c 0,
0 0 −
t R| − | →
∈
where constants c > 0 and C > 0 are independent of n.
Importantly, in (4), p can be large in comparison to n and be nearly
as large as eo(n1/7). For example, if x are uniformly bounded (namely,
ij
x 6 C for some constant C > 0 for all i and j) the Kolmogorov dis-
ij 1 1
| |
tance ρ converges to zero at a polynomial rate whenever (logp)7/n 0 at
→
a polynomial rate. We obtain similar results when x are sub-exponential
ij
and even non-sub-exponential under suitable moment assumptions. Figure
1 illustrates the result (4) in a non-subexponential example, which is moti-
vatedbytheanalysis oftheDantzig Selector of [11]innon-Gaussiansettings
(see Section 4).
The proof of the Gaussian approximation result (4) builds on a number
of technical tools such as Slepian’s smart path interpolation (which is re-
lated to the solution of Stein’s partial differential equation; cf. Appendix
E), Stein’s leave-one-out method, approximation of maxima by the smooth
CLT AND MULTIPLIER BOOTSTRAP WHEN p IS MUCH LARGER THAN n 3
n=100; p=5000 n=400; p=5000
1 1
0.8 0.8
0.6 0.6
0.4 0.4
0.2 0.2
0 0
0 0.5 1 0 0.5 1
Figure 1. P-P plots comparing distributions of T0 and Z0 in the
example motivated by the problem of seleting the penalty level of the
Dantzigselector. Herex aregeneratedasx =z ε withε ∼t(4),(a
ij ij ij i i
t-distribution with four degrees of freedom), and z are non-stochastic
ij
(simulatedonceusingU[0,1]distributionindependentlyacrossiandj).
Thedashedlineis45◦. ThedistributionsofT0andZ0areclose,as(qual-
itatively)predictedbytheCLTderivedinthepaper: seeCorollaries2.1
or 2.2. The quality of the Gaussian approximation is particularly good
for the tail probabilities, which is most relevant for practical applica-
tions.
functions (related to “free energy” in spin glasses), and exponential inequal-
ities for self-normalized sums. See, e.g., [41, 42, 20, 15, 43, 12, 19, 36] for
introduction and discussion of some of these tools. It also critically relies
on the anti-concentration and comparison bounds of maxima of Gaussian
vectors derived in [18] and restated in this paper as Lemmas 2.1 and 3.1.
Our new Gaussian approximation theorem has the following innovative
features. To the best of our knowledge, this is the first general result that
establishes that maxima of sums of random vectors can be approximated
in distribution by the maxima of sums of Gaussian random vectors when
p n and especially when p is of order enc for some c > 0. The existing
≫
techniques can also lead to results of the form (4) when p = p , but
n
→ ∞
under much stronger conditions on p. For example, Yurinskii’s coupling
implies (4) but requires p5/n 0; see Example 17 (Section 10) in [37].
→
Second,ourGaussianapproximationtheoremcoverscaseswhereT doesnot
0
have a limit distribution as n and p = p . In some cases, after a
n
→ ∞ → ∞
suitablenormalization,T couldhaveanextremevaluedistributionasalimit
0
distribution,buttheapproximationtoanextremevaluedistributionrequires
some restrictions on the dependency structure among the coordinates in x .
i
Our result does not require such restrictions on the dependency structure.
Third,the quality of approximation in (4)is of polynomialorder in n, which
is better than the logarithmic in n quality that we could obtain in some
(though not all) applications using the approximation of the distribution of
T by an extreme value distribution (see [33]).
0
4 CHERNOZHUKOV,CHETVERIKOV,ANDKATO
Our result also contributes to the literature on multivariate central limit
theorems, which are concerned with conditions under which
(5) P(X A) P(Y A) 0,
| ∈ − ∈ |→
uniformly in a collection of sets A, typically all convex sets. Such results
weredeveloped among others, by [35,38,25,7,14], underconditions of type
pc/n 0 (also see [13]). These results rely on the anti-concentration results
→
for Gaussian random vectors on the δ-expansions of boundaries of arbitrary
convex sets A (see [4]). Note that our result also establishes (5), but uni-
formly for all convex sets of the form A = a Rp : max a 6 t
max 16j6p j
for t R. These sets have a rather special stru{ctu∈re that allows us to dea}l
∈
with p n: in particular, concentration of measure on the δ-expansion of
≫
boundary of A is at most of order δ√logp for Gaussian random vectors
max
with unit variance, as shown in [18] (see also Lemma 2.1). (The relation (5)
with A= A explains the sense in which we have a CLT, as appearing in
max
the title of the paper.)
Note that the result (4) is immediately useful for inference with statistic
T , even though P(Z 6 t) needs not converge itself to a well behaved
0 0
distribution function. Indeed, if the covariance matrix n 1 n E[x x ]
− i=1 i ′i
is known, then c (1 α) := (1 α)-quantile of Z , can be computed
Z0 − − 0 P
numerically, and we have
(6) P(T 6 c (1 α)) (1 α) 6 Cn c 0.
| 0 Z0 − − − | − →
A chief application of this kind arises in determination of the penalty level
for the Dantzig selector of [11] in the high-dimensional regression with non-
Gaussian errors, which we examine in Section 5. There, underthe canonical
(homoscedastic) noise, the covariance matrix is known, and so quantiles of
Z can be easily computed numerically and used for choosing the penalty
0
level. However, if the noise is heteroscedastic, the covariance matrix is no
longer known, and this approach is no longer feasible. This motivates our
second main result.
The second main result of the paper establishes validity of the mul-
tiplier bootstrap for estimating quantiles of Z when the covariance ma-
0
trix n 1 n E[x x ] is unknown. More precisely, we define the Gaussian-
− i=1 i ′i
symmetrized version W of T by multiplying x with i.i.d. standard Gauss-
0 0 i
P
ian random variables e ,...,e :
1 n
n
1
(7) W := max x e .
0 ij i
16j6p√n
i=1
X
We show that the conditional quantiles of W given data (x )n are able to
0 i i=1
consistently estimate the quantiles of Z and hence those of T (where the
0 0
notion of consistency used is the one that guarantees asymptotically valid
inference). Here the primary factor driving the bootstrap estimation error
is the maximum difference between the empirical and population covariance
CLT AND MULTIPLIER BOOTSTRAP WHEN p IS MUCH LARGER THAN n 5
matrices:
n
1
∆ := max (x x E[x x ]) ,
ij ik ij ik
16j,k6p(cid:12)n − (cid:12)
(cid:12) Xi=1 (cid:12)
(cid:12) (cid:12)
whichcan converge tozero even(cid:12) whenp is much larger th(cid:12)an n. For example,
(cid:12) (cid:12)
when x are uniformly bounded, the multiplier bootstrap is valid for infer-
ij
ence if (logp)7/n 0. Earlier related results on bootstrap in the “p
→ → ∞
but p/n 0” regime were obtained in [34]; interesting results for the case
→
p n based on concentration inequalities and symmetrization are studied
≫
in [2, 3], albeit the approach and results are quite different from those given
here. In particular, in [2], either Gaussianity or symmetry in distribution is
imposed on the data.
Thekeymotivatingexampleofouranalysisisthehigh-dimensionalsparse
regression model. In this model, [11] and [8] assume Gaussian errors to an-
alyze the Dantzig selector and Lasso. Our results show that Gaussianity is
not necessary and the Gaussian-like conclusions hold approximately, with
just the fourth moment of the regression errors being bounded. Moreover,
ourapproximationallowstotakeintoaccountcorrelationsamongtheregres-
sors. This leads to a better choice of the penalty level and tighter bounds
on performancethan those that had beenavailable previously. For example,
some of the same goals had been accomplished using moderate deviations
for self-normalized sums, combined with the union bound [6]. However, the
union bound does not take into account correlations among the regressors,
and so it may be overly conservative in some applications.
Our results have a broad range of other applications. In addition to the
high-dimensionalestimationexample,weshowintheSupplementalMaterial
how to apply our results in the multiple hypothesis testing via the step-
down method of [40] and to specification testing. In either case number of
hypotheses to be tested or the number of moment restrictions to be tested
can be much larger than the sample size. Lastly, in a companion work
([17]), we are exploring the strong coupling for suprema of general empirical
processes based on the methods developed here and maximal inequalities.
These results represent a useful complement to the results based on the
Hungarian coupling developed by [32, 9, 30, 39] for the entire empirical
process and have applications to inference in nonparametric problems such
as construction of uniform confidence bands (see, e.g., [24]).
1.1. Organization of the paper. In Section 2, we give the results on
Gaussian approximation, and in Section 3 on the multiplier bootstrap. In
Section 4, we present an application to the Dantzig selector. Appendices
A-D contain proofs for each of these sections, with AppendixA stating aux-
iliary tools and lemmas. Due to the space limitation, we put additional re-
sultsandproofsinto SupplementalMaterial, AppendicesG-F.Inparticular,
Appendices G and H provide additional applications to multiple hypothesis
and adaptive specification testing.
6 CHERNOZHUKOV,CHETVERIKOV,ANDKATO
1.2. Notation. In what follows, unless otherwise stated, we will assume
that p > 3. In making asymptotic statements we assume that n
→ ∞
with understanding that p depends on n and possibly p as n
→ ∞ →
. Constants c,C,c ,C ,c ,C ,... are understood to be independent of n.
1 1 2 2
∞Throughoutthepaper,E []denotestheaverage overindex1 6 i6 n,i.e., it
n
simply abbreviates the not·ation n 1 n []. E.g., E [x2 ] = n 1 n x2.
− i=1 · n ij − i=1 ij
In addition, E¯[] = E [E[]]. For example, E¯[x2] = n 1 n E[x2]. For a
· n · P ij − i=1 iPj
function f :R R, we write ∂kf(x)= ∂kf(x)/∂xk for nonnegative integer
k; for a functio→n f :Rp R, we write ∂ f(x)= ∂f(x)/∂Px for j = 1,...,p,
j j
→
where x = (x ,...,x ). Denote by Ck(R) the class of k times continuously
1 p ′
differentiable functions from R to itself, and denote by Ck(R) the class of
b
aWllefwunrictteioans.fb∈if aCkis(Rsm) aslulechr tthhaant osrupezq∈uRal|∂tjof(bzu)|p<to∞a ufnorivejrs=al0p,o.s.i.t,ivke.
constant. For a,b R, we write a b = max a,b .
∈ ∨ { }
2. Central Limit Theorems for Maxima of Non-Gaussian Sums
2.1. ComparisonTheoremsandNon-AsymptoticGaussianApprox-
imations. The purpose of this section is to compare and bound the dif-
ference between the expectations and distribution functions of the non-
Gaussian to Gaussian maxima:
T := max X and Z := max Y ,
0 j 0 j
16j6p 16j6p
where vector X is defined in equation (1) and Y in equation (2). Here and
in what follows, without loss of generality, we will assume that (x )n and
i i=1
(y )n are independent. The following envelopes and bounds on moments
i i=1
will be used in stating the bounds in Gaussian approximations:
(8) S := max(x + y ), M := max(E¯[xk])1/k.
i 16j6p | ij| | ij| k 16j6p ij
The problem of comparing distributions of maxima is of intrinsic diffi-
culty since the maximum function z = (z ,...,z ) max z is non-
1 p ′ 16j6p j
7→
differentiable. To circumvent the problem, we use a smooth approximation
of the maximum function. For z = (z ,...,z ) Rp, consider the function:
1 p ′
∈
p
F (z) := β 1log exp(βz ) ,
β − j
j=1
X
which approximates the maximum function, where β > 0 is the smoothing
parameter that controls the level of approximation (we call this function the
“smooth max function”). Indeed, an elementary calculation shows that for
all z Rp,
∈
(9) 0 6 F (z) max z 6 β 1logp.
β j −
−16j6p
This smooth max function arises in the definition of “free energy” in spin
glasses; see, e.g., [43].
CLT AND MULTIPLIER BOOTSTRAP WHEN p IS MUCH LARGER THAN n 7
We start with the following “warm-up” theorem that conveys the main
qualitative feature of the problem. Here and in what follows, for a smooth
function g :R R, write
→
G := sup ∂kg(z), k > 0.
k
z R| |
∈
Theorem 2.1 (Comparison of Gaussian to Non-Gaussian Maxima, I). For
every g C3(R) and β > 0,
∈ b
E[g(F (X)) g(F (Y))] . n 1/2(G +G β +G β2)E¯[S3],
| β − β | − 3 2 1 i
and hence
E[g(T ) g(Z )] . n 1/2(G +G β +G β2)E¯[S3]+β 1G logp.
| 0 − 0 | − 3 2 1 i − 1
Comment 2.1 (Optimizingthebound). Thetheoremboundsthedifference
betweentheexpectationsofsmoothfunctionsofmaxima. Theoptimalvalue
of the last bound is given by
min n 1/2(G +G β+G β2)E¯[S3]+β 1G logp.
− 3 2 1 i − 1
β>0
We postpone choices of β to the proofs of subsequent corollaries, leaving
ourselves more flexibility in optimizing bounds in those corollaries. (cid:4)
Deriving a bound on the Kolmogorov distance between distributions of
T and Z from Theorem 2.1 is not a trivial issue and this step relies on
0 0
the following anti-concentration inequality for maxima of Gaussian random
variables, which is derived in [18].
Lemma 2.1 (Anti-Concentration). Let ξ ,...,ξ be (not necessarily inde-
1 p
pendent) centered Gaussian random variables with σ2 := E[ξ2] > 0 for all
j j
1 6 j 6 p. Let σ = min σ and σ¯ = max σ . Then for every
16j6p j 16j6p j
ς > 0,
supP max ξ z 6 ς 6 Cς 1 log(p/ς),
j
z R (cid:18)|16j6p − | (cid:19) ∨
where C > 0 is∈a constant depending only onpσ and σ¯. When σ are all
j
equal, log(p/ς) on the right side can be replaced by logp.
By Theorem 2.1 and Lemma 2.1, we can now derive a bound on the
Kolmogorov distance between distributions of T and Z .
0 0
Corollary 2.1 (Central Limit Theorem, I). Suppose that there are some
constants c > 0 and C > 0 such that c 6 E¯[x2] 6 C for all 1 6 j 6 p.
1 1 1 ij 1
Then there exists a constant C > 0 depending only on c and C such that
1 1
ρ:= sup P(T 6 t) P(Z 6 t) 6 C(n 1(log(pn))7)1/8(E¯[S3])1/4.
t R| 0 − 0 | − i
∈
Comment 2.2 (Main qualitative feature: logarithmic dependence on p).
Theorem 2.1 and Corollary 2.1 imply that the error of approximating the
maximum coordinate in the sum of independent random vectors by its
8 CHERNOZHUKOV,CHETVERIKOV,ANDKATO
Gaussian analogue depends on p (possibly) only through logp. This is the
main qualitative feature of all the results in this paper. Note also that the
term E¯[S3] implicitly encodes the complexity of the vectors, in particular
i
it will reflect the correlation structure of vectors X and Y. However, both
Theorem 2.1 and Corollary 2.1 and all subsequent results given below do
not limit the dependence among the coordinates in xi. (cid:4)
Comment 2.3 (Motivation for the next result). While Theorem 2.1 and
Corollary 2.1 convey an important qualitative aspect of the problem and
admit easy-to-grasp proofs, an important disadvantage of these results is
that the bounds depend on E¯[S3]. If E¯[S3] 6 C, Corollary 2.1 leads to
i i
ρ= O((n 1(log(pn))7)1/8) and ρ 0 as long as logp = o(n1/7). This is the
−
→
case when, for example, as in caption to Figure 1,
x = z ε , z are non-stochastic with z 6 C, E[ε 3] 6 C.
ij ij i ij ij i
| | | |
WhenE¯[S3]increases with n,however, theboundsneed notbeas good, and
i
can be improved considerably by using a truncation method. Using such a
method in conjunction with the proof strategy of Theorem 2.1, we derive
in Theorem 2.2 below a bound that can be much better in the latter sce-
nario. The improvement here comes at a cost of a more involved statement,
involving truncation parameters. (cid:4)
To derive our next main result, we employ a truncation method. Given a
threshold level u > 0, define a truncated version of x by
ij
(10) x˜ = x 1 x 6 u(E¯[x2 ])1/2 E x 1 x 6 u(E¯[x2])1/2 .
ij ij | ij| ij − ij | ij| ij
n o h n oi
Let ϕ (u) be the infimum, which is attained, over all numbers ϕ > 0 such
x
that
(11) E¯ x2 1 x > u(E¯[x2 ])1/2 6 ϕ2E¯[x2 ].
ij | ij| ij ij
h n oi
Note that the function ϕ (u) is right-continuous; it measures the impact of
x
truncation on second moments. Define u (γ) as the infimum u > 0 such
x
that
P x 6 u(E¯[x2 ])1/2,1 6 i6 n,1 6 j 6 p > 1 γ.
| ij| ij −
Also define ϕ(cid:16)(u) and u (γ) by the corresponding qu(cid:17)antities for the ana-
y y
logue Gaussian case, namely with (x )n replaced by (y )n in the above
i i=1 i i=1
definitions. Throughout the paper we use the following quantities:
ϕ(u) := ϕ (u) ϕ (u), u(γ) := u (γ) u (γ).
x y x y
∨ ∨
Here is the main theorem of this section. Recall the definition of M in (8).
k
Theorem 2.2 (Comparison of Gaussian to Non-Gaussian Maxima, II). Let
β > 0,u > 0 and γ (0,1) be such that 2√2uM β/√n 6 1 and u > u(γ).
2
Then for every g C∈3(R),
∈ b
E[g(F (X)) g(F (Y))] . D (g,β,u,γ),
β β n
| − |
CLT AND MULTIPLIER BOOTSTRAP WHEN p IS MUCH LARGER THAN n 9
and hence
E[g(T ) g(Z )] . D (g,β,u,γ)+β 1G logp,
0 0 n − 1
| − |
where
D (g,β,u,γ) := n 1/2(G +G β+G β2)M3+(G +βG )M2ϕ(u)
n − 3 2 1 3 2 1 2
+G M ϕ(u) log(p/γ)+G γ.
1 2 0
By Theorem 2.2 and Lemmap2.1, we can obtain a bound on the Kol-
mogorov distance between the distribution functions of T and Z .
0 0
Corollary 2.2 (Central Limit Theorem, II). Suppose that there are
some constants 0< c < C such that c 6E¯[x2 ] 6 C for 1 6j 6 p. Then
1 1 1 ij 1
for every γ (0,1),
∈
ρ6 C n 1/8(M3/4 M1/2)(log(pn/γ))7/8 +n 1/2(log(pn/γ))3/2u(γ)+γ ,
− 3 ∨ 4 −
where hC > 0 is a constant that depends on c and C only. i
1 1
In applications it is useful to bound the upper function u(γ). Here is
a simple and effective way of doing this. Let h : [0, ) [0, ) be a
∞ → ∞
Young-Orlicz modulus, i.e., a convex and strictly increasing function with
h(0) = 0. Denote by h 1 the inverse function of h. Standard examples
−
include the power function h(v) = vq with inverse h 1(γ) = γ1/q and the
−
exponential function h(v) = exp(v) 1 with inverse h 1(γ) = log(γ +1).
−
−
Thesefunctions describehow many moments therandomvariables have, for
example, a random variable ξ has finite q-th moment if E[ξ q] < , and
| | ∞
is sub-exponential if E[exp(ξ /C)] < for some C > 0. We refer to [44],
| | ∞
Chapter 2.2, for further details on Young-Orlicz moduli.
Lemma 2.2 (Bounds on the upper function u(γ)). Let h : [0, ) [0, )
∞ → ∞
be a Young-Orlicz modulus, and let B > 0 and D > 0 be constants such that
(E[x2 ])1/2 6 B for all 1 6 i 6 n,1 6 j 6 p, and E¯[h(max x /D)] 6
ij 16j6p| ij|
1. Then under the condition of Corollary 2.2,
u(γ) 6 Cmax Dh 1(n/γ),B log(pn/γ) ,
−
{ }
where C > 0 is a constant that depends on c1pand C1 only.
Inapplications,parametersB andD(withM andM aswell)areallowed
3 4
toincreasewithn. ThesizeoftheseparametersandthechoiceoftheYoung-
Orlicz modulus are case-specific.
2.2. Examples of Applications. The purpose of this subsection is to ob-
tain bounds on ρ for various leading examples frequently encountered in
applications. We are concerned with simple conditions under which ρ de-
cays polynomially in n.
Let c > 0,c > 0,C > 0 be some constants, and let B > 1 be a
1 2 1 n
sequence of constants. We allow for the case where B as n .
n
→ ∞ → ∞
We shall first consider applications where one of the following conditions is
satisfied uniformly in 1 6 i6 n and 1 6 j 6 p:
10 CHERNOZHUKOV,CHETVERIKOV,ANDKATO
(E.1) E¯[x2 ]> c and E¯[S3]6 C ;
ij 1 i 1
(E.2) E¯[x2 ]> c and E[exp(x /C )]6 2;
ij 1 | ij| 1
(E.3) c 6 E¯[x2 ]6 C and x 6 B .
1 ij 1 | ij| n
Comment 2.4. Condition(E.1) is perhapsthesimplestexamplein thispa-
per;underthiscondition application of Corollary 2.1 is effective. Aconcrete
example with condition (E.1) satisfied is the case where x = z ε , z are
ij ij i ij
non-stochastic with z 6 C, and E[ε 3] 6 C. Conditions (E.2)-(E.5) are
ij i
| | | |
more elaborate, intended to cover cases where moments of the envelopes S
i
and higher order moments M and M increase with n. In these cases the
3 4
use of Corollary 2.1 is not effective, and we shall use Corollary 2.2 instead.
Condition (E.2) covers vectors x made up from sub-exponential random
i
variables, including sub-Gaussian as a special case; this example is quite
often used in high-dimensional statistics. Condition (E.3) covers variables
that are bounded by B , which may increase with n; many applications,
n
after a suitable truncation, can be covered by it. (cid:4)
We shall also consider regression applications where one of the following
conditions is satisfied uniformly in 1 6 i 6 n and 1 6 j 6 p:
(E.4) x = z ε , where z are non-stochastic with z 6 B , E [z2] =
ij ij ij ij | ij| n n ij
1, and E[ε ] = 0, E[ε2 ]> c , and E[exp(ε /C )] 6 2; or
ij ij 1 | ij| 1
(E.5) x = z ε , where z are non-stochastic with z 6 B , E [z2] =
ij ij ij ij | ij| n n ij
1, and E[ε ] = 0, E[ε2 ]> c , and E[max ε4 ] 6 C .
ij ij 1 16j6p ij 1
Comment 2.5. The last two cases cover examples that arise in high-
dimensional regression, e.g., [11], which we shall revisit later in the paper.
Typically,ε areindependentofj (i.e.,ε = ε )andhenceE[max ε4 ] 6
ij ij i 16j6p ij
C in condition (E.5) reduces to E[ε4]6 C (we allow ε dependent on j so
1 i 1 ij
that Corollary 2.3 covers the multiple hypothesis testing example in Appen-
dix G). Interestingly, these examples are also connected to spin glasses, see
e.g., [43] and [36] (z can be interpreted as generalized products of “spins”
ij
and εi as their random “interactions”). (cid:4)
Corollary 2.3 (Central Limit Theorem in Leading Examples). Sup-
pose that one of the following conditions is satisfied: (i) condition (E.1) and
(log(pn))7/n 6 C n c2; (ii) condition (E.2) and (log(pn))7/n 6 C n c2;
1 − 1 −
(iii) condition (E.3) and B2(log(pn))7/n 6 C n c2; (vi) condition (E.4)
n 1 −
andB2(log(pn))7/n 6 C n c2; or(v)condition (E.5)andB4(log(pn))7/n 6
n 1 − n
C n c2. Then there exist constants c> 0 and C > 0 depending only on c ,c
1 − 1 2
and C such that
1
ρ 6 Cn c.
−
Comment 2.6. Cases (ii)-(v) indeed follow relatively directly from Corol-
lary 2.2 with help of Lemma 2.2. Moreover, from Lemma 2.2, it is routine
to find other conditions that lead to the conclusion of Corollary 2.3. (cid:4)
