Table Of ContentEIGENVALUE DISTRIBUTION OF LARGE SAMPLE COVARIANCE
MATRICES OF LINEAR PROCESSES
OLIVERPFAFFELANDECKHARDSCHLEMM
Abstract. Wederivethedistributionoftheeigenvaluesofalargesamplecovariancematrixwhen
2 thedataisdependentintime. Moreprecisely,thedependenceforeachvariablei=1,...,pis
1 modelledasalinearprocess(Xi,t)t=1,...,n=((cid:80)∞j=0cjZi,t−j)t=1,...,n,where{Zi,t}areassumed
0 tobeindependentrandomvariableswithfinitefourthmoments. Ifthesamplesizenandthe
2 number of variables p=pn both converge to infinity such that y =limn→∞n/pn >0, then
theempiricalspectraldistributionofp−1XXT convergestoanon-randomdistributionwhich
n
a only depends on y and the spectral density of (X1,t)t∈Z. In particular, our results apply to
(fractionallyintegrated)ARMAprocesses,whichweillustratebysomeexamples.
J
8
1
] 1. Introduction and main result
R
A typical object of interest in many fields is the sample covariance matrix (n−1)−1XXT of
P
. a data matrix X=(Xi,t)it, i=1,...,p, t=1,...,n. The matrix X can be seen as a sample of
h
size n of p-dimensional data vectors. For fixed p one can show, as n tends to infinity, that under
t
a certain assumptions the eigenvalues of the sample covariance matrix converge to the eigenvalues of
m
the true underlying covariance matrix [2]. However, the assumption p(cid:28)n may not be justified if
[ one has to deal with high dimensional data sets, so that it is often more suitable to assume that
the dimension p is of the same order as the sample size n, that is p=p →∞ such that
1 n
n
v
(1.1) lim =:y ∈(0,∞).
8 n→∞ p
2
For a symmetric matrix A with eigenvalues λ ,...,λ , we denote by
8 1 p
3 1(cid:88)p
. FA = δ
1 p λi
0 i=1
2 the spectral distribution of A, where δ denotes the Dirac measure located at x. This means
x
1 that pFA(B) is equal to the number of eigenvalues of A that lie in the set B. From now on we
v: will call p−1XXT the sample covariance matrix. Due to Eq. (1.1), this change of normalization
i can be reversed by a simple transformation of the limiting spectral distribution. For notational
X
convenience we suppress the explicit dependence of the occurring matrices on n and p where this
r
does not cause ambiguity.
a
The distribution of Gaussian sample covariance matrices of fixed size was first computed in [20].
Several years later, it was Marchenko and Pastur [14] who considered the case where the random
variables {X } are more general i.i.d. random variables with finite second moments EX2 =1,
i,t 11
and the number p of variables is of the same order as the sample size n. They showed that the
empiricalspectraldistribution(ESD)Fp−1XXT ofp−1XXT converges,asn→∞,toanon-random
distribution Fˆ, called limiting spectral distribution (LSD), given by
(1.2) Fˆ(dx)= 2π1x(cid:112)(x+−x)(x−x−)1{x−(cid:54)x(cid:54)x+}dx,
√
andpointmassFˆ({0})=1−y ify <1; inthisformula, x =(1± y)2. Hereandinthefollowing,
±
convergence of the ESD means almost sure convergence as a random element of the space of
probability measures on R equipped with the weak topology. In particular, the eigenvalues of the
2000 Mathematics Subject Classification. Primary: 15A52;Secondary: 62M10.
Key words and phrases. eigenvaluedistribution,fractionallyintegratedARMAprocess,limitingspectraldistri-
bution,linearprocess,randommatrixtheory,samplecovariancematrix.
1
2 OLIVERPFAFFELANDECKHARDSCHLEMM
sample covariance matrix of a matrix with independent entries do not converge to the eigenvalues
of the true covariance matrix, which is the identity matrix and therefore only has eigenvalue one.
This leads to the failure of statistics that rely on the eigenvalues of p−1XXT which have been
derived under the assumption of fixed p, and random matrix theory is a tool to correct these
statistics [4, 13]. In the case where the true covariance matrix is not the identity matrix, the LSD
can in general only be given in terms of a non-linear equation for its Stieltjes transform, which is
defined by
(cid:90) 1
m (z)= dFˆ ∀z ∈C+ :={z =u+iv ∈C:(cid:61)z =v >0}.
Fˆ λ−z
Conversely, the distribution Fˆ can be obtained from its Stieltjes transform m via the Stieltjes–
Fˆ
Perron inversion formula ([3, Theorem B.8]), which states that
1 (cid:90) b
(1.3) Fˆ([a,b])= lim (cid:61)m (x+i(cid:15))dx.
π (cid:15)→0+ a Fˆ
for all continuity points a < b of Fˆ. For a comprehensive account of random matrix theory we
refer the reader to [1, 3, 15], and the references therein.
Our aim in this paper is to obtain a Marchenko–Pastur type result in the case where there is
dependence within the rows of X. More precisely, for i=1,...,p, the ith row of X is given by a
linear process of the form
∞
(cid:88)
(Xi,t)t=1,...,n = cjZi,t−j , cj ∈R.
j=0
t=1,...,n
Here, (Z ) is an array of independent random variables that satisfies
i,t it
(1.4) EZ =0, EZ2 =1, and σ :=supEZ4 <∞,
i,t i,t 4 i,t
i,t
as well as the Lindeberg-type condition that, for each (cid:15)>0,
p n
1 (cid:88)(cid:88) (cid:16) (cid:17)
(1.5) E Z2 1 →0, as n→∞.
pn i,t {Zi2,t(cid:62)(cid:15)n}
i=1j=1
Clearly, Eq. (1.5) is satisfied if all {Z } are identically distributed.
i,t
The novelty of our result is that we allow for dependence within the rows, and that the equation
for m is given in terms of the spectral density
Fˆ
(cid:88)
f(ω)= γ(h)e−ihω, ω ∈[0,2π],
h∈Z
of the linear processes X only, which is the Fourier transform of the autocovariance function
i
∞
(cid:88)
γ(h)= c c , h∈Z.
j j+|h|
j=0
Potential applications arise whenever data is not independent in time such that the Marchenko–
Pastur law is not a good approximation. This includes e.g. wireless communications [19] and
mathematical finance [18, 17]. Note that a similar question is also discussed in [5]. However, they
have a different proof which relies on a moment condition to be verified. Furthermore, they assume
that the random variables {Z } are identically distributed so that the processes within the rows
i,t
areindependentcopiesofeachother. Moreimportantly,theirresultsdonotyieldconcreteformulas
exceptintheAR(1)caseandarethereforenotdirectlyapplicable. Inthecontextoffreeprobability
theory, the limiting spectral distribution of large sample covariance matrices of Gaussian ARMA
processes is investigated in [7].
Beforewepresentthemainresultofthisarticle,weexplainthenotationusedinthisarticle. The
symbolsZ,NR,andCdenotethesetsofintegers,natural,real,andcomplexnumbers,respectively.
For a matrix A, we write AT for its transpose and trA for its trace. Finally, the indicator of an
EIGENVALUE DISTRIBUTION OF LARGE SAMPLE COVARIANCE MATRICES OF LINEAR PROCESSES 3
expression E is denoted by I and defined to be one if E is true, and zero otherwise; for a set S,
{E}
we also write I (x) instead of I .
S {x∈S}
Theorem 1.1. For each i=1,...,p, let X =(cid:80)∞ c Z , t∈Z, be a linear stochastic process
i,t j=0 j i,t−j
with continuously differentiable spectral density f. Assume that
i) the array (Z ) satisfies conditions (1.4) and (1.5),
i,t it
ii) there exist positive constants C and δ such that |c |(cid:54)C(j+1)−1−δ for all j (cid:62)0,
j
iii) for almost all λ∈R, f(ω)=λ for at most finitely many ω ∈[0,2π], and
iv) f(cid:48)(ω)(cid:54)=0 for almost every ω.
Then the empirical spectral distribution Fp−1XXT of p−1XXT converges, as n tends to infinity,
almost surely to a non-random probability distribution Fˆ with bounded support. Moreover, there
exist positive numbers λ ,λ such that the Stieltjes transform z (cid:55)→ m (z) of Fˆ is the unique
− + Fˆ
mapping C+ →C+ satisfying
1 y (cid:90) λ (cid:88) 1
(1.6) =−z+ λ λ+ dλ.
m (z) 2π − 1+λm (z) |f(cid:48)(ω)|
Fˆ Fˆ ω∈[0,2π]:f(ω)=λ
The assumptions of the theorem are met, for instance, if (X ) is an ARMA or fractionally
i,t t
integrated ARMA process; see Section 3 for details.
Theorem 1.1, as it stands, does not contain the classical Marchenko–Pastur law as a special case.
For if the entries X of the matrix X are i.i.d., the corresponding spectral density f is identically
i,t
equal to the variance of X , and thus condition iv) is not satisfied. We therefore also present a
1,1
version of Theorem 1.1 that holds if the rows of the matrix X have a piecewise constant spectral
density.
Theorem 1.2. For each i=1,...,p, let X =(cid:80)∞ c Z , t∈Z, be a linear stochastic process
i,t j=0 j i,t−j
with spectral density f of the form
k
(cid:88)
(1.7) f :[0,2π]→R+, ω (cid:55)→ α 1 (ω), k ∈N,
j Aj
j=1
for some positive real numbers α and a measurable partition A ∪···∪A of the interval [0,2π].
j 1 k
If conditions i) and ii) of Theorem 1.1 hold, then the empirical spectral distribution Fp−1XXT
of p−1XXT converges, as n → ∞, almost surely to a non-random probability distribution Fˆ
with bounded support. Moreover, the Stieltjes transform z (cid:55)→m (z) of Fˆ is the unique mapping
Fˆ
C+ →C+ that satisfies
k
(1.8) 1 =−z+ y (cid:88) |Aj|αj ,
m (z) 2π 1+α m (z)
Fˆ j=1 j Fˆ
where |A | denotes the Lebesgue measure of the set A . In particular, if the entries of X are i.i.d.
j j
with unit variance, one recovers the limiting spectral distribution (1.2) of the Marchenko–Pastur
law.
Remark 1.3. In applications one often considers processes of the form X =µ+(cid:80)∞ c Z
i,t j=0 j i,t−j
with mean µ =(cid:54) 0. If we denote by x ∈ Rp the tth column of the matrix X, and define the
t
empirical mean by x=p−1(cid:80)n x , then the sample covariance matrix is given by the expression
t=1 t
p−1(cid:80)n (x −x)(x −x)T instead of p−1XXT. However, by [3, Theorem A.44], the subtraction
t=1 t t
of the empirical mean does not change the LSD, and thus Theorems 1.1 and 1.2 remain valid if the
underlying linear process has a non-zero mean.
Remark 1.4. TheproofofTheorems1.1and1.2caneasilybegeneralizedtocovernon-causallinear
processes, which are defined as X =(cid:80)∞ c Z . For this case one obtains the same result
i,t j=−∞ j i,t−j
except that the autocovariance function is now given by (cid:80)∞ c c .
j=−∞ j j+|h|
Remark 1.5. If one considers a matrix X which has independent linear processes in its columns
instead of its rows, one obtains the same formulas as in Theorems 1.1 and 1.2 except that y is
replacedbyy−1. ThisisduetothefactthatXTXandXXT havethesamenon-trivialeigenvalues.
4 OLIVERPFAFFELANDECKHARDSCHLEMM
In Section 2 we proceed with the proofs of Theorems 1.1 and 1.2. Thereafter we present some
interesting examples in Section 3.
2. Proofs
InthissectionwepresentourproofsofTheorems1.1and1.2. Dealingwithinfinite-ordermoving
average processesdirectly is dfficult, andwe thereforefirst prove avariant of thesetheorems for the
truncated processes X(cid:101)i,t =(cid:80)nj=0cjZi,t−j. We define the p×n matrix X(cid:101) =(X(cid:101)i,t)it, i=1,...,p,
t=1,...,n.
Theorem 2.1. Under the assumptions of Theorem 1.1 (Theorem 1.2), the empirical spectral
distribution of the sample covariance matrix of the truncated process X(cid:101) converges, as n tends to
infinity, to a deterministic distribution with bounded support. Its Stieltjes transform is uniquely
determined by Eq. (1.6) (Eq. (1.8)).
Proof. The proof starts from the observation that one can write X(cid:101) = ZH, where Rp×2n (cid:51) Z =
(Z ) , i=1,...,p, t=1−n,...,n, and
i,t it
T
c c ... c c 0 ... 0
n n−1 1 0
.
.
(2.1) H = 0... cn ...... c...2 c...1 c0 ... 0. ∈R2n×n.
0 ... 0 c c ... ... c
n n−1 0
In particular, X(cid:101)X(cid:101)T = ZHHTZT. In order to prove convergence of the empirical spectral dis-
tribution Fp−1X(cid:101)X(cid:101)T and to obtain a characterization of the limiting distribution, it suffices, by
[16, Theorem 1], to prove that the spectral distribution FHHT of HHT converges to a non-trivial
limiting distribution. This will be done in Lemma 2.2, where the LSD of HHT is shown to be
FˆHHT = 1δ + 1FˆΓ; the distribution FˆΓ is computed in Lemma 2.3 if we impose the assumptions
2 0 2
of Theorem 1.1, respectively in Lemma 2.4 if we impose the assumptions of Theorem 1.2. Inserting
this expression for FˆHHT into equation (1.2) of [16] shows that the ESD Fp−1X(cid:101)X(cid:101)T converges, as
n→∞, almost surely to a deterministic distribution, which is determined by the requirement that
its Stieltjes transform z (cid:55)→m(z) satisfies
(2.2) 1 =−z+2y(cid:90) λ+ λ dFˆHHT =−z+y(cid:90) λ+ λ dFˆΓ.
m(z) 1+λm(z) 1+λm(z)
λ− λ−
UsingtheexplicitformulasofFˆΓ computedinLemmas2.3and2.4,oneobtainsEqs.(1.6)and(1.8).
Uniqueness of a mapping m:C+ →C+ solving Eq. (2.2) was shown in [3, p. 88]. We complete
the proof by arguing that the LSD of p−1X(cid:101)X(cid:101)T has bounded support. For this it is enough, by
[3, Theorem 6.3], to show that the spectral norm of HHT is bounded in n, which is also done in
Lemma 2.2. (cid:3)
Lemma 2.2. Let H =(cn−i+j1{0(cid:54)n−i+j(cid:54)n})ij be the matrix appearing in Eq. (2.1), and assume
thatthereexistpositiveconstantsC,δ suchthat|c |(cid:54)C(j+1)−1−δ (assumptionii)ofTheorem1.1).
j
Then the spectral norm of the matrix HHT is bounded in n. If, moreover, the spectral distribution
of the Toeplitz matrix Γ=(γ(i−j)) converges weakly to some limiting distribution FˆΓ, then the
ij
spectral distribution FHHT converges weakly, as n→∞, to 1δ + 1FˆΓ.
2 0 2
Proof. We first introduce the notation H :=HHT ∈R2n×2n as well as the block decomposition
(cid:20) (cid:21)
H H
H= 11 12 , H ∈Rn×n. We prove the second part of the lemma first. There are several
HT H ij
12 22
ways to show that the spectral distributions of two sequences of matrices converge to the same
limit. In our case it is convenient to use [3, Corollary A.41] which states that two sequences A
n
and B , either of whose empirical spectral distribution converges, have the same limiting spectral
n
distribution if n−1tr(A −B )(A −B )T converges to zero as n tends to infinity. We shall
n n n n
employ this result twice: first to show that the LSDs of H=HHT and H(cid:101) :=diag(0,H22) agree,
EIGENVALUE DISTRIBUTION OF LARGE SAMPLE COVARIANCE MATRICES OF LINEAR PROCESSES 5
and then to prove equality of the LSDs of H22 and Γ. Let ∆H = n−1tr(H−H(cid:101))(H−H(cid:101))T; a
direct calculation shows that ∆ =n−1(cid:2)trH HT +2trH HT (cid:3), and we will consider each of
H 11 11 12 12
the two terms in turn. From the definition of H it follows that the (i,j)th entry of H is given by
Hij =(cid:80)nk=1cn−i+kcn−j+k1{max(i,j)−n(cid:54)k(cid:54)min(i,j)}. The trace of the square of the upper left block
of H therefore satisfies
2
n n min(i,j)
trH11H1T1 = (cid:88) (cid:8)Hij(cid:9)2 = (cid:88) (cid:88) cn−i+kcn−j+k
i,j=1 i,j=1 k=1
n
(cid:88)
(cid:54) |c ||c ||c ||c |
i+k−1 j+k−1 i+l−1 j+l−1
i,j,k,l=1
n+1
(cid:88)
(cid:54)C4 i−1−δj−1−δl−1−δk−1−δ
i,j,k,l=2
<[Cζ(1+δ)]4 <∞,
where ζ(z) denotes the Riemann zeta function. As a consequence, the limit of n−1trH HT as n
11 11
tends to infinity is zero. Similarly, we obtain for the trace of the square of the off-diagonal block of
H the bound
2
n 2n n n+i i
trH12H1T2 =(cid:88) (cid:88) (cid:8)Hij(cid:9)2 =(cid:88) (cid:88) (cid:88) cn−i+kcn−j+k
i=1j=n+1 i=1j=n+1 k=j−n
n n n−i+1n−i+1
(cid:88)(cid:88) (cid:88) (cid:88)
(cid:54) c c c c
i+k−1 k−j i+l−1 l−j
i=1j=1 k=j l=j
n n n n
(cid:88)(cid:88)(cid:88)(cid:88)
(cid:54) |c ||c ||c ||c |
i+r+j−1 r s+j−1 s
i=1j=1r=0s=0
n+1
(cid:88)
(cid:54)C4 i−1−δr−1−δs−1−δj−1−δ
i,j,r,s=1
<[Cζ(1+δ)]4 <∞,
which shows that the limit of n−1trH HT is zero. It follows that ∆ , as defined in Lemma 2.2,
12 12 H
converges to zero as n goes to infinity, and therefore that the LSDs of H and H(cid:101) = diag(0,H22)
coincide. The latter distribution is clearly given by FH(cid:101) = 12δ0+12FH22, and we show next that the
LSD of H agrees with the LSD of Γ=(γ(i−j)) . As before it suffices to show, by [3, Corollary
22 ij
A.41], that ∆ = n−1tr(H −Γ)(H −Γ)T converges to zero as n tends to infinity. It follows
Γ 22 22
from the definitions of H and Γ that n∆ can be estimated as
Γ
2
n n ∞
(cid:88) (cid:88) (cid:88)
n∆Γ = ck−ick−j − ck−1ck+|i−j|−1
i,j=1 k=max(i,j) k=1
2
n n ∞
(cid:88) (cid:88) (cid:88)
= ck−ick−j − ck−ick−j
i,j=1 k=max(i,j) k=max(i,j)
n ∞
(cid:88) (cid:88)
= c c c c
k+i−1 k+j−1 l+i−1 l+j−1
i,j=1k,l=1
n+1 ∞
(cid:54)C4 (cid:88) (cid:88) i−1−δj−1−δk−1−δl−1−δ <[Cζ(1+δ)]4 <∞.
i,j=2k,l=2
Consequently, ∆ converges to zero as n goes to infinity, and it follows that FˆH = 1δ + 1FˆΓ.
Γ 2 0 2
6 OLIVERPFAFFELANDECKHARDSCHLEMM
In order to show that the spectral norm of H=HHT is bounded in n, we use Gerschgorin’s
circle theorem ([8, Theorem 2]), which states that every eigenvalue of H lies in at least one of the
balls B(Hii,R ) with centre Hi and radius R , i = 1,...,2n, where the radii R are defined as
i i i
Ri =(cid:80)j(cid:54)=i(cid:12)(cid:12)Hij(cid:12)(cid:12). We first note that the centres Hii satisfy
min{i,n} n
Hii = (cid:88) c2 (cid:54)(cid:88)c2 (cid:54)[Cζ(2+2δ)]2 <∞.
n−i+k k
k=max{1,i−n} k=0
To obtain a uniform bound for the radii R we first assume that i=1,...,n. Then
i
n min{i,j} 2n i
(cid:88) (cid:88) (cid:88) (cid:88)
|R |(cid:54) |c ||c |+ |c ||c |
i n−i+k n−j+k n−i+k n−j+k
j=1 k=1 j=n+1k=j−n
n 2n−i n−j
(cid:54) (cid:88) |c ||c |+ (cid:88) (cid:88)|c ||c |(cid:54)2[Cζ(1+δ)]2 <∞.
n−i+k j+k−1 k+j k
j,k=1 j=n+1−ik=0
Similarly we find that, for i=n+1,...,2n,
n j 2n n
(cid:88) (cid:88) (cid:88) (cid:88)
|R |(cid:54) |c ||c |+ |c ||c |
i n−i+k n−j+k n−i+k n−j+k
j=1k=i−n j=n+1k=max{i,j}−n
i−1 n+1−j 2n n−max{i,j}
(cid:54) (cid:88) (cid:88) |c ||c |+ (cid:88) (cid:88) |c ||c |(cid:54)3[Cζ(1+δ)]2
k+j k k k+|j−i|
j=i−n k=0 j=n+1 k=0
is bounded, which completes the proof. (cid:3)
In the following two lemmas, we argue that the distribution FˆΓ exists and we prove explicit
formulas for it in the case that the assumptions of Theorem 1.1 or Theorem 1.2 are satisfied.
Lemma 2.3. Let (c ) be a sequence of real numbers, γ : h (cid:55)→ (cid:80)∞ c c , and f : ω (cid:55)→
j j j=0 j j+|h|
(cid:80) γ(h)e−ihω. Under the assumptions of Theorem 1.1 it holds that the spectral distribution FΓ
h∈Z
of Γ=(γ(i−j)) converges weakly, as n→∞, to an absolutely continuous distribution FˆΓ with
ij
bounded support and density
1 (cid:88) 1
(2.3) g :(λ ,λ )→R+, λ(cid:55)→ .
− + 2π |f(cid:48)(ω)|
ω:f(ω)=λ
Proof. We first note that under assumption ii) of Theorem 1.1 the autocovariance function γ is
absolutely summable because
∞ ∞ ∞ ∞
(cid:88)|γ(h)|(cid:54)(cid:88)(cid:88)|c ||c |(cid:54)C2 (cid:88) h−1−δj−1−δ <[Cζ(1+δ]2 <∞.
j j+h
h=0 h=0j=0 h,j=1
Szeg˝o’s first convergence theorem ([11] and [10, Corollary 4.1]) then implies that FˆΓ exists, and
that the cumulative distribution function of the eigenvalues of the Toeplitz matrix Γ associated
with the sequence h(cid:55)→γ(h) is given by
1 (cid:90) 2π 1
(2.4) G(λ):= 2π 1{f(ω)(cid:54)λ}dω = 2π Leb({ω ∈[0,2π]:f(ω)(cid:54)λ}),
0
for all λ such that the level sets {ω ∈ [0,2π] : f(ω) = λ} have Lebesgue measure zero. By
assumption iii) of Theorem 1.1, Eq. (2.4) holds for almost all λ. In order to prove that the LSD
FˆΓ is absolutely continuous with respect to the Lebesgue measure, it suffices to prove that the
cumulative distribution function G is differentiable almost everywhere. Clearly, for ∆λ>0,
1
G(λ+∆λ)−G(λ)= Leb({ω ∈[0,2π]:λ<f(ω)(cid:54)λ+∆λ}).
2π
Due to assumption iv) of Theorem 1.1, the set of all λ∈R such that the set {ω :∈[0,2π]:f(ω)=
λ and f(cid:48)(ω)=0} is non-empty is a Lebesgue null-set. Hence it is enough to consider only λ for
EIGENVALUE DISTRIBUTION OF LARGE SAMPLE COVARIANCE MATRICES OF LINEAR PROCESSES 7
which this set is empty. Let f−1(λ)={ω :f(ω)=λ} be the pre-image of λ, which is a finite set
by assumption iii). The implicit function theorem then asserts that, for every ω ∈f−1(λ), there
exists an open interval I around ω such that f restricted to I is invertible. It is no restriction
ω ω
to assume that these I are disjoint. By choosing ∆λ sufficiently small it can be ensured that
ω
(cid:84)
the interval [λ,∆λ] is contained in f(I ), and from the continuity of f it follows that
ω∈f−1(λ) ω
(cid:83)
outside of I , the values of f are bounded away from λ, so that
ω∈f−1(λ) ω
1
lim [G(λ+∆λ)−G(λ)]
∆λ→0∆λ
1 1 (cid:91)
=2π ∆lλim→0∆λLeb {ω(cid:48) ∈Iω :λ<f(ω(cid:48))(cid:54)λ+∆λ}
ω∈f−1(λ)
1 (cid:88) 1
= lim Leb({ω(cid:48) ∈I :λ<f(ω(cid:48))(cid:54)λ+∆λ}).
2π ∆λ→0∆λ ω
ω∈f−1(λ)
In order to further simplify this expression, we denote the local inverse functions by f−1 :f(I )→
ω ω
[0,2π]. Observing that the Lebesgue measure of an interval is given by its length, and that the
derivatives of f−1 are given by the inverse of the derivative of f, it follows that
ω
lim 1 [G(λ+∆λ)−G(λ)]= 1 (cid:88) lim 1 (cid:12)(cid:12)f−1(λ+∆λ)−f−1(λ)(cid:12)(cid:12)
∆λ→0∆λ 2π ∆λ→0∆λ ω ω
ω∈f−1(λ)
(cid:12) (cid:12)
= 1 (cid:88) (cid:12)(cid:12) d f−1(λ)(cid:12)(cid:12)
2π (cid:12)dλ ω (cid:12)
ω∈f−1(λ)
1 (cid:88) 1
= .
2π |f(cid:48)(ω)|
ω∈f−1(λ)
This shows that G is differentiable almost everywhere with derivative g :λ(cid:55)→ 1 (cid:80) 1 .
2π ω∈f−1(λ) |f(cid:48)(ω)|
It remains to argue that the support of FˆΓ is bounded. The absolute summability of γ(·) implies
boundedness of its Fourier transform f. The claim then follows from Eq. (2.4), which shows that
the support of g is equal to the range of f. (cid:3)
Lemma 2.4. Let f :ω (cid:55)→(cid:80)k α 1 (ω) be the piecewise constant spectral density of the linear
j=1 j Aj
process X = (cid:80)∞ c Z , and denote the corresponding autocovariance function by γ : h (cid:55)→
t j=0 j t−j
(cid:80)∞ c c . Under the assumptions of Theorem 1.2 it holds that the spectral distribution FΓ of
j=0 j j+|h|
Γ=(γ(i−j)) converges weakly, as n→∞, to the distribution FˆΓ =(2π)−1(cid:80)k |A |δ .
ij j=1 j αj
Proof. Without loss of generality we may assume that 0 < α < ... < α . As in the proof of
1 k
Lemma 2.3 one sees that FˆΓ exists, and that FˆΓ(−∞,λ) is given by
k
1 (cid:91)
G(λ):= Leb({ω ∈[0,2π]:f(ω)(cid:54)λ}), ∀λ∈[0,2π]\ {α }.
2π j
j=1
The special structure of f thus implies that G(λ) = (2π)−1(cid:80)kλ |A |, where k is the largest
j=1 j λ
integer such that α (cid:54)λ. Since G must be right-continuous, this formula holds for all λ in the
kλ
interval [0,2π]. It is easy to see that the function G is the cumulative distribution function of the
discrete measure (2π)−1(cid:80)k |A |δ , which completes the proof. (cid:3)
j=1 j αj
Proof. ofTheorems1.1and1.2ItisonlylefttoshowthatthetruncationperformedinTheorem2.1
does not alter the LSD, i.e. that the difference of Fp−1XXT and Fp−1X(cid:101)X(cid:101)T converges to zero almost
surely. By [3, Corollary A.42], this means that we have to show that
1 1
(2.5) tr(XXT +X(cid:101)X(cid:101)T) tr((X−X(cid:101))(X−X(cid:101))T)
p2 p2
(cid:124) (cid:123)(cid:122) (cid:125)(cid:124) (cid:123)(cid:122) (cid:125)
=I =II
8 OLIVERPFAFFELANDECKHARDSCHLEMM
converges to zero. To this end we show that I has a limit, and that II converges to zero, both
almost surely. By the definition of X and X(cid:101) we have
p n ∞ ∞
1 (cid:88)(cid:88) (cid:88) (cid:88)
II= c c Z Z .
p2 k m i,t−k i,t−m
i=1 t=1k=n+1m=n+1
We shall prove that the variances of II are summable. For this purpose we need the following two
estimateswhichareimpliedbytheCauchy–Schwarzinequality,theassumptionthatσ =sup EZ4
4 i,t i,t
is finite, and the assumed absolute summability of the coefficients (c ) :
j j
p n ∞ (cid:32) ∞ (cid:33)2
(cid:88)(cid:88) (cid:88) (cid:88)
(2.6a) E |c c Z Z |(cid:54)pn |c | <∞,
k m i,t−k i,t−m k
i=1 t=1k,m=1 k=1
p n ∞
(cid:88) (cid:88) (cid:88)
(2.6b) E |c c c c Z Z Z Z |
k m k(cid:48) m(cid:48) i,t−k i,t−m i(cid:48),t(cid:48)−k(cid:48) i(cid:48),t(cid:48)−m(cid:48)
i,i(cid:48)=1=1t,t(cid:48)=1k,k(cid:48),m,m(cid:48)=1
(cid:32) ∞ (cid:33)4
(cid:88)
(cid:54)(np)2σ |c | <∞.
4 k
k=1
Therefore we can, by Fubini’s theorem, interchange expectation and summation to bound the
variance of II as
p n ∞
1 (cid:88) (cid:88) (cid:88)
Var(II)(cid:54) c c c c E(Z Z Z Z ).
p4 k m k(cid:48) m(cid:48) i,t−k i,t−m i(cid:48),t(cid:48)−k(cid:48) i(cid:48),t(cid:48)−m(cid:48)
i,i(cid:48)=1t,t(cid:48)=1 k,k(cid:48)
=n+1
m,m(cid:48)
Considering separately the terms where i=i(cid:48) and i(cid:54)=i(cid:48), we can write
p n ∞
1 (cid:88) (cid:88) (cid:88)
Var(II)(cid:54) c c c c E(Z Z Z Z )
p4 k m k(cid:48) m(cid:48) i,t−k i,t−m i(cid:48),t(cid:48)−k(cid:48) i(cid:48),t(cid:48)−m(cid:48)
i,i(cid:48)=1t,t(cid:48)=1 k,k(cid:48)
=n+1
i(cid:54)=i(cid:48) m,m(cid:48)
p n ∞
1 (cid:88) (cid:88) (cid:88)
+ c c c c E(Z Z Z Z ).
p4 k m k(cid:48) m(cid:48) i,t−k i,t−m i,t(cid:48)−k(cid:48) i,t(cid:48)−m(cid:48)
i=1t,t(cid:48)=1 k,k(cid:48)
=n+1
m,m(cid:48)
For the expectation in the first sum not to be zero, k must equal m and k(cid:48) must equal m(cid:48), in which
case its value is unity. The expectation in the second term can always be bounded by σ , so that
4
we obtain
p2−p (cid:32) (cid:88)∞ (cid:33)2 pn2 (cid:32) (cid:88)∞ (cid:33)4
Var(II)(cid:54) n2 c2 +σ |c | .
p4 k 4 p4 k
k=n+1 k=n+1
Due to Eq. (1.1) and the assumed polynomial decay of c there exists a constant K such that the
k
right hand side is bounded by Kn−1−4δ, which implies that
∞ ∞
(cid:88) (cid:88)
Var(II)(cid:54)K n−1−4δ <∞,
n=1 n=1
and therefore, by the first Borel–Cantelli lemma, that II converges to a constant almost surely. In
order to show that this constant is zero, it suffices to shows that the expectation of II converges
to zero. Since EZ = 0, and the {Z } are independent, one sees, using Eq. (2.6a) and again
i,t i,t
Fubini’s theorem, that E(II) = np−1(cid:80)∞ c2, which converges to zero because the {c } are
k=n+1 k k
square-summable.
We now consider factor I of expression (2.5) and define ∆X =XXT −X(cid:101)X(cid:101)T. Then
1 1
(2.7) I= p2 tr(∆X)+2p2 tr(X(cid:101)X(cid:101)T).
(cid:124) (cid:123)(cid:122) (cid:125) (cid:124) (cid:123)(cid:122) (cid:125)
=Ia =Ib
EIGENVALUE DISTRIBUTION OF LARGE SAMPLE COVARIANCE MATRICES OF LINEAR PROCESSES 9
Because of
n n ∞ ∞
(cid:88) (cid:88)(cid:88) (cid:88)
(XXT) = X2 = c c Z Z ,
ii i,t k m i,t−k i,t−m
t=1 t=1k=0m=0
and similarly (X(cid:101)X(cid:101)T)ii =(cid:80)nt=1(cid:80)nk=0(cid:80)nm=0ckcmZi,t−kZi,t−m, we have that
p
(cid:88)(cid:104) (cid:105)
tr(∆X)= (XXT)ii−(X(cid:101)X(cid:101)T)ii
i=1
p n ∞ ∞
(cid:88)(cid:88) (cid:88) (cid:88)
= c c Z Z
k m i,t−k i,t−m
i=1 t=1k=n+1m=n+1
(cid:124) (cid:123)(cid:122) (cid:125)
=II→0a.s.
p n ∞ n
(cid:88)(cid:88) (cid:88) (cid:88)
(2.8) +2 c c Z Z .
k m i,t−k i,t−m
i=1 t=1k=n+1m=1
Equation (2.6b) allows us to apply Fubini’s theorem to compute the variance of the second term in
the previous display as
p n ∞ n
4 (cid:88) (cid:88) (cid:88) (cid:88)
c c c c E(Z Z Z Z ),
p4 k m k(cid:48) m(cid:48) i,t−k i,t−m i(cid:48),t(cid:48)−k(cid:48) i(cid:48),t(cid:48)−m(cid:48)
i,i(cid:48)=1t,t(cid:48)=1k,k(cid:48)=n+1m,m(cid:48)=1
which is, by the same reasoning as we did for II, bounded by
(cid:32) ∞ (cid:33)2(cid:32) n (cid:33)2
p (cid:88) (cid:88)
4σ n2 |c | |c | (cid:54)Kn−1−2δ,
4p4 k m
k=n+1 m=1
for some positive constant K. Clearly, this is summable in n. Having, by Eq. (2.6a), expected
value zero, the second term of Eq. (2.8) and, therefore, also tr(∆ ) both converge to zero almost
X
surely. Thus, we only have to look at the contribution of I in expression (2.7). From Theorem 2.1
b
we know that Fp−1X(cid:101)X(cid:101)T converges almost surely weakly to some non-random distribution Fˆ with
bounded support. Hence, denoting by λ1,...,λp the eigenvalues of p−1X(cid:101)X(cid:101)T,
Ib = p1tr(cid:18)p1X(cid:101)X(cid:101)T(cid:19)= p1(cid:88)p λi =(cid:90) λdFp1X(cid:101)X(cid:101)T→(cid:90) λdFˆ <∞,
i=1
almost surely. It follows that, in Eq. (2.5), factor I is bounded, and factor II converges to zero, and
so the proof of Theorems 1.1 and 1.2 is complete. (cid:3)
3. Illustrative examples
For several classes of widely employed linear processes, Theorem 1.1 can be used to obtain
an explicit description of the limiting spectral distribution. In this section we consider the class
of autoregressive moving average (ARMA) processes as well as fractionally integrated ARMA
models. The distributions we obtain in the case of AR(1) and MA(1) processes can be interpreted
as one-parameter deformations of the classical Marchenko–Pastur law.
3.1. Autoregressive moving average processes. Givenpolynomials a:z (cid:55)→1+a z+...a zp
1 p
and b : z (cid:55)→ 1+b z +...+b zq, an ARMA(p,q) process X with autoregressive polynomial a
1 q
and moving average polynomial b is defined as the stationary solution to the stochastic difference
equation
X +a X +...+a X =Z +b Z +...+b Z , t∈Z.
t 1 t−1 p t−p t 1 t−1 q t−q
If the zeros of a lie outside the closed unit disk, it is well known that X has an infinite-order
moving average representation X = (cid:80)∞ c Z , where {c } are the coefficients in the power
t j=0 j t−j j
series expansion of b(z)/a(z) around zero. It is also known ([6]) that there exist positive constants
ρ<1 and K such that |c |(cid:54)Kρj, so that assumption ii) of Theorem 1.1 is satisfied. While the
j
10 OLIVERPFAFFELANDECKHARDSCHLEMM
0.5 0.3 0.2
ϑ=.3 ϑ=.3 ϑ=.3
0.4 ϑ=.5 ϑ=.5 ϑ=.5
ϑ=.7 0.2 ϑ=.7 ϑ=.7
0.3
λp()0.2 λp() λp()0.1
0.1
0.1
0 0 0
0 5 10 0 5 10 15 0 5 10 15 20
λ λ λ
(a) y=1 (b) y=3 (c) y=5
Figure1. Limitingspectraldensitiesλ(cid:55)→p(λ)ofp−1XXT fortheMA(1)process
X =Z +ϑZ for different values of ϑ and y =n/p
t t t−1
0.4 0.3 0.2
φ=.3 φ=.3 φ=.3
0.3 φ=.5 φ=.5 φ=.5
φ=.7 0.2 φ=.7 φ=.7
p()λ0.2 p()λ p()λ0.1
0.1
0.1
0 0 0
0 5 10 15 20 0 10 20 30 0 10 20 30
λ λ λ
(a) y=1 (b) y=3 (c) y=5
Figure 2. Limitingspectraldensitiesλ(cid:55)→p(λ)ofp−1XXT fortheAR(1)process
X =ϕX +Z for different values of ϕ and y =n/p
t t−1 t
autocovariance function of a general ARMA process does not in general have a simple closed form,
its Fourier transform is given by
(cid:12)(cid:12)b(cid:0)eiω(cid:1)(cid:12)(cid:12)2
(3.1) f(ω)=(cid:12) (cid:12) , ω ∈[0,2π].
(cid:12)a(eiω)(cid:12)
(cid:12) (cid:12)
Sincef isrational,assumptionsiii)andiv)ofTheorem1.1aresatisfiedaswell. Inordertocompute
the LSD of Γ, it is necessary, by Lemma 2.3, to find the roots of a trigonometric polynomial of
possibly high degree, which can be done numerically.
We now consider the special case of the ARMA(1,1) process X =ϕX +Z +ϑZ , |ϕ|<1,
t t−1 t t−1
for which one can obtain explicit results. By Eq. (3.1), the spectral density of X is given by
1+ϑ2+2ϑcosω
f(ω)= , ω ∈[0,2π].
1+ϕ2−2ϕcosω
Equation (2.3) implies that the LSD of the autocovariance matrix Γ has a density g, which is given
by
1 (cid:88) 1
g(λ)=
2π |f(cid:48)(ω)|
ω∈[0,2π]:f(ω)=λ
1
= 1 (λ),
π(ϑ+ϕλ)(cid:112)[(1+ϑ)2−λ(1−ϕ)2][λ(1+ϕ)2−(1−ϑ)2] (λ−,λ+)
where
(1±ϑ)2
λ =min(λ−,λ+), λ =max(λ−,λ+), λ± = .
− + (1∓ϕ)2
