Resampling in Time Series Models

0.37 MB·English

by  Abhishek Bhattacharya

Project Report on Resampling in Time Series Models
Submitted by: Abhishek Bhattacharya
Project Supervisor: Arup Bose
Abstract
2 ThisprojectrevolvesaroundstudyingestimatorsforparametersindifferentTimeSeriesmod-
1 elsandstudyingtheirassymptoticproperties. Weintroducevariousbootstraptechniquesforthe
0 estimators obtained. Our special emphasis is on Weighted Bootstrap. We establish the con-
2 sistency of this scheme in a AR model and its variations. Numerical calculations lend further
supporttoourconsistencyresults. NextweanalyzeARCHmodels,andstudyvariousestimators
n
a used for different error distributions. We also present resampling techniques for estimating the
J distribution of the estimators. Finally by simulating data, we analyze the numerical properties

5 of theestimators.

]
T 1 Bootstrap in AR(1) model
S
. Let X be a stationary AR(1) process, that is,
h t
t
a Xt =θXt 1+Zt for t=1,2,... (1)
m −
Z iid (0,σ2); EZ4 < ; θ <1.
t t ∞ | |
[
Wehaveassumedσ tobeknown,andθ istheunknownparameterofinterest. ThentheLeastSquares

1 estimate for θ (which is approximately the MLE in case of normal errors) is given by
v
6 n X X
6 θˆn = t=n2 Xt 2t−1
1 P t=2 t 1
−
1 Then it can be established that P
.
1 √n(θˆ θ) d N(0,(1 θ2)) (2)
0 n− −→ −
2 Letus introduce two particularbootstraptechniques speciallyused to estimate the distributionofθˆ
n
1
from a realization of model (1).
:
v
Xi (a) Residual Bootstrap Let Z˜t = Xt θˆnXt 1, t = 2,3,...,n and let Zˆt be the standardized
version of Z˜ such that 1 Zˆ = 0 an−d 1 − Zˆ2 = 1. Now we draw Z , t = 1,2,...,N with
r t n 1 t n 1 t t∗
a replacement from Zˆ and−define −
t P P
X = Z
1∗ 1∗
X = θˆ X +Z ,t=2,...,N.
t∗ n t∗ 1 t∗
−
and form the statistic
n X X
θˆn∗ = tn=2(Xt∗ t∗−)21 (3)
Pt=2 t∗ 1
−
Then(3)formsanestimatorofθˆn andiscalledPthe Residual Bootstrap estimator. We repeatthe
simulation process several times to estimate the distribution of θˆ .
n∗
(b) Weighted Bootstrap Alternatively we define our resampling estimator
n w X X
θˆn∗ = nt=2wnt(Xt t−)12 (4)
Pt=2 nt t−1
where wnt;1 t n,n 1 is a triangulPar sequence of random variables, independent of Xt .
{ ≤ ≤ ≥ } { }
These are the so called “Bootstrap weights”, and the estimator (4) is the Weighted Bootstrap
Estimator.
1
1.1 A Bootstrap Central limit theorem
Undersuitableconditionsontheweightstobestatedbelow,weestablishthedistributionalconsistency
of the Weighted Bootstrap Estimator, θˆ defined in (4). To establish consistency, we will prove a
n∗
Bootstrap CLT for which we will need the following established results:
Result 1 (P-W theorem; see Praestgaard and Wellner(1993)) Let
c ; j =1,2,...,n; n 1 be a triangular array of constants, and let U j =1,2,...,n; n 1
nj nj
{ ≥ } { ≥ }
be a triangular array of row exchangeable random variables such that as n ,
→∞
1. 1 n c 0
n j=1 nj →
2. 1 Pn c2 τ2
n j=1 nj →
3. 1mPax c2 0
n 1≤j≤n nj →
4. E(U )=0 j =1,2,...,n. n 1
nj
≥
5. E(U2 )=1 j =1,2,...,n. n 1
nj ≥
6. 1 n U2 P 1
n j=1 nj →
P
7. lim limsup (E(U2 I )=0
k→∞ n→∞ nj {|Unj|>k}
q
Then under the above conditions,
n
1
c U d N(0,τ2) (5)
nj nj
√n −→
j=1
X
Result (1) can be generalized by taking c random variables, independent of U and the con-
nj nj
{ } { }
ditions (1), (2) and(3) replacedby convergencein probability. In that caseconclusion(5) is replaced
by
n
1
P c U C c ;j =1,...,n;n 1 P[Y C]=o (1) (6)
nj nj nj P
√n ∈ { ≥ }− ∈
j=1
X
 
where Y N(0,τ2) and C (R) such that P(Y ∂C)=0.
∼ ∈B ∈
Result 2 Let X ,X ,...,X be the realization of the stationary AR(1) process (1). Then
1 2 n
{ }
n1 tn=−1kXtaZtb+k a→.s. E(XtaZtb+k) whenever EZtmax(a,b) < ∞ ∀a,b,k ∈ Z+; a,b ≥ 0 ; k > 0. This
can be established using the Martingale SLLN; see Hall and Heyde 1980.
P
Letus use the notationsP , E , V to respectivelydenote probabilities,expectations andvariances
B B B
with respect to the distribution of the weights, conditioned on the given data X ,...,X . The
1 n
{ }
weights are assumed to be row exchangeable. We henceforth drop the first suffix in the weights w
ni
and denote it by w . Let σ2 = V (w ), W = σ 1(w 1). The following conditions on the row
i n B i i n− i −
exchangeable weights are assumed:
A1. E (w )=1
B 1
A2. 0<k <σ2 =o(n)
n
A3. c =Cov(w ,w )=O(n 1)
1n 1 2 −
A4. Conditions of Result(1) hold with U =W .
nj nj
2
Theorem 1 Under the conditions (A1)-(A4) on the weights,
P √nσ 1(θˆ θˆ ) xX ,...,X P [Y x]=o (1) x R (7)
B n− n∗ − n ≤ | 1 n − ≤ P ∀ ∈
h i
where Y N(0,(1 θ2)).
∼ −
Proof Note that
n w X X
θˆn∗ = t=n2 wt Xt 2t−1
P t=2 t t 1
n w X −(θX +Z )
= Pt=2 t nt−1w X2t−1 t
P t=2 t t 1
−
w X Z
= θ+ Pt t−1 t
w X2
P t t 1
−
Similarly P
X X X Z
θˆn = Xt 2t−1 =θ+ Xt−1 t
P t−1 P t−1
Hence P P
w X Z X Z
θˆn∗ −θˆn = wt Xt−21 t − Xt−21 t
P t t 1 P t 1
− −
w X Z X Z X Z X Z
= P t t−1 t P t−1 t + t−1 t t−1 t
w X2 − w X2 w X2 − X2
P t t 1 P t t 1 P t t 1 P t 1
− − − −
(w 1)X Z X Z (w 1)X2
= P t− t−1Pt t−1Pt t− tP−1
w X2 − X2 w X2
P t t 1 P t P1 t t 1
− − −
Now using Result (2), P P P
X Z
t 1 t a.s.
− E(Xt 1Zt) = 0 (8)
n −→ −
P
X2 Z2
P nt−1 t −a.→s. E(Xt2−1Zt2)
= σ4(1 θ2) 1 (9)
−
−
Claim 1. For τ2 = σ4(1 θ2) 1,
−
−
n
1 x
P W X Z x X ,...,X P Φ( ) x R
B t t 1 t 1 n
" √n − ≤ #−→ τ ∀ ∈
t=2
X
To see this let us verify the conditions of Result(1) with c = X Z and U = W for j =
nj j j+1 nj j
1,...,n 1.
−
1. 1 n X Z P 0
n t=2 t−1 t −→
FoPllows from (8).
2. 1 n X2 Z2 P σ4(1 θ2) 1(=τ2)
n t=2 t−1 t → − −
FoPllows from (9).
3. n 1max(X2 Z2) P 0
− t−1 t −→
3
Proof Let Y =X2 Z2 =X2X2 2θX X3 +θ2X4
t t−1 t t t−1− t t−1 t−1
Then given ǫ>0,
P(n 1maxY >ǫ) = P(maxY >nǫ)
− t t
n EY2 1
P(Y >nǫ) t = EY2 0
≤ t ≤ n2ǫ2 nǫ2 t −→
t=1
X X
as EY2 = E(X4 Z4)<
t t−1 t ∞
Conditions (4), (5), (6) and (7) follow from definition and condition on the weights. This proves
the claim.
Hence for τ2 =σ4(1 θ2) 1
−
−
n
1 x
P W X Z x X ,...,X P Φ( ) x R (10)
t t 1 t 1 n
" √n − ≤ #−→ τ ∀ ∈
t=2
X
Claim 2. With c=σ2(1 θ2) 1,
−
−
n
1
P w X2 c > ǫ P 0 ǫ>0
B" n t t−1− #−→ ∀
t=2
X
Proof
1 1
E ( w X2 ) = X2
B n t t−1 n t−1
V (Xw X2 ) = XX4 σ2 + X2 X2 Cov(w ,w )
B t t 1 t 1 n t 1 s 1 t s
− − − −
s=t
X X X6
= σ2 X4 +c X2 X2
n t 1 1n t 1 s 1
− − −
s=t
X X6
Therefore
1 σ2 c
V ( w X2 ) = n X4 + 1n X2 X2 (11)
B n t t−1 n2 t−1 n2 t−1 s−1
s=t
X X X6
1
σ2 0
n n →
1
X4 a.s. E(X4)
n t−1 −→ t
X
a.s.
Hence the first term in (11) 0
−→
Also
1 X2 2
X2 X2 t a.s. (EX2)2
n2 t−1 s−1 ≤ n −→ t
s=t (cid:18)P (cid:19)
X6
Hence 1 X2 X2 is bounded a.s., and as c 0,
n2 s6=t t−1 s−1 1n →
the second term in (11) also 0 a.s.
P −→
This shows that V 1 w X2 0 a.s.
B n t t−1 −→
Hence 1 w X2 -(cid:0) 1P X2 P(cid:1)B 0 a.s.
n t t−1 n t−1 −→
P P
4
Using Result (2), 1 X2 a.s. E(X2) = σ2(1 θ2) 1
n t−1 −→ t − −
This implies, 1 wPX2 PB σ2(1 θ2) 1 a.s.
n t t−1 −→ − −
This proves Claim 2.
P
In fact we have proved that, with c=σ2(1 θ2) 1
−
−
n
1
P w X2 c > ǫ a.s. 0 ǫ>0. (12)
B" n t t−1− #−→ ∀
t=2
X
Now
√nσn−1(θˆn∗ −θˆn)
(w 1)X Z X Z (w 1)X2
= √nσn−1 t−w X2t−1 t −√nσn−1 t−X1 2t wt−X2 t−1
P t t 1 P t P1 t t 1
− − −
W X Z /√n (w 1)X2 /n
= tw Xt−21Pt/n −√n(θˆn−θ)σn−1 Ptw−X2Pt/−n1 (13)
P t t 1 P t t 1
− −
= T T (say)
1P− 2 P
Then from (10) and (12), P (T x) P(T x)=o (1), where,
B 1 P
≤ − ≤
1
T N(0,σ4(1 θ2) 1) = N(0, (1 θ2)) (14)
−
∼ σ2(1 θ2) 1 − −
− −
Claim 3. Define A √n(θˆ θ)σ 11 (w 1)X2 .
≡ n− n− n t− t−1
P
P
Then ǫ>0, P ( A >ǫ) 0.
B
∀ | | −→
Proof Note that,
E (A) = 0 (15)
B
n σ2 c
V (A) = (θˆ θ)2 [ n X4 + 1n X2 X2 ] (16)
B σn2 n− n2 t−1 n2 s = t s−1 t−1
X XX6
X4 nc X2 X2
= (θˆn−θ)2 nt−1 + σ21n(θˆn−θ)2 s 6= tn2s−1 t−1 (17)
P n P P
= A +A (say) (18)
1 2
PXt4−1 converges a.s., and from (2), (θˆ θ) P 0, as a result, A P 0.
n n− −→ 1 −→
Moreover Ps6=tXs2−1Xt2−1 is bounded a.s., nc is bounded and σ2 is bounded away from 0. As a
n2 1n n
P
result A 0.
2
−→
P
Combining, V (A) 0.
B
−→
Hence
V (A)
B P
P ( A >ǫ) 0
B | | ≤ ǫ2 −→
A
Now T = .
2 w X2 /n
t t 1
−
From (12), we have, wtXt2−1/n is bounded awaPy from zero in PB a.s., which means that, ∀ǫ>0,
P
5
P ( T >ǫ) = o (1) (19)
B 2 P
| |
Hence from (13), (14) and (19), we have,
PB[√nσn−1(θˆn∗ θˆn) x] P[Y x]=oP(1) x R (20)
− ≤ − ≤ ∀ ∈
where Y N(0, (1 θ2)) and this was what was to be proved.
∼ −
1.2 Least Absolute Deviations Estimator
Another estimator of θ can be the LAD estimatior, that is,
0
n
1
θˆ =argmin X θX
2 t t 1
θ n − −
2
X
Now we reparametrize the model (1) in such a way that the median of Z , instead of the mean is
t
equal to 0, while VZ =σ2 remains unchanged.
t
1.3 Distributional Consistency of the LAD estimator
Under the following assumptions we establish the assymptotic normality of θˆ.
2
A1. CDF of Z , F has a pdf f, which is continuous at zero.
t
A2. F(x) F(0) xf(0) cx1+α inaneighborhoodofzero,say x M,wherec, α,M >0.
− − ≤ | | | |≤
To do so we use the the following result on random convex functions.
Result 3 (See Niemire (1992)) Suppose that h (a), a Rd is a sequence of random convex func-
n
∈
tions which converge in probability to h(a) for every fixed a. Then this convergence is uniform on any
compact set containing a.
Theorem 2 Under the conditions (A1)-(A2), √n(θˆ θ ) d N(0, 1 ) as n .
2 − 0 −→ 4f2(0)EXt2 →∞
Proof Define
f(X ,θ) = (X θX X )
t t t 1 t
| − − | − | |
g(X ,θ) = X [2I(Z (θ) 0) 1]
t t 1 t
− ≤ −
whereZ (θ) = X θX fort=2,...,n.
t t t 1
− −
Y (a) = f(X ,θ +n 1/2a) f(X ,θ ) n 1/2ag(X ,θ )
t t 0 − t 0 − t 0
− −
= Z n 1/2aX Z n 1/2aX [2I(Z 0) 1] for α R.
t − t 1 t − t 1 t
| − − | − | | − − ≤ − ∈
Also define
Q (θ) = f(X ,θ)
n t
U = Xg(X ,θ )
n t 0
V = XY (a)=Q (θ +n 1/2a) Q (θ ) n 1/2aU
n t n 0 − n 0 − n
− −
X
Step1 n Y (a) P a2f(0)EX2
t=2 t −→ 1
P
Step1.1 (Y E(Y )) 0
P t − t|At−1 −→
P
6
E(Y E(Y )) = 0
t t t 1
− |A−
V( Y E(Y )) = V(Y E(Y )) V(Y ) EY2
t − t|At−1 t − t|At−1 ≤ t ≤ t
X X X X
By convexity of f,
0 Y (a) n 1/2a[g(X ,θ +n 1/2a) g(X ,θ )]
t − t 0 − t 0
≤ ≤ −
Therefore
a2
E(Y2) E[g(X ,θ +n 1/2a) g(X ,θ )]2
t ≤ n t 0 − − t 0
a2
= 4 EX2 [I(Z n 1/2aX 0) I(Z 0)]2
n t−1 t− − t−1 ≤ − t ≤
Now
EY2 =nEY2 4aEX2[I(Z n 1/2aX 0) I(Z 0)]2
t 2 ≤ 1 2− − 1 ≤ − 2 ≤
which tends to zeXro using DCT. Therefore
V( (Y E(Y )) 0
t t t 1
− |A− →
X
This establishes Step 1.1.
Step1.2 E(Y ) a2f(0)EX2 P 0
tAt−1 − 1 −→
P E(Y ) = E(Z n 1/2aX ) E Z
t t 1 t − t 1 t 1 t
|A− | − − |A− − | |
= (z n 1/2aX z )dF(z)
− t 1
| − − |−| |
Z
Using the representation,
θ
x θ x =θ[2I(x 0) 1]+2 [I(x s) I(x 0)]ds
| − |−| | ≤ − ≤ − ≤
Z0
we have
n−1/2aXt−1
z n−1/2aXt 1 z =n−1/2aXt 1[2I(z 0) 1]+2 [I(z s) I(z 0)]ds
| − − |−| | − ≤ − Z0 ≤ − ≤
Therefore
n−1/2aXt−1
E(Y ) = n 1/2aX [2I(z 0) 1]dFz + 2 [I(z s) I(z 0)]dsd(F21z)
t t 1 − t 1
|A− − Z ≤ − Z Z0 ≤ − ≤
n−1/2aXt−1
= 2 [F(s) F(0)]ds (22)
−
Z0
a
= 2n 1/2X [F(n 1/2X x) F(0)]dx (23)
− t 1 − t 1
− Z0 − −
Under assumption A2,
F(n 1/2X x) F(0) = n 1/2X xf(0)+R (x)
− t 1 − t 1 nt
− − −
where R (x) cn (1+α)/2 X 1+α x1+α
nt − t 1
| | ≤ | − | | |
whenevern 1/2 X x M
− t 1
| − || | ≤
7
Hence
a
E(Yt t 1) = 2n−1/2Xt 1 [n−1/2Xt 1xf(0)+Rnt(x)]dx
|A− − Z0 −
1 a
= X2 a2f(0)+2n 1/2X R (x)dx
n t−1 − t−1Z0 nt
1 2 a
E(Y ) = a2f(0) X2 + X √nR (x)dx
t|At−1 n t−1 n t−1Z0 nt
X X X
= I +I (say)
1 2
Then I P a2f(0)EX2.
1 −→ 1
P
Remains to show I 0. To show this, let us assume:
2
−→
1. max n 1/2 X P 0
1 t n − t 1
≤ ≤ | − |−→
2. 1 X 2+α P 0
n1+α/2 | t−1| −→
Hence givePn ǫ>0,
P(max n 1/2 X M/a) 1
− t 1
c | − |≤ | | →
andP( X 2+α <ǫ) 1
n1+α/2 | t−1| →
X
Let A be the set where maxn 1/2 X M/a and c X 2+α <ǫ.
n − | t−1|≤ | | n1+α/2 | t−1|
Then N such that P(A )>1 ǫ n N. Then on A , R cn α/2 X 1+α, and hence
∃ n − ∀ ≥ n | nt|P≤ − | t−1|
2 a
I X cn α/2 X 1+α
2 t 1 − t 1
| | ≤ n | − |Z0 | − |
Xc
X 2+α
≤ n1+α/2 | t−1|
< ǫ X
P
ie P(I < ǫ) 1 ǫ > 0. In otherwords I 0. This completes Step1.2 and hence Step1. In
2 2
| | → ∀ −→
other words,
Q (θ +n 1/2a) Q (θ ) n 1/2aU a2f(0)EX2 P 0 (24)
n 0 − − n 0 − − n− 1 −→
Due to convexity of Q , the convergence in (24) is uniform on any compact set by Result3. Thus
n
∀
ǫ>0, and M >0, for n sufficiently large, we have
P sup Q (θ +n 1/2a) Q (θ ) n 1/2aU a2f(0)EX2 <ǫ 1 ǫ/2
"a M n 0 − − n 0 − − n− 1 #≥ −
| |≤ (cid:12) (cid:12)
(cid:12) (cid:12)
Call (cid:12) (cid:12)
An(a) = Qn(θ0+n−1/2a) Qn(θ0) ,
−
B (a) = n 1/2aU +a2f(0)EX2
n − n 1
and their minimizers a and b respectively. Then
n n
a = √n(θˆ θ )and
n 2 0
−
b = (2f(0)EX2) 1n 1/2U
n − 1 − − n
The minimum value of B ,
n
B (b )= n 1(4f(0)EX2) 1U2
n n − − 1 − n
8
Note that b is bounded in probability. Hence there exists M >0 such that
n
P (2f(0)EX2) 1n 1/2U <M 1 1 ǫ/2
|− 1 − − n| − ≥ −
h i
Let A be the set where,
sup A (a) B (a) <ǫ
n n
| − |
a M
| |≤
and |−(2f(0)EX12)−1n−1/2Un|<M −1
Then P(A)>1 ǫ. On A,
−
A (b )<B (b )+ǫ (25)
n n n n
Consider the value of A on the sphere S = a: a b =kǫ1/2 where k will be chosen later. By
n n n
{ | − | }
chosing ǫ sufficiently small, we have a M a S . Hence
n
| |≤ ∀ ∈
A (a)>B (a) ǫ a S . (26)
n n n
− ∀ ∈
Once we chose k =2(2f(0)EX2) 1/2,
1 −
B (a)>B (b )+2ǫ a S (27)
n n n n
∀ ∈
Comparingthebounds (25)and(26),wehaveA (a)>A (b )whenevera S . If a b >kǫ1/2,
n n n n n n
∈ | − |
by convexity of A , thereexists a on S such that A (a ) A (b ) which cannot be the case.
n ∗n n n ∗n ≤ n n
Therefore a b < kǫ1/2 on A. Since this holds with probability atleast 1 ǫ and ǫ is arbitrary,
n n
| − | −
P
this proves that a b 0. In otherwords,
n n
| − |−→
√n(θˆ2−θ0)=−n−1/2(2f(0)EX12)−1Un+oP(1) (28)
Step 2 n 1/2U d N(0,EX2)
− n −→ 1
n
U = X [2I(Z 0) 1]
n t 1 t
− ≤ −
t=2
X
n
= Y (say)
t
t=2
X
Then note that U is a 0-meanmartingale with finite variance increments. Hence to prove Step2, we
n
use the Martingale CLT. Write
n n
S2 = E(Y2 )= X2
n t |At−1 t−1
t=2 t=2
X X
ands2 = ES2 =(n 1)EX2
n n − 1
Then we need to to verify:
1. Sn2 P 1
s2n −→
This follows from Result 2.
2. s 2 n E(Y2I(Y ǫs )) 0 asn ǫ>0.
n− t=2 t | t|≥ n −→ →∞ ∀
To see this, note that
P
1 X
L.H.S. = E(X2I | 1| ǫ√n 1 )
EX12 1 EX12 ≥ − !
−→ 0asEX12 <∞p
9
Hence using Result 4, we have Un d N(0,1), which proves Step2.
sn −→
Combining Step2 and equation(28), we get,
1
√n(θˆ θ ) d N 0,
2− 0 −→ 4f2(0)EX2
(cid:18) 1(cid:19)
and this was what was to be proved.
Finally it remains to verify:
1. max n 1/2 X P 0
2 t n − t 1
≤ ≤ | − |−→
Proof: Given ǫ positive,
n 1
−
P(maxn 1/2 X >ǫ) P(X >ǫ√n)
− t 1 t
t | − ≤ | |
t=1
X
= (n 1)P(X >ǫ√n)
1
− | |
= (n 1) I(X >ǫ√n)dP
1
− | |
Z
X 2
(n 1) | 1| I(X >ǫ√ndP
≤ − ǫ2n | 1|
Z
1
= X2I(X >ǫ√n)dP
ǫ2 | 1 | 1|
Z
0asE X 2 <
1
→ | | ∞
2. 1 n X 2+α P 0
n1+α/2 t=2| t−1| −→
Proof: P
1 n max X α 1
n1+α/2 |Xt−1|2+α ≤ 1≤nt1≤+nα−/12| t| n Xt2−1
t=2
X X
max X α 1
| t| X2
≤ √n n t−1
(cid:18) (cid:19)
X
P
0
−→
This follows from (1) and the fact that 1 X2 is bounded in probability, since EX2 < . This
completes the proof. n t−1 1 ∞
P
1.4 WBS for LAD estimators
Now we define the weighted bootstrap estimators, θˆ of θˆ as the minimizers of
2∗ 2
n
Q (θ)= w X θX (29)
nB nt t t 1
| − − |
t=2
X
In the next section, we deduce the consistency of this bootstrap procedure.
1.5 Consistency of the Weighted Bootstrap technique
Now we prove that the Weighted Bootstrap estimator of θˆ is assymptotically normal with the same
2
assymptotic distribution. In particular WB provides a consistent resampling scheme to estimate the
LAD estimator.
10