Statistical Methods for Forecasting by Boyas Abraham and Johannes Ledolter.

166 Pages·2009·1.2 MB·English

A Solution Manual and Notes for:
Statistical Methods for Forecasting
by Boyas Abraham and Johannes Ledolter.
John L. Weatherwax
∗
July 27, 2009
Introduction
From amazon:
Presents the statistical methods and models used in the production of short-term forecasts.
Deals with special topics such as transfer function analysis, Kalman filtering, state space
models, Bayesian forecasting, and forecast evaluation. Explains their interconnections, and
bridges the gap between theory and practice. Provides time series, autocorrelation, and
partial autocorrelation plots. Includes examples and exercises using real data.
I would appreciate constructive feedback (sent to the email below) on any errors that are
found in these notes. I hope you enjoy this book as much as I have and that these notes
might help the further development of your skills in forecasting.
∗[email protected]
1
Chapter 2: The Regression Model and its
Applications in Forecasting
Notes On The Text
Derivations of some Properties of Least Square Estimators
The sum of squares total (SSTO) is defined as and can be simplified as
SSTO = (y y¯)2 = (y (y y¯) y¯(y y¯))
t t t t
− − − −
= Xy (y y¯) Xy¯ (y y¯)
t t t
− − −
= Xy (y y¯), X
t t
−
X
since this second sum vanishes, that is (y y¯) = 0. With this, the above becomes
t
−
SSTO = y2P y¯ y = y2 ny¯2. (1)
t − t t −
X X X
Now the sum of squares due to the regression (SSR) is defined and can be simplified as
SSR = (yˆ y¯)2 = (yˆ2 2yˆy¯+y¯2)
t − t − t
= Xyˆ2 2y¯ Xyˆ +ny¯2.
t − t
X X
Now the predicted response yˆ is equal to the true response y plus some error residual e as
t t t
yˆ = y +e , so that the second sum above becomes
t t t
yˆ = y + e = ny¯,
t t t
X X X
as e = 0 if a constant (say β ) is included in the regression. Finally we find for SSR given
t 0
by
P
SSR = yˆ2 ny¯2. (2)
t −
Defining the sum of squares due to regressiXon error (SSE) as (y yˆ)2 we can decompose
t t
−
the total sum of squares as
P
SSTO = (y y¯)2 = (y yˆ +yˆ y¯)2
t t t t
− − −
= X(y yˆ)2 +X2(y yˆ)(yˆ y¯)+(yˆ y¯)2
t t t t t t
− − − −
= XSSE+2 (y yˆ)(yˆ y¯)+SSR.
t t t
− −
X
The middle term of the above expression becomes
(y yˆ)(yˆ y¯) = e (yˆ y¯) = e yˆ y¯ e = e yˆ .
t t t t t t t t t t
− − − −
X X X X X
This last expression (in vector notation) is yˆe. Now again in vector notation since the
′
ˆ ˆ
predicted values yˆ are given by yˆ = Xβ the expression yˆe becomes β X e, which is zero
′ ′ ′
since X e = 0.
′
Prediction from regression models with estimated coefficients
In this section of the book we derive several properties of least squares estimates. One in
particular is the variance of the forecast error. This can be derived as follows
V(y yˆpred) = V(ε +x (β βˆ))
k − k k ′k −
= σ2 +V(x (β βˆ)) (3)
′k −
= σ2 +x V(β βˆ)x (4)
′k − k
= σ2 +x V(βˆ)x (5)
′k k
= σ2 +σ2x (X X) 1x .
′k ′ − k
Where in going from Equation 3 to Equation 4 we have used the fact that V(ax) = aV(x)a,
′ ′
when a is a constant vector and x is a random random vector. In going from Equation 4 to
Equation 5 we have used the fact that adding a constant (here the true value of β) does not
affect the value of a variance.
Some Discussion on the Examples
Example 2.1: Running Performance
When we add the additional variables X ,X ,X ,X , representing height, weight, skin-fold
2 3 4 5
sum and relative bodyfat, to test their individualsignificance asif they were the last variable
added in the regression we compare the magnitude of their computed t statistic t , against
βˆi
thevalueoft (n p 1)asdiscussed inthesectionentitled: HypothesisTests forIndividual
α/2
− −
Components. This example has n = 14 data points and when we have added all variables
to the model we have p = 5 so that t (n p 1) = t (8). For a significance level for our
α/2 α/2
− −
test of α = 0.05 this is the value of 2.306. The magnitude of all of the computed t statistics
for the variables X ,X ,X ,X are smaller than this value and heuristically we can conclude
2 3 4 5
that they are insignificant.
Example 2.2: Gas Mileage Data
For the gas mileage data set with n = 38 to determine if the estimate of the coefficient of
the quadratic term β is statistically significant we consider the partial F or partial t test.
2
The partial t test could compare the magnitude of t against the value of
βˆ2
t (n p 1) = t (38 2 1) = t (35) = 2.032.
α/2 0.025 0.025
− − − −
Since t is less than this value we conclude the coefficient β is actually insignificant.
| βˆ2| 2
The partial F test would consider the statistic F = (t )2 = ( 1.50)2 = 2.25, which is to
∗ βˆ2 −
be compared against the value
F (1,n p 1) = F (1,35) = 4.1213.
α 0.05
− −
Since F < F (1,35) we cannot reject H at the level α. In other words there is not
∗ 0.05 0
significant evidence that value of β is different than zero, the same conclusion reached
2
above. As discussed in the text, because of the identity
F (1,n p 1) = t2 (n p 1), (6)
α − − α/2 − −
the ordinary t test and the extra sum of squares test (the F test) are equivalent.
Exercise Solutions
Exercise 2.1 (an example with least squares)
Part (a): Forming the design matrix the rows of which are the augmented observations we
find
1 x x
11 12
1 x x
 21 22 
X = 1 x31 x32 ,
 .. .. .. 
 . . . 
 
 1 x x 
n1 n2
 
 
where in this example n = 10. We desire estimations of the coefficients of our model
y = β +β x +β x +ε . (7)
t 0 1 t1 2 t2 t
The coefficients β are given by solving the normal equations
i
ˆ
(X X)β = X y. (8)
′ ′
The right-hand side of this expression is given by
y 10
t
X y = x y = 40 .
′ t1 t
 P   
x y 40
t2 t
P
   
Next we compute the normal matrix XPX given by
′
1 x x
11 12
1 1 1 1  1 x21 x22 
···
X′X = x11 x21 x31 xn1 1 x31 x32
 x12 x22 x32 ··· xn2  ... ... ... 
···
 
  1 x x 
n1 n2
 
n x x  10 0 0
t1 t2
= x x2 x x = 0 20 0 .
 t1 P t1 Pt1 t2   
x x x x2 0 0 40
t2 t1 t2 t2
P P P
   
P P 1 P
Thus βˆ is given by βˆ = (X X) 1X y = 2 .
′ − ′
 
1
 
Part (b): The concept of ANOVA is discussed in the text and the basic formfor anANOVA
table given there looks like
Sum of Degrees Mean F
Source Squares of Freedom Square ratio
Regression SSR = (yˆ y¯)2 = βˆXy ny¯2 p MSR = SSR MSR
t − ′ ′ − p MSE
Error SSE = ee n p 1 MSE = SSE
P ′ − − n p 1
Total SSTO = (y y¯)2 = yy ny¯2 n 1 − −
t ′
− − −
(correlated for mean)
P
Now for this problem we have y¯ = 1 y = 1 and using the above we find the product in
10 t
ˆ
SSR given by β X y = 130 and we have enough information to calculate the value of SSR.
′ ′ P
Now to calculate SSE with the given information recall that
ˆ ˆ
SSE = ee = (y Xβ)(y Xβ)
′ ′
− −
ˆ ˆ ˆ ˆ
= y y y Xβ β X y +β X Xβ
′ ′ ′ ′ ′ ′
− −
ˆ ˆ ˆ
= y y 2βX y +βX Xβ.
′ ′ ′
−
Note that from the given problem statement one can compute each of these terms. Doing
so we find the SSE given by ee = 115. Finally, the coefficient of determination
′
SSR SSE
R2 = = 1 = 0.774.
SSTO − SSR+SSE
Larger values of R2 are “better” implying the ability to explain more of the variation in y
around it mean. So for this problem we find that the specific ANOVA table is given by
Sum of Degrees Mean F
Source Squares of Freedom Square ratio
Regression SSR = 120 p = 2 MSR = 60 12
Error SSE = 35 n p 1 = 7 MSE = 5
− −
Total SSTO = 155 n 1 = 9
−
(correlated for mean)
Part (c): Because the estimated covariance matrix of the least square estimator ofβˆis given
by
Vˆ(βˆ) = s2(X X) 1, (9)
′ −
with s2 is estimated by
1 1
s2 = SSE = (y yˆ)2. (10)
t t
n p 1 n p 1 −
− − − −
X
ˆ
The estimated standard error on an individual component β or β is given by s = s√c ,
i i βˆi ii
with c the diagonal element in (X X) 1. Defining C to be the inverse of X X we compute
ii ′ − ′
that for this problem
4 0 0
1
C (X X) 1 = 0 2 0 .
′ −
≡ 40  
0 0 1
 
The t statistic using in testing the null hypothesis H : β = β = 0 against the alternative
0 i i0
H : β = β = 0 is obtained by computing
1 1 i0
6
ˆ
β β
i i0
t = − for i = 0,1,2.
i
s√c
ii
Forthethreeregressioncoefficientsgivenherewecomputeasavectoroftstatistics(assuming
1.4142
β = 0) given by 4.0000 . To test significance of these values we select a value for α (say
i0
 
2.8284
α = 0.05) and then compute the corresponding t (n p 1) = t (7) = 2.36 threshold.
α/2 0.025
− −
If t > t (n p 1) we reject H in favor of H at the significance of α. Since the value
i α/2 0 1
| | − −
of the t statistics for β i = 1,2 is larger than this value we conclude that the hypothesis H
i 0
can be rejected at the level α.
Part (d): For the joint test H : β = β = 0 we need a simultaneous test of H against the
0 1 2 0
hypothesis H : at least one of β = 0. If the null hypothesis is true then F = MSR, is given
1 i 6 MSE
by an F distribution with p and n p 1 degrees of freedom. Thus to asses the significance
− −
of the observed F value we should compare it with the value
F (p,n p 1) = F (2,7) = 4.73.
α 0.05
− −
Where if F > F (p,n p 1) = 4.73 we reject H in favor of H while if this is not true we
α 0 1
− −
cannot reject H in favor of H . For this problem we find F = 12 considerably greater than
0 1
F (2,7) indicating that we can reject H at the significance of α. The simple numerical
0.05 0
calculations performed in this problem are worked in the MATLAB script prob 2 1.m.
Exercise 2.2 (the normal equations for a linear trend model)
For this linear trend model y = β + β t + ε for t = 1, ,n, the normal equations are
t 0 1 t
···
given by Equation 8 where in this case the design matrix X X is given by
′
1 1
1 2
 
X = 1 3 .
 .. .. 
 . . 
 
 1 n 
 
 
So that with this expression X X then looks like
′
n n i n 1n(n+1)
X X = i=1 = 2
′ n i n i2 1n(n+1) 1n(n+1)(2n+1)
(cid:20) i=1 Pi=1 (cid:21) (cid:20) 2 6 (cid:21)
Thus our least squares PestimatePof β is given by
βˆ = (X X) 1X y = n 12n(n+1) −1 yt .
′ − ′ 1n(n+1) 1n(n+1)(2n+1) n ty
(cid:20) 2 6 (cid:21) (cid:20) Pt=1 t (cid:21)
P
We can take the inverse of the above X X matrix to find
′
1 2(1+n)(1+2n) 6(1+n)
(X′X)−1 = − .
n(n2 1) 6(1+n) 12
− (cid:20) − (cid:21)
See the Mathematical file prob 2 2.nb where we perform these manipulations. Once we
have this, our estimates of β can then be written by multiplying the above together as
βˆ 1 2(n+1)(2n+1) y 6(n+1) ty
0 = t − t .
βˆ n(n2 1) 6(n+1) y +12 ty
(cid:20) 1 (cid:21) − (cid:20) − P t t P (cid:21)
P P
ˆ
With this we see that β is given by
1
n
12 1
ˆ
β = t (n+1) y . (11)
1 n(n2 1) − 2 t
!
− t=1 (cid:18) (cid:19)
X
ˆ
Since we can solve for ty in terms of β as
t 1
P 12 6
ˆ
ty = β + y ,
n(n2 1) t 1 n(n 1) t
− −
X X
ˆ ˆ ˆ
we can put that in the equation for β to get that β in terms of β given by
0 0 1
2(1+2n) 1 1 1
ˆ ˆ
β = y 6(n+1) β + y
0 t 1 t
n(n 1) − 12 2n(n 1)
− (cid:18) − (cid:19)
1 X1 X
ˆ
= y (n+1)β
t 1
n − 2
X1
ˆ
= y¯ β .
1
− 2
These expressions will be used when we initialize the procedure used in performing double
exponential smoothing in Chapter 3, see page 35.
Exercise 2.3 (an incomplete ANOVA table)
We are told to consider a linear model, y = β + β x + β x + ε , with explanatory
t 0 1 t,1 2 t,2 t
variables X per capital real income, X relative price of beer, and a dependent variable Y
1 2
of per-capita beer consumption.
Part (a): The requested test H : β = 0 vs. H : β = 0 at the α = 0.05 level is a test of
0 1 1 1
6
the individual coefficient and depends on the value of
ˆ
β β 1.14 0
i i0
t = − = − = 7.1250.
s√c 0.16
ii
Thisistobecomparedwiththevalueoft (n p 1) = t (17 2 1) = t (14) = 2.144.
α/2 0.025 0.025
− − − −
If t > t (14) we reject H in favor of H while if t < t (14) there is not enough
0.025 0 1 0.025
| | | |
evidence for rejecting H at the significance level 0.05. In this case here t is significantly
0
larger than t (14) and we reject H (therefore we “accept” the hypothesis H ).
0.025 0 1
Part (b): Because SSTO = SSR +SSE and we are told that SSTO = 100 and SSE = 34
we know SSR = 66. Since n = 17 and p = 2 we find
SSR 66
MSR = = = 33,
p 2
and
SSE 34
MSE = = = 2.42.
n p 1 14
− −
Finally F = MSR = 13.58. Thus the ANOVA table in this case looks like
MSE
Sum of Degrees Mean F
Source Squares of Freedom Square ratio
Regression SSR = 66 p = 2 MSR = 33 13.58
Error SSE = 34 n p 1 = 14 MSE = 2.42
− −
Total SSTO = 100 n 1 = 16
−
(correlated for mean)
and R2 = SSR = 66 = 0.66. To test the simultaneous hypothesis that H : β = β = 0
SSTO 100 0 1 2
against the hypothesis that H : at least one β = 0, recall the F statistic above follows an
1 i
6
F distribution with p = 2 and n p 1 = 14 degrees of freedom. Thus we need to compare
− −
the value of F obtained above with the value of F (p,n p 1) = F (2,14) = 3.7389. If
α 0.05
− −
the value of the F ratio above is such that F > F (p,n p 1) then we can reject H in
0.05 0
− −
favor of H at the level α. Since this is true we can reject the hypothesis H . Some simple
1 0
calculations for this problem are done in the MATLAB script prob 2 3.m.
Exercise 2.4 (trigonometrically spaced observations)
Part (a): For a problem of such small size we can write down the normal equations exactly.
Our designs matrix in this case is given by
1 cos(θ) sin(θ)
1 sin(θ) cos(θ)
X =  − −  ,
1 sin(θ) cos(θ)
 1 cos(θ) sin(θ) 
 − − 
 
so that X X becomes
′
1 cos(θ) sin(θ)
1 1 1 1
1 sin(θ) cos(θ)
X′X = cos(θ) sin(θ) sin(θ) cos(θ)  − − 
 − −  1 sin(θ) cos(θ)
sin(θ) cos(θ) cos(θ) sin(θ)
− −  1 cos(θ) sin(θ) 
  − − 
 
4 0 0 4 0 0
= 0 2 4cos(θ)sin(θ) = 0 2 2sin(2θ) ,
   
0 4cos(θ)sin(θ) 2 0 2sin(2θ) 2
   
so that our least squares estimate of β is given by (X X) 1X y or
′ − ′
1
4 0 0 − y
t
ˆ
β = 0 2 2sin(2θ) y cos(θ) y sin(θ)+y sin(θ) y cos(θ) .
1 2 3 4
   − P − 
0 2sin(2θ) 2 y sin(θ) y cos(θ)+y cos(θ) y sin(θ)
1 2 3 4
− −
   
Part (b): Because the variance of our estimate βˆis given by V(βˆ) = σ2(X X) 1 to evaluate
′ −
this we thus need to compute (X X) 1. We find since this matrix is block diagonal that its
′ −
inverse can be computed as
1/4 0 0
(X′X)−1 = 0 2 2sin(2θ) −1 .
 
0 2sin(2θ) 2
(cid:20) (cid:21)
 
Now
1
2 2sin(2θ) − 1 2 2sin(2θ)
= −
2sin(2θ) 2 (4 4sin2(2θ)) 2sin(2θ) 2
(cid:20) (cid:21) − (cid:20) − (cid:21)
1 2 2sin(2θ)
= − .
4cos2(2θ) 2sin(2θ) 2
(cid:20) − (cid:21)
Thus
1 0 0
1
(X X) 1 = 0 2 2 sin(2θ) .
′ − 4  cos2(2θ) − cos2(2θ) 
0 2sin(2θ) 2
 −cos2(2θ) cos2(2θ) 
 
From which we see that when we multiply this by σ2 and consider the second and third
element of the diagonal we have
σ2
ˆ ˆ
V(β ) = = V(β ),
1 2cos2(2θ) 2
as we were to show.
Exercise 2.5 (some linear regression)
Part (a): For the given regression model we have p = 2, n = 20, n p 1 = 17. Since
− −
SSTO = SSR+ SSE we see that SSE = SSTO SSR = 200 60 = 134. Then with these
− −
values the ANOVA table looks like
Sum of Degrees Mean F
Source Squares of Freedom Square ratio
Regression SSR = 66 p = 2 MSR = 33 4.1866
Error SSE = 134 n p 1 = 17 MSE = 7.88
− −
Total SSTO = 200 n 1 = 19
−
(correlated for mean)
Part (b): Now we have R2 = SSR = 1 SSE = 0.33.
SSTO − SSTO
Part (c): To test the joint hypothesis that β = β = 0 (or that both coefficients are in fact
1 2
zero) at the 0.05 significance level we compare the F ratio computed above with the value of
F (p,n p 1) = F (2,17) = 3.59. If F > F (2,17) we reject H : β = β = 0 at the
α 0.05 0.05 0 1 2
− −
level α. In this case this is indeed true and we can reject the hypothesis H at the level α.
0
Part (d): Under the simpler model SSR = 50 so the extra regression sum of squares
SSR(X X ) = SSR(X ,X ) SSR(X ) = 66 50 = 16.
2 1 1 2 1
| − −
To test the hypothesis that the addition of X has significantly reduced SSR we compute
2
the F statistic as
∗
SSR(X X )/(p q) SSR(X X )/(2 1) 16
2 1 2 1
F = | − = | − = = 2.03.
∗
SSE(X ,X )/(n p 1) SSE(X ,X )/(20 2 1) (134/17)
1 2 1 2
− − − −
If F > F (p q,n p 1) = F (1,17) = 4.45 then we reject H the hypothesis that
∗ α 0.05 0
− − −
all additional variables are indeed non-significant and should be taken as zero simultaneity.
Since in this case we find that the value of F is not large enough we cannot reject the
∗
hypothesis H . This gives the indication that the coefficient β does not provide enough
0 2
reduction in the SSR when included in the model and should be dropped.
Part (e): If we consider the correlation of X against the residuals, e = y yˆ in Part (d),
2t t t t
−
if X has predictive power this correlation should be non-zero.
2t
Some simple calculations for this problem are performed in the MATLAB script prob 2 5.m.
Exercise 2.6 (the extra sum of squares)
Part (a): To test the hypothesis H : β = β = β = 0 at the α = 0.05 level against
0 1 2 3
the hypothesis H : at least one β = 0 we considering a simultaneous test of all regression
1 i
6
coefficients. In that case if the null hypothesis is true then the statistic F
MSR SSR/p
F = = .
MSE SSE/(n p 1)
− −
follows an F distribution with p and n p 1 degrees of freedom. For this problem p = 3,
− −
n = 10, so n p 1 = 6. To compute SSE recognized that since R2 = SSR and SSTO = 100
− − SSTO
we see that SSR = 88 so that SSE = SSTO SSR = 12. Thus our F statistic for this joint
−
test in this case becomes
88/3
F = = 14.667.
12/6
While the value of F (p,n p 1) = F (3,6) = 4.75. Since F > F (3,6) we conclude
α 0.05 0.05
− −
that we can reject H at the significance level of α.
0