Distances between time-series and their autocorrelation statistics

0.14 MB·English

by  Tryphon T. Georgiou

Distances between time-series and their
autocorrelation statistics
7
0
0
2 Tryphon T. Georgiou

n
Department of Electrical and Computer Engineering, University of Minnesota
a
Minneapolis, MN 55455; email: [email protected]
J

5
This paper is dedicated to Giorgio Picci on the occasion of his sixty-fifth birthday

]
C Summary. WebeginwithaninterpretationoftheL1-distancebetweentwopower
O spectral densities and then, following an analogous rationale, we develop a natural
metric for quantifyingdistance between respective covariance matrices.
.
h
t
a
m 1 Introduction
[

Consider two discrete-time,stationary,zero-mean,(real-valuedfor notational
1
convenience) random processes y and yˆ (k ∈ Z) having power spectral
v k k
1 densities fy(θ) and fyˆ(θ) (θ ∈[−π,π]), and autocorrelationfunctions Rℓ and
8 Rˆ (ℓ∈Z), respectively, i.e.,
ℓ
1
1 1 π
0 Rℓ =E{ykyk+ℓ}= f(θ)e−jℓθdθ,
7 2π Z−π
0
/ andsimilarly for the “hatted” quantities.When the powerspectrum contains
h
a singular part, then f(θ)dθ needs to be replaced by a non-negative finite
t
a spectral measure dµ(θ).
m
We are interested in quantifying the distance between respective spectra
v: and statistics for two such random process yk and yˆk. When two vectors
i
X Rn := R0 R1 ...Rn−1 , and
ar Rˆn :=(cid:2)R0 R1 ...Rn−1(cid:3)
(cid:2) (cid:3)
of autocorrelation samples are available and need to be compared, one may
use any metric in Rn for that purpose, as for instance kR − Rˆ k =
n n 2
(R −Rˆ )2. However, we are not aware of any significance that can
k k k
qbePattached to such a distance other than the fact that it is a metric in Rn.
Ourgoalinthis paper,is to seek ametric whichcanbe physicallymotivated.
2 TryphonT. Georgiou
Similarly,ifwearetocomparef (θ)andf (θ),itappearsdifficulttomoti-
y yˆ
vatetheuseofanL -distancekf (θ)−f (θ)k .Foronething,theL -distance
2 y yˆ 2 2
cannot be generalizedto deal with spectralmeasures when singular parts are
present. There are certainly other alternatives. In the speech processing lit-
erature in particular there is a plethora of distances that, however, are not
metrics [6] but have been motivated by specific needs. Function theoretic al-
ternatives that one can use (e.g., L -norms, etc.) including Wasserstein-like
p
transportation measures typically lack a physical interpretation. In a recent
study [4, 5] a pseudometric was constructed as a geodesic between spectral
densities/measureswithrespecttoarathernaturalRiemannianmetric—this
metric quantifies the degradation of predictive-error variance when the pre-
dictor is designed based on the wrong choice between two alternatives and
the geometry is, in essence, Euclidean but only after we transform spectral
densities using the logarithmic map.
In the current paper we focus on the L distance
1
1 π
kf (θ)−f (θ)k := |f (θ)−f (θ)|dθ
y yˆ 1 y yˆ
2π Z−π
which has also a rather natural interpretation. After a brief discussion of the
relevance of the L -distance, following a similar rationale, we will develop an
1
analogous metric between finite partial covariance data of the corresponding
random processes.
2 Interpretation of the L distance
1
Given y and yˆ we postulate that ther exist two random processes ψ and
k k k
ψˆ so that
k
y +ψ =yˆ +ψˆ . (1)
k k k k
Alternatively, we postulate that there exists a random process z and that
k
the two original random processes relate to z via
k
y =z −ψ
k k k
yˆ =z −ψˆ .
k k k
It is natural to seek such perturbations of minimal total combined variance
E{ψ2}+E{ψˆ2} (2)
k k
that is sufficient to “reconcile” the two processes. The combined variance
E{ψ2}+E{ψˆ2} represents the minimal amount of “energy”of perturbations
k k
inthetwotime-seriesthatisneededtorenderthetwoindistinguishable.Intu-
itively, the minimal combined variance which is consistent with the available
data quantifies the distance between the two.
Distances between time-series & statistics 3
Given f ,f , the optimal choice consists of random processes ψ and ψˆ
y yˆ k k
such that ψ and ψˆ are independent, yˆ and ψˆ are also independent, and
k k k k
f (θ)−f (θ) if f (θ)−f (θ)≥0,
f (θ)= yˆ y yˆ y (3a)
ψ (cid:26) 0 otherwise,
f (θ)−f (θ) if f (θ)−f (θ)≤0,
f (θ)= y yˆ yˆ y (3b)
ψˆ (cid:26) 0 otherwise.
Then, the power spectrum of the “sum”
z :=y +ψ =yˆ +ψˆ
k k k k k
is simply
f (θ):=max{f (θ), f (θ)}, θ ∈[−π,π],
z y yˆ
and
d(f ,f ) :=E{ψ2}+E{ψˆ2} (4)
y yˆ k k
1 π
= (f (θ)+f (θ))dθ
2π Z−π ψ ψˆ
1 π
= |f (θ)−f (θ)|dθ
y yˆ
2π Z−π
= kf −f k . (5)
y yˆ 1
Obviously, this construction extends in the obvious way to the case of not-
necessarily absolutely continuous power spectra as well, and the metric in-
cludes the measureofanydiscrepanciesbetweenthe singularpartsofthe two
spectral measures.1 Clearly, d(f ,f ) is a metric as seen from (5). Building
y yˆ
on a similar rationale,in the next section, we developa metric for covariance
matrices.
3 A distance between covariance matrices
It is often the case that only a finite segment of the autocorrelation func-
tion of time-seriesy and yˆ is available(and eventhen, possibly uncertain).
k k
Thus, it is of interest to consider distances between the partial autocorrela-
tion statistics R and Rˆ. To this end, we follow the dictum of the previous
section and define as a distance measure the minimal combined variance of
1
Itwillbeinterestingtoexplorethepracticalsignificance ofotherpossibilities for
quantifyingdistance such as
R−ππ(fψ−π(πθf)z+(θf)ψdˆ(θθ))dθ or Z−ππ fψ(θf)z+(θf)ψˆ(θ)2dπθ.
R
4 TryphonT. Georgiou
randomprocessesψ andψˆ forwhich(1)holds.Naturally,sinceonly partial
k k
covariance samples are available, the random processes ψ ,ψˆ and (1) need
k k
to be consistent with these data.
First denote by
R0 R1 ...Rn−1
 R−1 R0 ...Rn−2
Rn := ... ... ... ... 
 
R−(n−1) R−(n−2) ... R0 
then×ncovariancematrixcorrespondingtoy andthecovariancesamplesin
k
R and, similarly,Rˆ for the Toeplitz matrix based on Rˆ . If Q ,Qˆ denote
n n n n n
the correspondingfinite Toeplitz covariancesof the randomprocesses ψ and
k
ψˆ , respectively, for which (1) holds, then
k
R +Q =Rˆ +Qˆ (6)
n n n n
andtheminimalsumQ +Qˆ oftherespectivevariancescanserveasametric
0 0
quantifying the distance between R and Rˆ.
The computation of Q ,Qˆ minimizing the sum Q +Qˆ , or equivalently
n n 0 0
minimizing
1
trace(Q +Qˆ ),
n n
n
isaconvexproblem–sincethe positivityconstraintsareconvex.TheToeplitz
structure is peripheral, and the idea of defining such metrics extends equally
welltonon-negativedefinite Hermitianmatricesandto moregeneralpositive
operators.Fornotationalconveniencewedeveloptheframeworkinthecontext
of real symmetric matrices.
So, we let
M :={M ∈Rn×n |M =M′ ≥0}
n
be the cone of non-negative symmetric n×n-matrices and
T :={R∈M |R is a Toeplitz matrix}
n n
be the cone of non-negative Toeplitz matrices in M . We address the case of
n
matrices in M and define a suitable metric, which is then specialized to T .
n n
Proposition 1.Let M ,M ∈M and
1 2 n
1
τ(M ,M ) :=min trace(M)|M ∈M ,
1 2 n
(cid:26)n
M ≥M and M ≥M }.
1 2
Then
δ(M ,M ):=2τ(M ,M )−trace(M )−trace(M ) (7)
1 2 1 2 1 2
defines a metric on M .
n
Distances between time-series & statistics 5
Proof. Given M ,M ∈M ,
1 2 n
C(M ,M ):={M |M ≥M and M ≥M }
1 2 1 2
is a (convex) cone of non-negative definite matrices. It follows that there is
an element M ∈C(M ,M ) having minimal trace.
12 1 2
Clearlyδ(M ,M )issymmetricinitsargumentsandtakespositivevalues
1 2
unless M = M , in which case δ(M ,M ) = 0. Thus, we only need to prove
1 2 1 2
the triangle inequality. Given M ∈ M for i ∈ {1,2,3}, we denote by M
i n ik
corresponding minimal elements as above for i,k ∈{1,2,3},and we let
∆ :=M −M .
ik ik k
These matrices are non-negative by construction, the identities
M +∆ =M +∆
i ki k ik
hold, and
1
δ(M ,M )= trace(∆ +∆ )
i k ik ki
n
for i,k ∈{1,2,3}.But then,
M +∆ −∆ =M
1 21 12 2
=M +∆ −∆ ,
3 23 32
and hence,
M +∆ +∆ =M +∆ +∆ .
1 21 32 3 23 12
Fromthe minimalpropertyof∆ andof∆ withregardtohavingthe least
13 31
value for the combined trace so that M +∆ =M +∆ , it follows that
1 31 3 13
trace(∆ +∆ )≤trace(∆ +∆ +∆ +∆ ).
13 31 21 32 23 12
Therefore,
δ(M ,M )+δ(M ,M )= δ(M ,M ),
1 2 2 3 1 3
which completes the proof. 2
We now observe that the steps of the proof of Proposition 1 permit in-
corporating linear constraints on the structure of elements of M , such as
n
the constraint of all matrices being Toeplitz. Hence, whereas δ(·,·) may be
used directly as a distance measure between elements of T , the correspond-
n
ing minimal-trace perturbations ∆ may not belong to T in general. But,
ik n
since the Toeplitz property is a linear constraint,we may define a completely
analogousdistance measure enforcing such perturbations (if so desired)to be
Toeplitz.
6 TryphonT. Georgiou
Proposition 2.Let M ,M ∈T and
1 2 n
1
τ (M ,M ):= min trace(M)|M ∈T , and
T 1 2 (cid:26)n n
M ≥M , M ≥M .
1 2
(cid:27)
Then
δ (M ,M ):=2τ (M ,M )−trace(M )−trace(M ) (8)
T 1 2 T 1 2 1 2
defines a metric on T .
n
Proof. The proof follows the steps of the proof of Proposition 1 verbatim,
except for the fact that we now constraint all matrices to belong to T . 2
n
Proposition 3.Let f , f be power spectral densities, i.e., nonnegative and
y yˆ
integrable on [−π, π]. Let as before R , Rˆ denote the corresponding Toeplitz
n n
covariance matrices, and let n∈{1, 2, ...}. Then
lim δ (R ,Rˆ )=kf −f k .
n→∞ T n n y yˆ 1
Proof. Clearly
lim δ (R ,Rˆ )≤kf −f k
n→∞ T n n y yˆ 1
sinceachoiceofψ andψˆ withpowerspectraasin(3a-3b)givesrisetopartial
k k
covariancematricesQ ,Qˆ ,foralln,forwhich(6)holds.The respective0th
n n
elements Q and Qˆ remain the same for all n and the left hand side is
0 0
kf −f k =Q +Qˆ
y yˆ 1 0 0
since the power spectra in (3a-3b) have no overlapin their support.
To show the converseinequality, consider the sequence of minimizing Q ,
n
Qˆ .TheseareToeplitzmatriceswithboundedentries(sincetheircorrespond-
n
ing0thelementisboundedbykf −f k ).Eachcanbeextendedtoaninfinite
y yˆ 1
Toeplitz matrix, and thereby, gives rise to power spectral densities q and qˆ
n n
such that the first n Fourier coefficients of f +q and f +qˆ coincide. The
y n yˆ n
spectral densities q and qˆ can be obtained from Q , Qˆ by any particu-
n n n n
lar positive extension, for instance a “maximum entropy” one. We can take
those as pairs, and since they are bounded there exists a subsequence weakly
convergentto possibly non-negative measures, dµ and dµˆ, such that
f dθ+dµ=f dθ+dµˆ
y yˆ
since their Fourier coefficients must coincide.If dµ, dµˆ do have singularparts
then these should be identical and the absolutely continuous parts must bal-
ance as well, so there exist power spectral densities q and qˆsuch that
f +q =f +qˆ. (9)
y yˆ
Distances between time-series & statistics 7
But then,
1 π
lim δ (R ,Rˆ )≥ lim (q (θ)+qˆ (θ))dθ
n→∞ T n n n→∞2π Z−π n n
1 π
= (dµ+dµˆ)
2π Z−π
1 π
≥ (q+qˆ)dθ
2π Z−π
≥kf −f k ,
y yˆ 1
the last inequality from (9). 2
3.1 An example
The metric δ (R ,Rˆ ) of the previous section admits no simple expression
T n n
in terms of the respective eigenvalues. This should be contrasted with its
limiting value d(f ,f ) which is the L distance between the corresponding
y yˆ 1
power spectral densities. We highlight this with an example.
Let
1 1 1
R3 = 1 1 1 
1 1 1
 
and
1 1/21/2
Rˆ3 =1/2 1 1/2.
1/21/2 1
 
Then, clearly,
x y y
Q3 = y x y 
y y x
 
and
x v v
Qˆ3 = v x v 
v v x
 
where
1+y =1/2+v
andxisminimalsubjecttoQ ≥0aswellasQˆ ≥0.Thelasttwoinequalities
3 3
imply that
0≤ x as well as
1
− x≤ y,v ≤x.
2
It follows that the optimal choice (minimal x) is
8 TryphonT. Georgiou
x= 1/3
y = −1/6
v = 1/3.
Then
δ (R ,Rˆ )=2x=2/3,
T 3 3
while the respective eigenvalues are
spec(R )= {3, 0, 0} and
3
spec(Rˆ )= {1, 1, 1/2}.
3
It appears that there is no simple expression for δ (R ,Rˆ ) based solely on
T 3 3
knowledge of spec(R ) and spec(Rˆ ).
3 3
The covariance R has a unique extension and corresponds to a measure
3
withunitweightatθ =0,i.e.,aspectralline(Diracdelta)atθ =0.Assuming
that Rˆ originates from a spectral measure which has a similar weight of
3
amplitude1/2atθ =0andauniformabsolutelycontinuouspartofamplitude
1/2, then
kdµ−dµˆk =1/2+1/2=1
1
adding the L -norm of the difference of the absolutely continuous parts with
1
theabsoluteintegralofthediscrepancybetweenthetwomeasures.Weleaveit
asanexercisetothe readertoverifythatifRˆ isaswejustassumed,namely
n
Rˆ =1/2 for k ≥1, and similarly, R =1 for all k, then δ (R ,Rˆ )→1 as
k k T n n
n→∞.
4 Approximating sample covariances
It is often the case that the autocovariance matrix R of a random process
n
y is estimated in a way that does not guarantee this to be Toeplitz. For
k
instance,itis quite commonfor R to be estimatedby averagingobservation
n
samples
y
N 1+ℓ
Rˆ = 1  ..  y ...y
n . 1+ℓ n+ℓ
N +1
Xℓ=0yn+ℓ(cid:2) (cid:3)
 
The estimate Rˆ is non-negative definite by construction, but may not be
n
Toeplitz. Yet, for purposes of analysis it is often beneficial to approximate
R by a Toeplitz one, or possibly, by one with additional structure (e.g.,
n
corresponding to a moving average process or, more generally, to the state
of a known dynamical system). The problem of seeking such an approximant
which is closest to R in δ(·,·), is readily solvable via convex optimization.
n
Distances between time-series & statistics 9
4.1 Comparison with the von Neumann entropy
In[1],thequestionwasraisedastowhatareappropriatewaystoapproximate
a given sample covariance with one that abides by a known linear structure.
It was proposed that the Kullback-Leibler-vonNeuman distance
S(RˆkR):=trace Rˆ logRˆ −logR
(cid:16) (cid:16) (cid:17)(cid:17)
providesaconvenientconvexfunctionalforwhichtheoptimalapproximantis
uniquely defined. An academic example was presentedin [1] which is recapit-
ulatedhereas ithelps underscoredifferenceswith approximationinthe sense
of minimizing δ(Rˆ,R).
Consider the positive-definite matrix below as the estimated value for a
covariance matrix
1.1 .9 1.05
1
Rˆ3 =  .9 .8 .9 .
3
1.05.9 1.1
 
The minimizer of
{S(Rˆ,R)|R being Toeplitz, R>0, trace(R)=trace(Rˆ)}
is unique (see [1]) and given by
1 .942.957
1
R3,vN = .942 1 .942.
3
.957.942 1
 
It is interestingto point outthe the closestToeplitz matrixto Rˆ in the least-
squares sense fails to be positive-definite ([1], cf. [2]). On the other hand, the
optimal approximant in δ(·,·)-sense can be obtained by observation and is
equal to
1.1 .9 1.05
1
R3,δ =  .9 1.1 .9 .
3
1.05 .9 1.1
 
In the above, a second subscript indicates the sense in which the matrix ap-
proximates Rˆ . Obviously the traces of R and Rˆ are not the same, in
3 3,δ 3
general. However, equality of the traces can be easily imposed as an added
linear constraint.
4.2 Structured covariances
For purposes of illustration, consider a moving averageprocess
yk =wk+wk−1+wk−2
10 TryphonT. Georgiou
where w is a zero-mean, unit-variance, Gaussian white noise process. The
k
autocorrelationsequence of y is
k
R R R R 0... = 32 10 0... .
0 1 2 3
(cid:2) (cid:3) (cid:2) (cid:3)
Simulating y over a window k ∈ {0,1,...,100}, and based on a particular
k
suchrealization,thecorrespondingn×nsamplecovariancematrix,forn=5,
was computed to be
4.03622.90531.80430.40420.1718
2.90534.05472.92681.79450.3800
Rˆ = 1.80432.92684.07922.91431.7733 .
5  
0.40421.79452.91434.08192.9421
 
0.17180.38001.77332.94214.0237
 
Obviously,this matrix is not Toeplitz due to the finiteness of the observation
record. The closest Toeplitz approximant to Rˆ , in the sense of the metric
5
δ(·,·), turns out to be
4.06772.92371.79120.39790.1822
2.92374.06772.92371.79120.3979
R = 1.79122.92374.06772.92371.7912
5,Toeplitz  
0.39791.79122.92374.06772.9237
 
0.18220.39791.79122.92374.0677
 
for which
δ(Rˆ ,R )=0.0308.
5 5,Toeplitz
Interestingly, R does not correspond to a moving average process of
5,Toeplitz
order 2 (or even, of order 3,4) as it can be readily verified by the fact that
the trigonometric polynomials, e.g.,
4
R ejkθ
k
kX=−4
takes negative values.
The set of covariance matrices which are generated by moving average
processesofagivenorder,isconvexandadmitsacharacterizationviaasetof
linear matrix inequalities ([8, 3]). Thus, the closest approximant to Rˆ which
corresponds to a moving average process of any given order can be readily
computed. In particular, if we specify the order to be 2, then the optimal
approximant to Rˆ becomes
5
3.99452.15880.5693 0 0
2.15883.99452.15880.5693 0 
R = 0.56932.15883.99452.15880.5693
5,MA(2)  
 0 0.56932.15883.99452.1588
 
 0 0 0.56932.15883.9945
 
for which
δ(Rˆ ,R )=1.2161.
5 5,MA(2)