Table Of ContentOptimal Nonstationary Reproduction Distribution
for Nonanticipative RDF on Abstract Alphabets
Photios A. Stavrou, Charalambos D. Charalambous and Christos K. Kourtellaris
ECE Department, Univercity of Cyprus, Nicosia, Cyprus
Email:{stavrou.fotios, chadcha, kourtellaris.christos}@ucy.ac.cy
Abstract—THIS PAPER ISELIGIBLE FOR THE STUDENT our results apply forthe stationary case too. Note that our re-
PAPER AWARD. In this paper we introduce a definition for sultsextendtheboundsdevelopedrecentlyin[6]forstationary
nonanticipative Rate Distortion Function (RDF) on abstract al-
Gaussian sources to general sources.
phabets,andweinvokeweakconvergenceofprobabilitymeasures
Thispaperisorganizedasfollows.SectionIIformulatesthe
toshowvariousofitsproperties,suchas,existenceoftheoptimal
reproductionconditionaldistribution,compactnessofthefidelity nonanticipativeRDF on abstractspaces. Section III, discusses
set,lower semicontinuityof theRDFfunctional,etc.Further,we the relation between nonanticipative RDF with Gorbunov-
3 derive the closed form expression of the optimal nonstationary Pinsker’s nonanticipatory ǫ-entropy and sequential RDF. Sec-
1 reproduction distribution. This expression is computed recur-
tion IV provides the conditions under which the existence of
0 sively backward in time. Throughout the paper we point out an
the optimal reproduction conditional distribution of nonantic-
2 operationalmeaningofthenonanticipativeRDFbyrecallingthe
codingtheoremderivein[1],andwestaterelationstoGorbunov- ipative RDF is derived. Finally, Section V gives the optimal
n
Pinsker’s nonanticipatory ǫ−entropy [2]. solutionfornonstationaryprocesses.Throughoutthepaperwe
a
J also include comments on the reduction of our results to the
8 I. INTRODUCTION stationary nonanticipative RDF.
2
Motivated by communication and control applications in II. NONANTICIPATIVERDF ONABSTRACT SPACES
T] which lossy nonanticipative reproductions of nonstationary In this section, we define the information theoretic nonantici-
I sources subject to a fidelity set are desirable, in this paper pative RDF based on the methodology described in [7].
s. we introduce a definition for nonanticipative RDF. Notation. Let N =△ {0,1,2,...}, and Nn =△ {0,1,2,...,n}.
c One envisioned application is quantization of two dimen- Introduce two sequence of spaces {(X ,B(X )) : n ∈ N}
[ sional sources, represented by space-time processes, in which and {(Y ,B(Y )) : n ∈ N}, wherenX ,Yn,n ∈ N, are
n n n n
1 an IID assumption is impossed on the spatial index, when Polishspaces,andB(X )andB(Y )areBorelσ−algebrasof
n n
2v tthoedteimriveeinadecxodiisngfixtehde.orSeumchwahimchodsetalteiss cthoantsidtheeredOpintim[1a]l subsetsofXn andYn, respectively.Pointsin XN =△ ×n∈NXn,
2 Performance Theoretically Achievable (OPTA) by sequential YN =△ ×n∈NYn are denoted by x =△ {x0,x1,...} ∈ XN,
5 quantizersisgivenbytheso-calledsequentialRDF,expressed y =△ {y ,y ,...} ∈ YN, respectively, while their restrictions
6 0 1
. intermsofmutualinformationandaconditionalindependence to finite coordinates by xn =△ {x ,x ,...,x } ∈ X ,
1 0 1 n 0,n
0 impossedonthefidelityset.Anotherenvisionedapplicationis yn =△ {y0,y1,...,yn}∈Y0,n, for n∈N.
source-channelmatching [3], [4], in which nonanticipationof
13 the reproductiondistribution is a necessary conditions for the Let B(XN)=△ ⊙i∈NB(Xi) denote the σ−algebra on XN gen-
: realizationofthisconditionaldistributionbyencoder-channel- eratedbycylindersetsandsimilarlyforB(YN)=△ ⊙i∈NB(Yi),
v
i decodermapsthatoperatewithoutanticipation,andhencethe whileB(X0,n)andB(Y0,n)denotetheσ−algebraswithbases
X delayincurredontheend-to-endsystemislimited,oruncoded over Ai ∈ B(Xi), and Bi ∈ B(Yi), i = 0,1,...,n,
r transmission [3]. respectively.Let Q(Y;X) denote the set of stochastic kernels
a
Inthispaper,thenonanticipativeRDFisdefinedonabstract on Y given X [8].
alphabets, using an information measure which is a special
A. Construction of the Measures
case of directed information [5] from the source sequence
to the reproduction sequence. The main contributions are Source Distribution. The source distribution
the treatment of the nonanticipative RDF without assuming {pn(dxn;xn−1):n∈N} satisfies the following conditions.
stationarity of the source, the derivation of various results i) For n∈N, pn(·;xn−1) is a probabilitymeasure on B(Xn);
regarding existence of the optimal nonstationary reproduction ii) For every An ∈ B(Xn), n ∈ N, pn(An;xn−1) is a
distribution, its closed form expression via backward recur- ⊙in=−01B(Xi)-measurable function of xn−1 ∈X0,n−1.
sions, relations to existing coding theorems, and relations to Any distribution satisfying i), ii) is denoted by
the nonanticipatory ǫ−entropy introduced by Gorbunov and pn(·;·)∈Q(Xn;X0,n−1).
Pinsker [2]. Since nonstationarynonanticipative RDF embeds Let A ∈ B(X0,n) be a cylinder set of the form
thestationarynonanticipativeRDF,undercertainassumptions, A =△ x ∈ XN : x ∈ A ,x ∈ A ,...,x ∈ A , A ∈
0 0 1 1 n n i
(cid:8) (cid:9)
−→
B(X ), i= 0,1,...,n. Define a family of measures P(·) on P3: The product distribution Π : B(X )⊙B(Y ) 7→
i 0,n 0,n 0,n
B(XN), denoted by the set M (XN), given by [0,1] defined uniquely for A ∈ B(X ), B ∈ B(Y ), i =
1 i i i i
0,1,...,n, by
P(A)=△ p (dx )... p (dx ;xn−1)≡µ (A )
0 0 n n 0,n 0,n
ZA0 ZAn −→Π (×n (A ×B ))=△ (µ ⊗ν )(×n (A ×B )).
(1) 0,n i=0 i i 0,n 0,n i=0 i i
where A0,n = ×ni=0Ai. The notation µ0,n(·) is used to Theinformationtheoreticmeasureofinterestisa specialcase
denote the restriction of the measure P(·) on cylinder sets ofdirectedinformation[1]definedbyrelativeentropy D(·||·)1
A∈B(X ), for n∈N.
0,n
Reproduction Distribution. The reproduction distribution I (Xn →Yn)=△ D(µ ⊗−→Q ||−→Π ) (4)
{qn(dyn;yn−1,xn) : n ∈ N} satisfies the following µ0,n −→ 0,n 0,n 0,n
conditions. = log Q0,n(dyn|xn) (µ ⊗−→Q )(dxn,dyn) (5)
iv) For n ∈ N, qn(·|yn−1,xn) is a probability measure on Z (cid:16) ν0,n(dyn) (cid:17) 0,n 0,n
B(Yn); ≡I (µ ,−→Q ). (6)
v) For every B ∈ B(Y ), n ∈ N, q (B ;yn−1,xn) is Xn→Yn 0,n 0,n
n n n n
a ⊙n−1 B(Y ) ⊙ B(X ) ⊙ B(X )-measurable function of
i=0 i i n Theequivalencebetween(4) and(5) followsfromthe Radon-
xn ∈X0(cid:0),n, yn−1 ∈Y0,n−(cid:1)1. Nikodym Derivative (RND). The notation IXn→Yn(·,·) in-
Given a cylinder set B =△ y ∈ YN : dicates the functional dependence of I (Xn → Yn) on
→− µ0,n
Qy0(∈·|Bx0),oyn1∈BB(Y1,N.)..b,yyn∈Bn(cid:9), define a (cid:8)family of measures {µ0,n,Q0,n}.
Q(B|x)=△ q (dy ;x )... q (dy ;yn−1,xn) (2) B. Nonanticipative RDF
0 0 0 n n
Z Z
−→B0 Bn We are now ready to introduce the information definition of
≡ Q0,n(B0,n|xn), B0,n ∈B(Y0,n). (3) nonanticipative RDF. The distortion function d0,n(xn,yn) :
Then Q(·|x) is a unique measure on (YN,B(YN)) for which X0,n×Y0,n 7→[0,∞)isameasurablefunction,andthefidelity
the family of distributions {q (dy ;yn−1,xn) : n ∈ N} is of reproduction is defined by
n n
obtained.
−→ △ −→ C1
Considerany family of measures Q(·|x) on B(YN) satisfying Q0,n(D)= Q0,n ∈Q (Y0,n;X0,n):
n
tCh1e:foIflloDwi∈ngBc(oYn0s,inst)e,ntchyencoQnd(iDtio|xn). is B(X0,n)−measurable ℓd0,n(−→Q0,n)=△ n+1 1Z d0,n(xn,yn)(µ0,n⊗−→Q0,n)(dxn,dyn)
function of x∈XN.
The set of such measures is denoted by QC1(YN,XN). ≤D , D ≥0. (7)
o
Then, for any family of measures Q(·|x) on (YN,B(YN))
The information nonanticipative RDF is defined by
satisfying consistency condition C1 one can construct a col-
lection of functions {q (dy ;yn−1,xn) : n ∈ N} satisfying
conditions iv) and v) wnhichn are connected with Q(·|x) via −→R0,n(D)=△→− i→−nf IXn→Yn(µ0,n,−→Q0,n). (8)
relation (2) (see also [7]). Q0,n∈Q0,n(D)
Byfollowingthemethodologyin[7],giventhebasicmeasures −→
P(·)onXNandQ(·|x)onYNsatisfyingconsistencycondition If the infimum in (8) does not exist then R0,n(D)=∞.
The nonanticipative RDF rate is defined by
C1, we can uniquely define the collection of conditional
distributions {p (·;·) ∈ Q(X ;X ) : n ∈ N} via (1),
n n 0,n−1 −→ 1 −→
and {q (·;·,·) ∈ Q(Y ;Y × X ) : n ∈ N} via (2), R(D)= lim R (D) (9)
n n 0,n−1 0,n n→∞n+1 0,n
respectively,andviceversa,hencethedistributionofthe RV’s
{(Xi,Yi):i∈Nn} is well defined. provided the limit exists.
Next, we introduce the information definition of nonanticipa-
tive RDF. Given the source distribution P(·)∈M (XN) and
reproduction distribution Q(·|·) ∈ QC1(YN;XN)1define the III. NONANTICIPATORYǫ-ENTROPY, MESSAGE
GENERATIONRATES AND SEQUENTIALRDF
following measures.
P1: The joint distribution on XN×YN defined uniquely by
In this section, we establish some preliminary relations be-
(µ ⊗−→Q )(×n A ×B ),A ∈B(X ), B ∈B(Y ). tween (8), (9) and 1) Gorbunov-Pinsker’s nonanticipatory ǫ-
0,n 0,n i=0 i i i i i i
entropy and message generation rates [2] and 2) sequential
P2:ThemarginaldistributiononYNdefineduniquelyforBi ∈ RDF and coding theorem [1].
B(Y ), i=0,1,...,n, by
i
ν (×n B )=(←P− ⊗−→Q )(×n (X ×B )). 1Unless stated otherwise, integrals with respect to measures are over the
0,n i=0 i 0,n 0,n i=0 i i spaces onwhichthesearedefined.
A. Nonanticipatoryǫ-Entropy and Message Generation Rates lim 1 Rna(D) = Rna(D) < ∞, or 2) for
k→∞ k+1 0,k
We recall Gorbunov-Pinsker’s definition of nonanticipatory any k ∈N, R0n,ak(D)=Rna(D)=∞.
GP 3:For a single letter distortion and stationary source,
ǫ-entropy [2]. Given a source P ∈ M (X ) and a
Xn 1 0,n
reproduction P ∈ Q(Y ;X ) introduce the fidelity if Rna(D) = Rna,+(D) then the analysis of in-
Yn|Xn 0,n 0,n
fimum in (12) is realizable in terms of stationary
set
source-reproduction pairs {(X ,Y ) : i ∈ N}
Q0,n(D)=△ PYn|Xn(dyn|xn): such that PYn|X∞(dyn|x∞) =i iPYn|Xn(dyn|xn)
n
1 d (xn,yn)P (dyn|xn)⊗P (dxn)≤D . and PYn|Xn ⊗PXn ∈Q0,n(D).
n+1Z 0,n Yn|Xn Xn o With respect to Gorbunov-Pinsker’sdefinition of nonanticipa-
tory ǫ-entropy and message generation rate, 1) we show that
Next we introduce two definitions from [2].
(10) reduces to the information definition of nonanticipative
Definition 1. (1) The source X∞ =△ {X : i ∈ N} is RDF (8), 2) under certain general conditions the infimum in
i
called “specified” if PYk|Xk ∈ Q0,k(D) and PYn |Xn ∈ (10) exists and it is finite, and, 3) we derive an expression
k+1 k+1
Q (D) implies P ∈ Q (D), and it is of the optimal reproduction distribution which achieves the
k+1,n Yk,Yn |Xk,Xn 0,n
called “consistent” if the rke+v1erse hko+ld1s. infimum in (8) or (10).
(2) The source is called stationary if {X : i ∈ N} is First, we establish the connection between nonanticipatory
i
stationaryandforanyk,Q (D)andQ (D)arecopies ǫ-entropy (10) and nonanticipative RDF (8) by utilizing the
0,n k,n+k
of the same set. following general equivalent statements of MCs.
Typical example are stationary sources with single letter Lemma 1. The following are equivalent for each n∈N.
fidelities d0,n(xn,yn) = ni=0ρ(xi,yi). For a specified (1) PYn|Xn(dyn|xn)=−→PYn|Xn(dyn|xn)-a.s.;
source, Gorbunov and PinskPer restricted the set Q0,n(D) to (2) For each i = 0,1,...,n − 1, Yi ↔ (Xi,Yi−1) ↔
those reproduction distributions which satisfy the Markov (X ,X ,...,X ), forms a MC;
i+1 i+2 n
chain (MC) Xn∞+1 ↔ Xn ↔ Yn ⇔ PYn|X∞(dyn|x∞) = (3) For each i = 0,1,...,n−1, Yi ↔ Xi ↔Xi+1 forms
PYn|Xn(dyn|xn) − a.s., n = 0,1,..., and introduced the a MC;
nonanticipatory ǫ-entropy defined by (4) For each i=0,1,...,n−1, Xn ↔Xi ↔Yi, forms
i+1
a MC.
Rna(D)=△ inf I(Xn;Yn). (10)
0,n
PYn|Xn∈Q0,n(D): Proof: The derivation is straightforward.
Xin+1↔Xi↔Yi, i=0,1,...,n−1 By utilizing Lemma 1, then −→R (D) = Rna(D). Then the
0,n 0,n
If the infimum in (10) does not exist then Rna(D)=∞. extremum of the nonanticipatory ǫ-entropy (10) is equivalent
0,n −→
The difference between the classical RDF [9] and (10) is the to the extremum of R (D) given by (8).
0,n
presence of the MC, which implies that for each i, Y is a
i
function of the past and present source symbols Xi, and it is B. Sequential RDF and Coding Theorems
−→
independent of the future source symbols Xn , n∈N. Next, we establish a coding theorem for R (D) using the
i+1 0,n
Moreover, the authors in [2] also introduced the nonanticipa- information sequential RDF introduced by Tatikonda in [1],
tory message generation rate of the source defined by which utilizes a similar formulation to the nonanticipatory
ǫ-entropy. The coding theorem is derived by considering a
1
Rna(D)=nl→im∞n+1R0n,an(D) (11) two dimensional source Xn,s =△ {Xi,j : i = 0,...,n,j =
0,...,s} ∈ ⊗n ⊗s X , where i represents time index
provided the limit exists. i=0 j=0 i,j
and j represents spatial index. The coding theorem is based
An alternative definition of the nonanticipatory message gen-
on the following definitions.
eration rate of the source is defined by [2]
I(Xn;Yn) Definition 2. A sequential quantizer is a sequence of mea-
Rna,+(D)= inf lim (12) surable functions fn = {f : i = 0,1,...,n} defined by
PY∞|X∞∈Q0,∞(D): n→∞ n+1 f :Xi,s×Yi−1,s 7→Ys, Yis =f (xi,s,yi−1,s), i=1,...,n.
X∞ ↔Xi↔Yi, i=0,1,... i i i i
i+1 The set of all such quantizers is denoted by Fn,s.
whenever the limit exists. Clearly, in general Rna,+(D) ≥
Rna(D). Definition 3. Let QS0,RnD,s,o(D) denote the fidelity set
The main results derived in [2] are the following. n
GP 1:RIfntah(eDs)ou<rcedi=s sctaotniosntaanryt f<or∞so,mtheenfinRitenak(D, )i.ei.s, QS0,RnD,s,o =△ nfn ∈Fn,s : n+1 1Xi=0EPXi,sρs(Xis,Yis)≤Do
0,k
defined and it is finite, and Rna(D) ≥ Rna,+(D), whereρ :Xs×Ys 7→[0,∞): i=0,1,...,nismeasurable.
s i i
hence Rna(D)=Rna,+(D). The operational sequential RDF is defined by
GP 2:If the source is stationary and consistent then
Rna(D) is always defined, and either 1) for RSRD,o(D)=△ inf 1 H(Ys,...,Ys)
some finite k, Rna(D) < ∞ which implies 0,n,s fn∈QS0,RnD,s,o(D)(n+1)(s+1) 0 n
0,k
and the operational sequential RDF rate is defined by Theorem 2. [7] Let {X : n ∈ N} and {Y : n ∈ N} be
n n
Polish spaces. Then
RSRD,o(D)=△ sl→im∞R0S,Rn,Ds,o(D). (1) The set QC1(Y−→0,n;X0,n) is convex.
The information sequential RDF for which a coding theo- (2) −→IXn→Yn(µ0,n,Q0,n) is a convex functional of
rem is derived in [1] is the following. Given the two di- Q0,n ∈ QC1(Y0,n;X0,n) for a fixed µ0,n ∈
mensional source PXn,s(dxn,s), a reproduction distribution M1(X0,−→n).
PYn,s|Xn,s(dyn,s|xn,s), and a fidelity set (3) The set Q0,n(D) is convex.
QSRD(D)=△ P (dyn,s|xn,s): Let BC(Y0,n) denotes the set of bounded continuous real-
0,n,s Yn,s|Xn,s valued functions on Y . Below, we introduce the main
n 0,n
1 n conditions for the existence of nonanticipative RDF (8).
E ρ (Xs,Ys)≤D (13)
n+1Xi=0 PXis,Yis s i i o Assumption 1. The following are assumed.
(A1) Y is a compact Polish space, X is a Polish
the information sequential RDF is defined by 0,n 0,n
space;
R0S,Rn,Ds(D)= inf (A2) For all h(·)∈BC(Y0,n), the function
PYn,s|Xn,s∈QS0,RnD,s(D) n mapping (xn,yn−1) ∈ X0,n × Y0,n−1 7→
(Xis+1,..1.,Xns)↔(XIi(,Xs,Yni,−s;1,Ys)n↔,sY)is,i.=0,1,...,n (14) RjoYinnthly(yin)qtnh(edyva;ryina−b1le,sx(nx)n,y∈n−1)R∈Xis0,nc×onYti0n,nu−ou1s;
(n+1)(s+1) o (A3) d0,n(xn,·) : Y0,n 7→ [0,∞) is continuous on Y0,n,
The sequential source coding theorem is the following. uniformly in xn ∈X0,n;
(A4) There exist sequence(xn,yn)∈X ×Y satisfy-
0,n 0,n
Theorem1. (SequentialSourceCodingTheorem[1])Suppose ing d (xn,yn)<D.
0,n
{X : i = 0,1,...,n,j = 0,1,...,s} are finite alphabets,
i,j
P (dxn,s) = ⊗s P(dxn), {Xn : j = 0,1,...,s} Thefollowingweakcompactnessresultcanbeobtained,which
Xn,s j=0 j j
will be used to establish existence.
identically distributed, and there exists an x and D >0
0 max
sju=ch0th,a1t,.E.P.,Xsi,.jρTsh(eXni,fjo,rxa0n)y<ǫD>m0axa,nfodrfianliltei=n 0∈,1N,,..th.,enre, Le(m1)ma 2T.he[4s]etSuQpCpo1s(eYAss;uXmpti)onis1w(eAa1kl)y, (cAo2m)phaocltd.. Then
0,n 0,n
exists s(ǫ,n) such that for all s≥s(ǫ,n) we have
(2) Under the additional conditions (A3), (A4) the set
−→
RSRD,o(D+ǫ)≤RSRD(D)+ǫ Q0,n(D) is a closed subset of QC1(Y0,n;X0,n)
0,n,s 0,n
(hence compact).
where
The next theorem establishes existence of the minimizing
RSRD(D)=△ inf I(Xn;Yn).reproductiondistribution for (8). First, we need the following
0,n PYn|Xn:n+11EPXn,Yn{Pni=0ρ(Xi,Yi)≤D} n+1 Lemma.
Xin+1↔(Xi,Yi−1)↔Yi: i=0,1,...,n Lemma 3. [7] Under Assumptions 1 (A1), (A2),
(15) −→
I (µ ,Q ) is lower semicontinuous on
NoticethatR0S,RnD(D)ispreciselyGorbunov-Pinsker’snonan- −→QX0n,→nY∈nQC01,n(Y0,n0,;nX0,n) for a fixed µ0,n ∈M1(X0,n).
ticipatory ǫ-entropy 1 Rna(D) [2]. Moreover,by Lemma1
−→n+1 0,n Next, we state the main Theorem.
RSRD(D)≡ 1 R (D)givenby(8).However,thecoding
0,n n+1 0,n
Theorem1 is validforfinite time index n, since its derivation Theorem3. (Existence[4])SupposeAssumption1hold.Then
−→
is based on taking s → ∞. Consequently, with respect R (D) has a minimum.
0,n
to the information sequential RDF (15), our objective is to
Thus, Theorem 3 implies the following results. By GP1,
find the expression of the nonstationary optimal reproduction
for a stationary source Rna(D) = Rna,+(D) is defined
distribution.
and it is finite, by GP2, for a stationary consistent source
IV. EXISTENCE OFREPRODUCTION CONDITIONAL limk→∞ k+11R0n,ak(D) = Rna(D) < ∞, and by GP3, for
DISTRIBUTION OFNONANTICIPATIVERDF a stationary source and single letter distortion, Rna(D) =
Rna,+(D) and the infimum in (12) is realized by stationary
In this section, the existence of the minimizing (n+1)-fold
source-reproductionpairs.
convolution of conditional distributions in (8) is established
by using the topology of weak convergence of probability V. OPTIMALREPRODUCTION OFNONANTICIPATIVERDF
measuresonPolishspaces.Infact,ourresultsaremoregeneral
In this section, we derive the expression of reproductioncon-
than what is envisioned by the assumptions in [1], [2], since −→
ditional distribution which achieves the infimum of R (D)
0,n
we work with abstract Polish spaces and general distortion or RSRD(D). We assume a distortion function of the form
functions. First, we recall some properties from [7]. 0,n
d (xn,yn)=△ n ρ (xi,yi). We shall need, the Gateaux
0,n i=0 0,i
P
differential of IXn→Yn(µ0,n,−→Q0,n) in every direction of For k =0,1,...,n, i=△ n−k, we get
e00{Tv,,qhe11ie(r,,doy..ry..ei−→..Q;m,,ynn0i,−})n41.→,=△∈x(iGI)µIQa:X0t,ieCnna=1(→uq(xiYY0(,n0·;1D,(n·,µ,e;.·0rX.),in.v0:,a,,nn→−tQii)}v.0=e(,)nTd)uh0Lee,en1btt,eo.{I.nqwµ.oi0,e(n,nnl·sl;()t·aq,dtii·ise)ofinG:na:eardititieya)fuo.==xr qqni∗∗((ddyyin;;yyin−−1,1x,xi)n=)=RYReiYseeρnss0,ρρei00s(,,xρni0(i(,x,xnyini(,)yx,−yin)ng,−y)i,νngnin)∗(,νnx(n∗d(ix,(yyidin,)yy;νiny)i∗;νn(yi∗−dn(y1d−i)y;1iy);iy−i1((−11)187))).
differentiable at every point in Q(Y ;Y ×X ), and the
i 0,i−1 0,i
Gateaux derivative at the points q∗(·;·,·) in each direction The nonanticipative RDF is given by
i
δqi =qi−qi∗, i=0,...,n, is −→ n
δIµ0,n(qi∗,qi−qi∗ :i=0,...,n) R0,n(D)=s(n+1)D−Xi=0Z (cid:18)Z gi,nqi∗(dyi;yi−1,xi)+
=Xi=n0Z log(cid:18)qi∗ν(i∗d(ydiy;iy;iy−i1−,1x)i)(cid:19)ddǫ−→Qǫ0,i(dyi|xi)(cid:12)(cid:12)ǫ=0µ0,i(dxi) logZ esρ0,i−gi,nνi∗(dyi;yi−1)(cid:19)×−→Q∗0,i−1(dyi−1|xi−1)µ0,i(dxi)
where −→Qǫ =△ ⊗i qǫ(dy ;yj−1,xj), qǫ =q∗+ǫ(cid:12) q −q∗ , Discussion. The above recursions illustrate the nonanticipa-
0,i j=0 j j j j j j tion, since g (xi,yi), i =n−k, k = 0,1,...,n, appearing
j =0,1,...,i, i=0,1,...,n, (cid:16) (cid:17) i,n
in the exponentof the reproductiondistribution (18) integrate
d −→
Qǫ(dy ;x ) =δq (dy ;x ) out future reproduction distributions. Note also that for the
dǫ 0 0 0 (cid:12)ǫ=0 0 0 0 stationary case all reproduction conditional distributions are
ddǫ−→Qǫ0,1(dy1|x1(cid:12)(cid:12))(cid:12)ǫ=0 =δq0(dy0;x0)⊗q1∗(dy1;y0,x1) tihsegsivaemnebaynd(1h7e)n.cTe,hegi,anb(o·,v·e) r=ecu0r,siwonhsichareimgpelineesraqln,∗(w·;h·,il·e)
(cid:12)(cid:12) +q0∗(dy0;x0)⊗δq1(dy1;y0,x1) depending on the assumptions imposed on the distortion
. function and source they can be simplified considerably.
.
.
d −→ VI. CONCLUSION
Qǫ (dyi|xi) =δq (dy ;x0)⊗i q∗(dy ;yj−1,xj)
dǫ 0,i (cid:12)ǫ=0 0 0 j=1 j j Inthispaper,wederiveananalyticalclosedformexpression
+q0∗(dy0;x0)δq(cid:12)(cid:12)1(dy1|y0,x1)⊗ij=2qj∗(dyj;yj−1,xj) for the nonanticipative RDF for nonstationary processes, and
werelatethedefinitionofnonanticipativeRDFtootherworks
...+⊗i−1q∗(dy ;yj−1,xj)⊗δq (dy ;yi−1,xi).
j=0 j j i i in the literature.
Proof: The derivation is lengthy, hence it is omitted.
The constrained problem defined by (8) can be reformulated REFERENCES
using Lagrange multipliers (due to its convexity) by utilizing [1] S. C. Tatikonda, “Control over communication constraints,” Ph.D. dis-
Lagrange Duality Theorem [10] to obtain sertation, Mass.Inst.ofTech.(M.I.T.),Cambridge, MA,2000.
[2] A.K.GorbunovandM.S.Pinsker,“Nonanticipatoryandprognosticep-
−→ −→
R (D)= inf I (µ ,Q ) silonentropies andmessagegeneration rates,” ProblemsofInformation
0,n →− →− Xn→Yn 0,n 0,n Transmission,vol.9,no.3,pp.184–191, July-Sept. 1973.
Q0,n∈Q0,n(D) [3] M. Gastpar, B. Rimoldi, and M. Vetterli, “To code, or not to code:
−→
=max inf I (µ ,Q ) Lossysource-channel communication revisited,” IEEETransactions on
s≤0 →−Q0,n∈→−Q0,n(D)n Xn→Yn 0,n 0,n [4] IPn.foArm.atSiotanvrTohueorayn,dvolC..49D,n.o.C5h,aprapl.am11b4o7u–s1,15“8N,oMnaanyti2c0ip0a3ti.ve rate
−→
−s ℓ (Q )−(n+1)D distortion function and filtering theory: A weak convergence
d0,n 0,n approach,” CoRR, vol. abs/1212.6643v1, 2013. [Online]. Available:
(cid:16) (cid:17)o
http://arxiv.org/abs/1212.6643v1.
where s∈(∞,0] is the Lagrange multiplier.
[5] J. L. Massey, “Causality, feedback and directed information,” in In-
Since qi(dy;yi−1,xi) ∈ Q(Yi;Y0,i × X0,i), one should in- ternational Symposium on Information Theory and its Applications
troduceanotherLagrangemultiplierto obtain an optimization (ISITA’90),Nov.27-301990,pp.303–305.
[6] M.S.DerpichandJ.Østergaard,“Improvedupperboundstothecausal
problem without constraints. This process is involved; hence
quadraticrate-distortionfunctionforgaussianstationarysources,”IEEE
we state the final results. TransactionsonInformationTheory,vol.58,no.5,pp.3131–3152,May
General Recursions of Optimal Non-stationary Reproduc- 2012.
[7] C.D.CharalambousandP.A.Stavrou,“Directedinformationonabstract
tion Distribution. The general recursions are the following.
spaces: properties and extremum problems,” in IEEE International
Define Symposium onInformationTheory(ISIT),July1-62012,pp.518–522.
[8] P.DupuisandR.S.Ellis,AWeakConvergenceApproachtotheTheory
gn,n(xn,yn)=0 ofLargeDeviations. JohnWiley&Sons,Inc.,NewYork,1997.
[9] T. Berger, Rate Distortion Theory: A Mathematical Basis for Data
gn−k,n(xn−k,yn−k)=− pn−k+1(dxn−k+1;xn−k) Compression. Englewood Cliffs,NJ:Prentice-Hall, 1971.
Z [10] D.G.Luenberger,OptimizationbyVectorSpaceMethods. JohnWiley
&Sons,Inc.,NewYork,1969.
log esρ0,n−k+1(xn−k+1,yn−k+1)−gn−k+1,n(xn−k+1,yn−k+1)
(cid:18)Z
×ν∗ (dy ;yn−k) , k=1,...,n. (16)
n−k+1 n−k+1 (cid:19)
