Cascade and Triangular Source Coding with Side Information at the First Two Nodes

0.37 MB·English

by  Haim Permuter

1
Cascade and Triangular Source Coding with
Side Information at the First Two Nodes
Haim Permuter and Tsachy Weissman
Abstract
We consider the cascade and triangular rate-distortion problem where side information is known to the source
0
1 encoder andtothefirstuserbutnottotheseconduser.Wecharacterizetherate-distortionregionfortheseproblems.
0
For the quadratic Gaussian case, we show that it is sufficient to consider jointly Gaussian distributions, a fact that
2
leads to an explicit solution.
n
a
J
Index Terms
1
1
Cascade sourcecoding, empirical coordination, quadratic Gaussian, Paretofrontier, source coding, side informa-

] tion, rate distortion, triangular source coding
T
I
.
s
c I. INTRODUCTION
[

Yamamoto [1] considered the cascade source coding problem, where a source sends a message to User 1, and
1
v then User 1 sends a message to User 2. In this paper, we extend Yamamoto’s cascade source coding problem to
9
7 the case where side information is known to the source and to User 1, but not to User 2. The problem is depicted
6
1 in Fig. 1.
.
1
0 Xˆ
1
0
1
:
v
i X Encoder R1 User 1 R2 User 2 a Xˆ
X 2
r
a
Y
Fig.1. Acascaderatedistortionproblemwiththreenodes(encoder,User1,User2),wherethefirsttwonodeshavesideinformationY.User
1andUser2needtoreconstruct thesourseX,withindistortion criteria.
More recently, Vasudevan, Tian and Diggavi [2] considered the cascade source coding problem, where side
information, Y, is known to the source encoder and to User 1, additional side information Z is known to User 2,
andtheMarkovchainX−Z−Y holds.Vasudevanetal.[2]providedaninnerandanouterboundandshowedthat
Author’s emails:[email protected], [email protected]
2
TABLEI
LITERATUREOVERVIEWOFCASCADESOURCECODINGWITHSIDEINFORMATIONASSHOWNINFIG.2
Switcha Switchb Switchc Gaussianquadratic case Generalcase
open open open Solved[1] Solved[1]
open open closed Solved[2] Upperandlowerbounds[2]
open closed open Upperandlowerbounds[3] Upperandlowerbounds[3]
open closed closed Solved[2] Upperandlowerbounds[2]
closed open open Solved[1] Solved[1]
closed open closed Solved[2] Upperandlowerbounds[2]
closed closed open Section IV Section II
the boundscoincide for the Gaussian case. Cuff, Su and El-Gammal[3] consideredthe cascade problemwhere the
side information is known only to the intermediate node and provided an inner and an outer bound. An additional
related problem, which was considered and solved in [4], is that of cascade source coding when side information
is known to all nodes with a limited rate. Table I summarizes the literature on cascade source coding with side
information.
Xˆ
1
X Encoder R1 User 1 R2 User 2 a Xˆ2
a b c
Y
Fig.2. Acascade ratedistortion problemwithseveraloptions ofsideinformation. TableIsummarizesthelietrtaure onthisproblem.
Of special interest in lossy source coding is the Gaussian case with quadratic distortion, which in many source
coding problems results in an analytical solution such as in the Wyner-Ziv problem [5] where side information is
available to the decoder, the Heegard-Berger problem [6] where side information at the decoder may be absent,
Kaspi’s problem [7], [8] where side information is known to the encoder and may or may not be known to the
decoder, the multiple description problem [9], [10], the two-way source coding problem [11], the multi-terminal
problem [12] [13], the CEO problem [14]–[16], rate distortion with a helper [17], [18], and successive refinement
[19] and its extension to successive refinement for the Wyner-Ziv problem [20].
Our main result in this paper is that the achievable region for the problem depicted in Fig. 1 is given by
R(D ,D ), which is defined as the set of all rate-pairs (R ,R ) that satisfy
1 2 1 2
R ≥ I(Y,X;Xˆ ), (1)
2 2
3
R ≥ I(X;Xˆ ,Xˆ |Y), (2)
1 1 2
for some joint distribution P(x,y)P(xˆ ,xˆ |x,y) for which
1 2
Ed (X,Xˆ ) ≤ D , i=1,2. (3)
i i i
An extension of the cascade source coding problem is the triangular setting [21], where there is an additional
direct link from the source encoder to User 2. We solve this problem where side information exists at the source
encoder and User 1, but not at User 2.
The remainder of the paper is organized as follows. In Section II, we formally define the cascade problem and
present the theorem establishing the achievable region. In Section III, we provide a converse and achievability
proofsof the theorem,and in Section IV we explicitlycompute the rate regionfor the Gaussian case. In Section V
we extend our result to the triangular case (cf. Fig. 5), and in Section VI we further extend the results to multiple
users and discuss the corresponding empirical coordination problem.
II. CASCADE RATE DISTORTION:PROBLEM DEFINITIONSAND MAIN RESULTS
Here we formally define the cascade rate-distortion problem where side information is known to the source
encoder and to User 1. We present a single-letter characterization of the achievable region. We use the regular
definitions of rate distortion, and we follow the notation of [22]. The source sequences {X ∈ X, i = 1,2,···},
i
andthesideinformationsequence{Y ∈Y, i=1,2,···}arediscreterandomvariablesdrawnfromfinitealphabets
i
X and Y, respectively. The random variables (X ,Y ) are i.i.d. ∼ P(x,y). Let Xˆ and Xˆ be the reconstruction
i i 1 2
alphabets, and d : X ×Xˆ →[0,∞), i=1,2, are single letter distortion measures. Distortion between sequences
i i
is defined in the usual way
n
1
d (xn,xˆn) = d (x ,xˆ ), i=1,2. (4)
i i n i j i,j
j=1
X
Let M denote a set of positive integers {1,2,..,M } for i=1,2.
i i
Definition 1 (Cascade rate distortion code with side information at the first two nodes): An
(n,M ,M ,D ,D ) code for source X and side information Y consists of two encoders
1 2 1 2
f : Xn×Yn →M
1 1
f : Yn×M →M (5)
2 1 2
and two decoders
g : Yn×M →Xˆn
1 1 1
g : M →Xˆn (6)
2 2 2
such that
n
1
E d (X ,Xˆ ) ≤ D , j =1,2 (7)
j i j,i j
n
" #
i=1
X
4
The rate pair (R ,R ) of the (n,M ,M ,D ,D ) code is defined by
1 2 1 2 1 2
1
R = logM ; i=1,2. (8)
i i
n
Definition 2: Given a distortion pair (D ,D ), a rate pair (R ,R ) is said to be achievable if, for any ǫ > 0,
1 2 1 2
andsufficientlylargen,thereexistsan(n,2nR1,2nR2,D +ǫ,D +ǫ)codeforthesourceX with sideinformation
1 2
Y.
Definition 3: The (operational) achievable region RO(D ,D ) of cascade rate distortion is the closure of the
1 2
set of all achievable rate pairs.
Theorem 1 is the main result of this work.
Theorem 1: For the cascade rate distortion problem with side information at the source and User 1, as depicted
in Fig. 1, the achievable region is given by
RO(D ,D )=R(D ,D ), (9)
1 2 1 2
where the region R(D ,D ) is defined in (1)-(3).
1 2
III. PROOF OF THEOREM1
Achievability: The proof follows classical arguments, and therefore the technical details will be omitted. We
describe only the coding structure and justify why the indicated region is achievable. We fix a joint distribution
P for which (3) holds, and an ǫ>0, and we show that there exists a code with rates
X,Y,Xˆ1,Xˆ2
R = I(Y,X;Xˆ )+ǫ, (10)
2 2
R = I(X;Xˆ ,Xˆ |Y)+3ǫ, (11)
1 1 2
complying with the distortion constraints.
Generate randomly 2n(I(X,Y;Xˆ2)+ǫ) codewords using an i.i.d. ∼ P . Then bin the codewords into
Xˆ2
2n(I(X;Xˆ2|Y)+2ǫ) bins. In each bin, there are 2n(I(X,Y;Xˆ2)−I(X;Xˆ2|Y)−ǫ) = 2n(I(Y;Xˆ2)−ǫ) codewords. In addi-
tion, for any typical sequences yn,xˆn generate 2n(I(X;Xˆ1|Y,Xˆ2)+ǫ) codewords using the pmf P(xˆn|yn,xˆn) =
2 1 2
n P (xˆ |y ,xˆ ).
i=1 Xˆ1|Y,Xˆ2 1,i i 2,i
QThe source-encoder receives the sequences xn,yn and first looks for a codeword xˆn that is jointly typical with
2
xn,yn. If there is such a codeword, the source encoder sends the index of the bin that includes this codeword to
User1.User 1lookswhichcodewordinthereceivedbinisjointlytypicalwith thesideinformationyn.Sincethere
are less than 2n(I(Y;Xˆ2) in the bin, with high probability only one codeword will be jointly typical with yn and it
would be the codeword sent by the encoder. User 1 then forwards the codeword to User 2.
Now we can think of a new problem where the source-encoder and User 1 have side information Yn,Xˆn and
2
hence a rate I(X;Xˆ |Y,Xˆ )+ǫ is needed to generate Xˆn that is jointly typical with (Xn,Yn,Xˆ ). Therefore, a
1 2 1 2
total rate to User 1 of R = I(X;Xˆ |Y)+2ǫ+I(X;Xˆ |Y,Xˆ )+ǫ = I(X;Xˆ ,Xˆ |Y)+3ǫ is needed, and an
1 2 1 2 1 2
additional rate R =I(Y,X;Xˆ )+ǫ is needed from User 1 to User 2.
2 2
5
Converse: Assume that we have an (n,M =2nR1,M =2nR2,D ,D ) code as in Definition 1. We will show
1 2 1 2
the existenceof a jointdistributionP thatsatisfies (1)-(3). Denote T =f (Xn,Yn)∈{1,...,2nR1}, and
X,Y,Xˆ1,Xˆ2 1 1
T =f (T ,Yn)∈{1,...,2nR2}. Then,
2 2 1
nR ≥ H(T )
2 2
≥ I(Xn,Yn;T )
2
n
= H(X ,Y )−H(X ,Y |T ,Xi−1,Yi−1)
i i i i 2
i=1
X
n
(=a) H(X ,Y )−H(X ,Y |Xˆ ,T ,Xi−1,Yi−1)
i i i i 2,i 2
i=1
X
n
≥ I(X,Y;Xˆ ), (12)
2,i
i=1
X
where equality (a) follows from the fact that the reconstruction at time i, Xˆ , is a deterministic function of T .
2,i 2
Now consider
nR ≥ H(T )
1 1
≥ H(T |Yn)
1
(=a) H(T ,T |Yn)
1 2
≥ I(Xn;T ,T |Yn)
1 2
n
= H(X |Y )−H(X |Yn,T ,T ,Xi−1)
i i i 1 2
i=1
X
n
(=b) H(X |Y )−H(X |Yn,T ,T ,Xi−1,Xˆ ,Xˆ )
i i i 1 2 1,i 2,i
i=1
X
n
≥ H(X |Y )−H(X |Y ,Xˆ ,Xˆ )
i i i i 1,i 2,i
i=1
X
n
= I(X ;Xˆ ,Xˆ |Y ), (13)
i 1,i 2,i i
i=1
X
where equality (a) follows from the fact that T is a deterministic function of T and Yn, and, similarly, equality
2 1
(b) follows from the fact that Xˆ and Xˆ are deterministic functions of (T ,Yn) and T , respectively.
1,i 2,i 1 2
TheproofisconcludedinthestandardwaybylettingQbearandomvariableindependentofXn,Yn,uniformly
distributed over the set {1,2,3,..,n}, and considering the joint distribution of X ,Y ,Xˆ ,Xˆ . For this joint
Q Q 1,Q 2,Q
distribution, inequalities (12) and (13) imply that (1) and (2) hold, respectively, and (7) implies that (3) holds.
IV. CASCADE RATEDISTORTION:THE GAUSSIAN CASE
InthissectionweexplicitlycalculatetherateregionR(D ,D )forthecaseswhereX andY arejointlyGaussian
1 2
andthedistortionisthesquare-errordistortion.Theconverseandtheachievabilityintheprevioussectionsareproved
for the finite alphabet case, but it can be extended to the Gaussian case [5].
6
OurfirststepinfindingtheachievableregionforthequadraticGaussiancaseistoshowthatitsufficestoconsider
only jointly Gaussian distributions P in order to exhaust the rate region. Then we solve an optimization
X,Y,Xˆ1,Xˆ2
problem to find the achievable rate-region explicitly.
Lemma 2 (Optimality of jointly Gaussian distributions): ForthequadraticGaussiancascaderate-distortionprob-
lem with side information known to the source-encoder and to User 1, i.e., X,Y are jointly Gaussian and
d (x,xˆ )=(x−xˆ )2, d (x,xˆ )=(x−xˆ )2, itsufficesto consideronlyjointlyGaussian distributionsP
1 1 1 2 2 2 X,Y,Xˆ1,Xˆ2
in order to exhaust the rate region R(D ,D ) given in (1)-(3).
1 2
Proof: Let us fix a point (R ,R ,D ,D ) in the rate region and let P be a joint distribution that
1 2 1 2 X,Y,Xˆ1,Xˆ2
satisfies (1)-(3). Such a distribution must exist since Inequalities (1)-(3) define the rate region (Theorem 1). Let
K denote the covariance matrix induced by P and let P˜ denote a normal joint distribution
X,Y,Xˆ1,Xˆ2 X,Y,Xˆ1,Xˆ2
with mean zero and covariance matrix K. Now let us show that (1)-(3) also hold where the joint distribution is
P˜ . Inequality (3) is automatically satisfied, since it depends on the distribution of (X,Y,Xˆ ,Xˆ ) only
X,Y,Xˆ1,Xˆ2 1 2
through the covariance matrix K. Consider,
R ≥ I(X;Xˆ ,Xˆ |Y),
1 1 2
= h(X|Y)−h(X|Xˆ ,Xˆ ,Y),
1 2
(=a) h(X|Y)−h(X −(α Xˆ +α Xˆ +α Y)|Xˆ ,Xˆ ,Y),
1 1 2 2 3 1 2
(b)
≥ h(X|Y)−h(X −(α Xˆ +α Xˆ +α Y))
1 1 2 2 3
(c)
≥ h(X|Y)−h (X −(α Xˆ +α Xˆ +α Y))
P˜ 1 1 2 2 3
(=d) I (X;Xˆ ,Xˆ |Y), (14)
P˜ 1 2
equality(a)istrueforanysetofscalars(α ,α ,α )andinparticularifwechoosethosethatarethelinearestimator
1 2 3
ofX givenXˆ ,Xˆ ,Y.Notethatthecoefficients(α ,α ,α )andthevarianceE(X−(α Xˆ +α Xˆ +α Y))2 are
1 2 1 2 3 1 1 2 2 3
a function only of the covariancematrix K. Inequality (b) follows from the fact that conditioningreduces entropy,
and (c) follows from the fact that, given a variance, the Gaussian distribution maximizes the differential entropy.
The term I (X;Xˆ ,Xˆ |Y) denotes the mutual information induced by the Gaussian distribution P˜ , and
P˜ 1 2 X,Y,Xˆ1,Xˆ2
equality (d) follows from the fact that for the Gaussian distribution the error, i.e., X−(α Xˆ +α Xˆ +α Y), is
1 1 2 2 3
independent of the observations Xˆ ,Xˆ ,Y.
1 2
Similarly, we have
R ≥ I(Y,X;Xˆ )
2 2
= I(Y;Xˆ )+I(X;Xˆ |Y)
2 2
≥ I (Y;Xˆ )+I (X;Xˆ |Y), (15)
P˜ 2 P˜ 2
where the last inequality follows from the same steps as (14).
The next theorem provides an explicit expression for the Gaussian case. The proof is provided in Appendix A
andisbased onLemma2 andonsolvinganoptimizationproblemwithquadraticconstraintsanda linearobjective.
7
Theorem 3 (Cascade Gaussian case): The rate region of the cascade source coding with side information at the
first two nodes, where the source X and the side information Y =X +Z are jointly Gaussian distributed, where
X and Z are mutually independent, and the distortion is quadratic, is given by
1 σ2 σ2
X|Y X|Y
R (D ,D ,R )= max log ,log ,0 , (16)
1 1 2 2 2 σ2 D
X|W,Y 1 !
where σ2 is given by the following four cases
X|W,Y
22R2D2−σX2 +σ−2 −1, if D ≤σ2 and σX2 ≤22R2 ≤ σZ2(σX2−D2) σX2
σZ2σX2α2 X|Y 2 X|Y D2 σZ2σX2−D2σZ2−D2σX2 D2
σX2|W,Y(D1,D2,R2)= (cid:16)D222,σRZ22DσX22−ασ2X2 +σX−2|Y(cid:17)−1, iiff DD22 ≤≥σσXX22||YY aanndd 2σD2X22R2≤≥22σRZ22σX2≤σZ2−(σDσX2X22σD−Z22D−+2Dσ)Zσ22X4DσX22−σDσX22X2σZ2
(cid:16)σ2 , (cid:17) if D ≥σ2 , and 22R2 ≥ σX4
and α= σZ σX2−D2 X−|Y1 −1. 2 X|Y σX2D2+σZ2D2−σX2σZ2 (17)
(cid:18)σXrD2−σX22−2R2 (cid:19)
Fig. 3 depicts the regions for two specific values of D and D such that it captures all four cases of Eq. (17).
1 2
12lo↓gσDX22 12lo↓gσZ2σX2σZ2−(DσX22σ−Z2D−2D)2σX2 σDX22 21lo↓gσDX22 12lo↓gσX2D2+σZσ2X4D2−σX2σZ2
Case (a) Case (b) 0.12 Case (c) Case (d)
0.4
0.1
0.08
0.35
1 1
R R0.06
0.3 0.04
1max logσX2|Y,logσX2|Y −→ 0.02
2 (cid:18) D2 D1 (cid:19)0.25 0 ←−1max logσX2|Y,logσX2|Y
1 R 1.5 2 0.5 R 1 1.5 2 (cid:18) D2 D1 (cid:19)
2 2
Fig.3. TheGaussianquadraticrateregion.ThegraphonthelefthandsideshowstherateregionforthecasewhereσX2 =σZ2 =1,D2=0.35
andD1=0.4.SinceD2<σX2|Y,therateregionisgivenbyCases (a)and(b)inEq.(17).Therighthandsidegraphshowstherateregion
forthecasewhereσX2 =σZ2 =1,D2=0.65andD1=0.5.SinceD2>σX2|Y,therateregionisgivenbyCases (c)and(d)inEq.(17)
Now, let us consider several extreme cases that can be easily solved using Theorem 3.
1) Side information is independent of the source X ⊥Y: This means that σ2 =σ2 and σ2 =∞. For such
X|Y X Z
a case (17) becomes
σ2 , if D ≤σ2 and σX2 ≤22R2 ≤ σX2
X 2 X D2 D2
σX2|W,Y(D1,D2,R2)= D∞2,, iiff DD2 ≤≥σσX22 ,aanndd222R2R22≥≥σD0X22 (18)
2 X

8
and this implies that
1 σ2 σ2
R (D ,D ,R )= max log X,log X|Y ,0 , (19)
1 1 2 2
2 D D
2 1 !
recovering a result that appears in the successive refinement source coding paper [19].
2) Side information equals the source, i.e., X = Y: For this case, σ2 = 0; hence R = 0 and 22R2 ≥ σX2 ,
X|Y 1 D2
consistent with the well known rate distortion function of the Gaussian source.
3) R →∞: If D ≤σ2 then
2 2 X|Y
1 σ2 σ2
X|Y X|Y
R (D ,D ,R )= max log ,log ,0 , (20)
1 1 2 2
2 D D
2 1 !
and if D ≥σ2
2 X|Y
1 σ2
X|Y
R (D ,D ,R )= max log ,0 . (21)
1 1 2 2
2 D
1 !
Note that for this case we can assume that the side information Y is known to all three nodes; hence only σ2
X|Y
is manifested in the expression.
4) ThemessagethatUser2receivesdependsonlyontheside information: Inthisextremecase, therateR and
2
the distortion D are large enough so that the message that User 2 receives depends only on the side information.
2
This case is depicted in Fig. 4.
Xˆ
1
X Encoder R1 User 1
Encoder R2 User 2 a Xˆ
2
Y
Fig.4. AnextremecasewheretherateR2 andthedistortionsD2 arelargeenoughsothatthemessagethatUser2receivesdependsonlyon
thesideinformation.
For this extreme, the rate region is simply
R ≥ I(X;Xˆ |Y),
1 1
R ≥ I(Y;Xˆ ), (22)
2 2
for all joint Gaussian distributions that satisfy σ2 ≤D and σ2 ≤D .
X|Y,Xˆ1 1 X|Xˆ2 2
More explicitly, this region is given by
σ2 (σ2 2−2R2 +σ2)
D ≥ X X Z (23)
2 σ2 +σ2
X Z
1 σ2
R ≥ max log X|Y ,0 . (24)
1
2 D
1 !
9
Indeed, if (23) holds, then according to Theorem 3, R (D ,D ,R )= 1max logσX2|Y ,0 .
1 1 2 2 2 D1
(cid:18) (cid:19)
V. TRIANGULARSOURCE CODING WITH SIDE INFORMATION
In this section, we extend the cascade source coding discussed in previous sections by adding a direct link from
theencoderto theseconduser,as depictedinFig. 5. Thedefinitionofthecode(n,M ,M ,M ,D ,D )issimilar
1 2 3 1 2
to the one given in Def. 1 for the cascade case, with an additional message M at rate R sent from the source to
3 3
User 2.
Y Y
X Encoder R1 User 1 Xˆ a
1
R
2
R
3
User 2
Xˆ
2
Fig. 5. A triangular rate distortion problem with three nodes (encoder, User 1, User2), where side information Y is known to the encoder
andUser1,butnottoUser2.User1andUser2needtoreconstruct thesourseX towithindistortion criteria.
A. Main theorem and its proof
Theorem 4 (The achievable rate region for the triangular case): Theachievableregionfortheproblemdepicted
in Fig. 5 is given by R (D ,D ), which is defined as the set of all rate-triples (R ,R ,R ) that satisfy
∆ 1 2 1 2 3
R ≥ I(X;Xˆ ,U|Y), (25)
1 1
R ≥ I(Y,X;U), (26)
2
R ≥ I(X;Xˆ |U), (27)
3 2
for some joint distribution P(x,y)P(xˆ ,xˆ ,u|x,y) satisfying
1 2
Ed (X,Xˆ ) ≤ D , i=1,2, (28)
i i i
where the cardinality of the auxiliary variable U may be bounded by |U|≤|X||Y||Xˆ ||Xˆ |+2.
1 2
Lemma 5 below shows that one can restrict the joint distribution P(x,y)P(xˆ ,xˆ ,u|x,y) to
1 2
P(x,y)P(xˆ ,u|x,y)P(xˆ |x,u) without affecting the region.
1 2
10
ProofofConversePartofTheorem4:Assumethatwehavean(n,2nR1,2nR2,2nR3,D ,D )code.Wewillshow
1 2
the existenceof a jointdistributionP thatsatisfies (25)-(28). DenoteT =f (Xn,Yn)∈{1,...,2nR1},
X,Y,U,Xˆ1,Xˆ2 1 1
and T =f (T ,Yn)∈{1,...,2nR2}, and T =f (Xn,Yn)∈{1,...,2nR3}. Then,
2 2 1 3 3
nR ≥ H(T )
1 1
≥ H(T |Yn)
1
(=a) H(T ,T |Yn)
1 2
≥ I(Xn;T ,T |Yn)
1 2
n
= H(X |Y )−H(X |Yn,T ,T ,Xi−1)
i i i 1 2
i=1
X
n
(=b) H(X |Y )−H(X |Yn,T ,T ,Xi−1,Xˆ ,U )
i i i 1 2 1,i i
i=1
X
n
≥ H(X |Y )−H(X |Y ,Xˆ ,U )
i i i i 1,i i
i=1
X
n
= I(X ;Xˆ ,U |Y ), (29)
i 1,i i i
i=1
X
whereequality(a)followsfromthefactthatT isadeterministicfunctionofT andYn,and,similarly,equality(b)
2 1
follows from the fact that Xˆ is a deterministic function of (T ,Yn) and from defining Uˆ , (T ,Xi−1,Yi−1).
1,i 1 i 2
Now, consider
nR ≥ H(T )
2 2
≥ I(Xn,Yn;T )
2
n
= H(X ,Y )−H(X ,Y |T ,Xi−1,Yi−1)
i i i i 2
i=1
X
n
(a)
= H(X ,Y )−H(X ,Y |U )
i i i i i
i=1
X
n
≥ I(X,Y;U ), (30)
i
i=1
X
where equality (a) follows from definition of U =(T ,Xi−1,Yi−1). In addition, consider
i 2
nR ≥ H(T )
3 3
≥ H(T |T )
3 2
≥ I(Xn,Yn;T |T )
3 2
n
= H(X ,Y |T ,Xi−1,Yi−1)−H(X ,Y |T ,T ,Xi−1,Yi−1)
i i 2 i i 2 3
i=1
X
n
(=a) H(X ,Y |U )−H(X ,Y |Xˆ ,U )
i i i i i 2,i i
i=1
X