Table Of Content1
Strong Functional Representation Lemma and
Applications to Coding Theorems
Cheuk Ting Li and Abbas El Gamal
Department of Electrical Engineering, Stanford University
Email: [email protected], [email protected]
Abstract
This paper shows that for any random variables X and Y, it is possible to represent Y as a function of (X,Z) such that
7 Z is independent of X and I(X;Z|Y) ≤ log(I(X;Y)+1)+4. We use this strong functional representation lemma (SFRL)
1 to establish a tighter bound on the rate needed for one-shot exact channel simulation than was previously established by Harsha
0 et. al., and to establish achievability results for one-shot variable-length lossy source coding, multiple description coding and
2 Gray-Wynersystem.WealsoshowthattheSFRLcanbeusedtoreducethechannel withstatenoncausally knownattheencoder
to a point-to-point channel, which provides a simple achievability proof of the Gelfand-Pinsker theorem. Finally we present an
n
example in which the SFRLinequality is tight to within 5 bits.
a
J
Index Terms
4
2 Functional representation lemma, channel simulation, one-shot achievability, lossy source coding, channel with state.
]
T I. INTRODUCTION
I Thefunctionalrepresentationlemma[1,p.626]statesthatforanyrandomvariablesX andY,thereexistsarandomvariable
.
s Z independentof X such that Y can be represented as a function of X and Z. This result has been used to establish several
c
[ results in network information theory beginning with the early work of Hajek and Pursley on the broadcast channel [2] and
WillemsandvanderMeulenonthe multipleaccess channelwith cribbingencoders[3].In thispaper,westrengthenthisresult
2
by showing that for any X and Y, there exists a Z independent of X such that Y is a function of X and Z, and
v
7
I(X;Z|Y)≤log(I(X;Y)+1)+4.
2
8 We use this strong functional representation lemma (SFRL) together with an optimal prefix code such as a Huffman code
2
to establish one-shot, variable-length coding results for channel simulation [4], Shannon’s lossy source coding [5], multiple
0
descriptioncoding[6], [7] and lossy Gray–Wynersystem [8]. We then show how the SFRL can be used to reduce the channel
.
1 with state known at the encoderto a point-to-pointchannel, providinga simple proofto the Gelfand-Pinskertheorem [9]. The
0
asymptotic block coding counterparts of these one-shot results can be readily obtained by convertingthe variable-length code
7
into a block code and incurring an error probability that vanishes as the block length approaches infinity.
1
: A weaker form of the SFRL can be obtained using the result by Harsha et. al. [4] on the one-shotexact channelsimulation
v
with unlimited common randomness. Assuming the input X has a given pmf, then [4] implies that
i
X
I(X;Z|Y)≤(1+ǫ)log(I(X;Y)+1)+c
ǫ
r
a
is achievable,whereǫ>0 and c is a functionof ǫ. This resultwas established using a rejectionsamplingscheme and applies
ǫ
onlytodiscreterandomvariables.IncomparisonSFRLusesaconstructionthatwerefertoasPoissonfunctionalrepresentation
which provides a tighter bound and applies to arbitrary random variables.
One-shot achievability results using fixed length (random) coding have been recently established for lossy source coding
and several setting in network information theory. In [10], Liu, Cuff and Verdú established a one-shot achievability result for
lossy source coding using channel resolvability. One-shot quantum lossy source coding settings were investigated by Datta
et. al. [11]. In [12], Verdú introduced non-asymptotic packing and covering lemmas and used them to establish one-shot
achievability results for several settings including Gelfand-Pinsker. In [13], Liu, Cuff and Verdú proved a one-shot mutual
covering lemma and used it to establish a one-shotachievability result for the broadcast channel. In [14], Watanabe, Kuzuoka
andTanestablishedseveralone-shotachievabilityresultsforcodingwithside-information(includingGelfand-Pinsker).In[15],
Yassaee, ArefandGohariestablishedseveralone-shotachievabilityresults, includingGelfand-Pinskerandmultipledescription
coding. Most of these results are stated in terms of information density and various other quantities. In contrast, our one-shot
achievabilityresultsusing variable-lengthcodesare all stated in terms onlyof mutualinformation.Moreover,giventhe SFRL,
our proofs are generally simpler.
Variable-length (asymptotic or finite blocklength) lossy source coding settings have been studied, e.g., see [16], [17], [18],
[19]. Most of these works concern the universal setting in which the distribution of the source is unknown, hence the use of
variable-length codes is justified. In contrast, the reason we consider variable-length code in this paper is that it allows us to
give one-shot results that subsume their asymptotic fixed-length counterparts.
2
Inthefollowingsection,westatetheSFRLandproveitfordiscreterandomvariableY.InSectionsIIIandIVweuseSFRL
to establish one-shot achievability results for channel simulation and three source coding settings, respectively. In Section V,
we use SFRL together with Shannon’schannelcodingtheorem to providea simple achievabilityproofof the Gelfand–Pinsker
theorem. In Section VI we extend the proof of the SFRL to general random variables. Finally in Section VII we demonstrate
the tightness of the SFRL inequality and discuss several other properties of this inequality.
Notation
Throughout this paper, we assume that log is base 2 and the entropy H is in bits. The binary entropy function is H(x)=
−xlogx−(1−x)log(1−x). We use the notation: Xb =(X ,...,X ), Xn =Xn, [a:b]=[a,b]∩Z and [a]=[1:a].
a a b 1
For discrete X, we write the probability mass function as p . For continuousX, we write the probability density function
X
as f . For general random variable X, we write the probability measure (push-forward measure by X) as P .
X X
II. STRONG FUNCTIONAL REPRESENTATIONLEMMA
The main result in this paper is given in the following.
Theorem 1 (Strong functional representation lemma). For any pair of random variables (X,Y)∼P with I(X;Y)<∞,
XY
there exists a random variable Z independentof X such that Y can be expressed as a function g(X,Z) of X and Z, and
I(X;Z|Y)≤log(I(X;Y)+1)+4.
Moreover, if X and Y are discrete with cardinalities |X| and |Y|, respectively, then |Z|≤|X|(|Y|−1)+2.
Note that SFRL can be applied conditionally; given P , we can represent Y as a function g(X,Z,U) such that Z is
XY|U
independent of (X,U) and
I(X;Z|Y,U)≤log(I(X;Y|U)+1)+4.
ThereasonwecanhaveaZ independentofU isthatbythefunctionalrepresentationlemma,wecanrepresentZ asafunction
of U and Z˜ such that Z˜ is independentof U and use Z˜ instead of Z.
Note that SFRL applies to general distributions P . Although H(Y) may be infinite, the cardinality of Y conditioned
XY
on Z can still be countable and H(Y|Z) can be finite. Note that Z ⊥⊥ X and H(Y|X,Z) = 0 imply that I(X;Z|Y) =
H(Y|Z)−I(X;Y). Hence the SFRL implies the existence of a Z ⊥⊥X such that H(Y|Z) is close to I(X;Y).
For simplicity of presentation, we first prove the SFRL for discrete Y. The proof is extended to general Y in Section VI.
To prove the SFRL, we use the following random variable Z and function g construction.
Definition 1 (Exponential functional representation). Let X and Y be random variables, where Y ∈ {1,...,|Y|} and |Y| is
finite or countablyinfinite. The exponentialfunctionalrepresentationof Y given X is defined as Y =g (X,Z|Y|), where
X→Y
Z|Y| is a sequence of i.i.d. Exp(1) random variables independent of X, and
z
g (x,z|Y|)=argmin y .
X→Y
p (y|x)
y∈Y Y|X
Notethatif|Y|isfinite,wecangenerateZ|Y| uniformlyovertheprobabilitysimplexonY.Thisisequivalenttotheoriginal
scheme after normalization such that Z = 1. We now proceed to prove Theorem 1 for discrete Y by showing that the
y y
exponential functional representation satisfies the constraints.
P
Proof: Let Φ =Z /p (y),
y y Y
Z
y
Θ=inf ,
y pY|X(y|x)
and K be the indexof Φ in {Φ } sorted in ascendingorder(hence |{y : Φ <Φ }|=K−1 with probability1). Since
Y y y∈Y y Y
Y is a function of Z|Y| and K, we have H(Y|Z|Y|) ≤ H(K). We now proceed to bound H(K). Let r(x,y) = pY|X(y|x).
pY(y)
Since Θ|{X =x,Y =y}∼Exp(1),
E[logK|X =x]= p (y|x)E[logK|X =x,Y =y]
Y|X
y
X
∞
= p(y|x) e−θE[logK|X =x,Y =y,Θ=θ]dθ.
ˆ
y 0
X
Consider
E[logK|X =x,Y =y,Θ=θ]
θp(y|x) Z
=E log y′ 6=y : Φ < +1 y′ ≥θ for ally′
y′ p(y) p(y′|x)
(cid:20) (cid:18)(cid:12)(cid:26) (cid:27)(cid:12) (cid:19) (cid:12) (cid:21)
(cid:12) (cid:12) (cid:12)
(cid:12) (cid:12) (cid:12)
(cid:12) (cid:12) (cid:12)
3
θp(y|x) Z
≤log E y′ 6=y : Φ < y′ ≥θ for ally′ +1
y′ p(y) p(y′|x)
(cid:18) (cid:20)(cid:12)(cid:26) (cid:27)(cid:12) (cid:12) (cid:21) (cid:19)
(cid:12) (cid:12) (cid:12)
(cid:12) θp(y|x) Z (cid:12) (cid:12)
=log (cid:12) P Φ < y′(cid:12) (cid:12)≥θ +1
y′ p(y) p(y′|x)
y′6=y (cid:26) (cid:12) (cid:27)
X (cid:12)
(cid:12)
θp(y|x)p(cid:12)(y′)
=log P Z < Z ≥θp(y′|x) +1
y′ y′
p(y)
y′6=y (cid:26) (cid:12) (cid:27)
X (cid:12)
(cid:12)
(cid:12)
=log (1−exp(−θp(y′)(r(x,y)−r(x,y′))))+1
y′:r(x,Xy′)<r(x,y)
≤log θp(y′)(r(x,y)−r(x,y′))+1
y′:r(x,Xy′)<r(x,y)
≤log(θr(x,y)+1).
Hence
∞
E[logK|X =x]≤ p(y|x) e−θlog(θr(x,y)+1)dθ
ˆ
y 0
X
≤ p(y|x)log(r(x,y)+1)
y
X
= p(y|x)log(r(x,y)+1)+ p(y|x)log(r(x,y)+1)
y:r(Xx,y)≥1 y:r(Xx,y)<1
≤ p(y|x)(logr(x,y)+1)+ p(y|x)
y:r(Xx,y)≥1 y:r(Xx,y)<1
= p(y|x)logr(x,y)+1
y:r(Xx,y)≥1
=D(p (·|x)kp )− p(y|x)logr(x,y)+1
Y|X Y
y:r(Xx,y)<1
≤D(p (·|x)kp )+e−1loge+1,
Y|X Y
where the last inequality follows from Appendix A in [4]. Therefore E[logK]≤I(X;Y)+e−1loge+1. Note that by the
maximum entropy distribution subject to a given E[logK], we have
H(K)≤E[logK]+log(E[logK]+1)+1.
The proof of this bound is given in the Appendix for the sake of completeness. Hence
H(K)≤I(X;Y)+e−1loge+2+log I(X;Y)+e−1loge+2
≤I(X;Y)+log(I(X;Y)+1)+e−1loge+2+log e−1loge+2
(cid:0) (cid:1)
<I(X;Y)+log(I(X;Y)+1)+4.
(cid:0) (cid:1)
To prove the cardinality bound, first note that if |X|, |Y| are finite, then |Z|≤ |Y||X| can be assumed to be finite since it is
the number of differentfunctions x7→g (x,z) for different z. To further reduce the cardinality, we apply Carathéodory’s
X→Y
theorem on the (|X|(|Y|−1)+1)-dimensional vectors with entries H(Y|Z = z) and p(x,y|z) for x ∈ {1,...,|X|}, y ∈
{1,...,|Y|−1}; see [20], [21].
III. ONE-SHOT CHANNEL SIMULATION
Channel simulation aims to find the minimum amount of communication over a noiseless channel needed to simulate a
memorylesschannelP . Severalsettingsof this problemhave been studied,e.g., see [22], [23], [24]. Consider the one-shot
Y|X
channelsimulationwithunlimitedcommonrandomnesssetup[4]inwhichAliceandBobshareunlimitedcommonrandomness
W. Alice observes X ∼P and sends a prefix-freedescription M to Bob via a noiseless channelsuch that Bob can generate
X
Y (from M and W) according to a prescribed conditional distribution P . The problem is to find the minimum expected
Y|X
description length of M, E[L(M)], needed. It is straightforward to show that E[L(M)] ≥ I(X;Y). In [4], it is shown that
for X and Y discrete,
E[L(M)]≤I(X;Y)+(1+ǫ)log(I(X;Y)+1)+c
ǫ
4
is achievable for ǫ>0, where c is a function of ǫ.
ǫ
We now show that the SFRL provides a tighter upper bound on E[L(M)] that applies to arbitrary (not only discrete)
memoryless channels. By the SFRL, there exists a Z independent of X such that Y =g (X,Z) and
X→Y
H(Y|Z)≤I(X;Y)+log(I(X;Y)+1)+4.
We use W = Z as the common randomness. Upon observing X = x, Alice generates Y ∼ P (·|x) and encodes Y using
Y|X
a Huffman code for the pmf p (·|Z) into the description M (note that Y can be arbitrary but by the SFRL Y|{Z =z} is
Y|Z
discrete). Bob then recovers Y from M and Z. The expected length is
E[L(M)]≤I(X;Y)+log(I(X;Y)+1)+5.
In Section VII, we give an example which shows that the log(I(X;Y)+1) term is in general necessary.
Moreover,fordiscreteX,Y,theamountofthecommonrandomnesscanbeboundedbylog|W|≤log(|X|(|Y|−1)+2).In
comparison,the amountofthe commonrandomnessin[4]can beboundedby O(log(|X||Y|)) onlyif the expecteddescription
length is increased by O(loglog(|X|+|Y|)).
We can use the exponential functional representation to construct W =Z|Y| (if |W| is unlimited). Upon observing W and
X,AlicegeneratesY ∼P andfindsK,theindexofZ /p (Y)intheset{Z /p (y)} sortedinascendingorderasin
Y|X Y Y y Y y∈Y
the proof of the SFRL, and encodes K into M using the optimal prefix-free code for the power-law distribution q(k)∝k−λ,
k =1,2,..., where λ=1+1/(I(X;Y)+e−1loge+1). Bob recovers K from M and obtains Y using K and Z|Y|.
Remark 1. In [4], the setting in which X =x is an arbitrary input (instead of X ∼p ) is studied. It is shown that
X
E[L(M)]≤C+(1+ǫ)log(C+1)+c
ǫ
for all x∈X is achievable, where C is the capacity of the channel p and c is a function of ǫ.
Y|X ǫ
Theexponentialfunctionalrepresentationcanstillbeappliedtothissetting.IfweencodeK (definedintheproofoftheSFRL)
into M using the optimal prefix-free code for the power-law distribution q(k)∝k−λ, where λ=1+1/(C+e−1loge+1),
then by the same argument in the proof of the SFRL, and Claim 3.1 in [4],
E[L(M)]≤C+log(C+1)+5
is achievable.
IV. LOSSY SOURCE CODING
We use the SFRL to establish one-shot achievability results for three lossy source coding settings.
A. Lossy source coding
Consider the following one-shot variable-length lossy source coding problem. We are given a random variable (source)
X ∈ X with X ∼ P , a reproduction alphabet Y, and a distortion function d : X ×Y → [0,∞] (note that X,Y can be
X
arbitrary, and d(x,y) can be infinite). Given X, the encoder selects Y˜ ∈ Y and encodes it using a prefix-free code into
M ∈ {0,1}∗. The decoder recovers Y˜ from M. Let R¯ = E[L(M)] be the expected value of the length of the description
M and E[d(X,Y˜)] be the average distortion of representing X by Y˜. An expected length-distortion pair (R¯,D) is said to be
achievable if there exists a variable-length code with expected description length R¯ such that E[d(X,Y˜)]≤D.
In the following we use the SFRL to establish a set of achievable (R¯,D) pairs.
Theorem 2. A pair (R¯,D) is achievable for the one-shot variable-length lossy source coding problem with source X ∼P ,
X
reproduction alphabet Y, and distortion measure d(x,y) if
R¯ >R(D)+log(R(D)+1)+6,
where
R(D)= inf I(X;Y)
PY|X:E[d(X,Y)]≤D
is the (asymptotic) rate-distortion function [5].
Proof: Let Y be the random variable that attains E[d(X,Y)] ≤D and I(X;Y)≤R(D)+ǫ. By the SFRL, there exists
Z independent of X such that Y =g (X,Z) and
X→Y
H(g (X,Z)|Z)≤I(X;Y)+η,
X→Y
where η =log(I(X;Y)+1)+4. Consider the set
A={(H(g (X,z)), E [d(X, g (X,z))]) : z ∈Z}.
X→Y X X→Y
5
Since(H(g (X,Z)|Z), E[d(X,Y)])isaweightedaverageofthepointsinA,itisintheconvexhullofA.ByCarathéodory’s
X→Y
theorem, there exists z ,z and λ∈[0,1] such that
0 1
(1−λ)H(g (X,z ))+λH(g (X,z ))≤H(g (X,Z)|Z)≤I(X;Y)+η,
X→Y 0 X→Y 1 X→Y
(1−λ)E [d(X, g (X,z ))]+λE [d(X, g (X,z ))]≤E[d(X,Y)].
X X→Y 0 X X→Y 1
Note that to satisfy the above inequalities, we need one point less than stated in Carathéodory’stheorem. Take Q∼Bern(λ),
Y˜ =g (X,z ). Then
X→Y Q
H(Y˜)≤H(Y˜|Q)+1≤I(X;Y)+η+1.
We use a Huffman code to encode Y˜ and obtain an expected length R¯ ≤H(Y˜)+1. The result follows by letting ǫ→0.
Althoughtheaboveachievabilityproofdoesnotuse randomcoding,itcanbeinterpretedasusingthefollowingsoftrandom
coding scheme (for discrete Y).
Softcodebookgeneration.TherandomvariableZ producedbytheSFRL intheproofofTheorem2 representsthechoice
of the codebook. We select a “soft codebook” by fixing Z|Y| = z|Y|. Unlike conventional codebook C ⊆ Y in which
each y can either be in C or not, a soft codebook z|Y| assigns to each y a weight w = p (y)/z , which indicates the
y Y y
likelihood that y is used.
Encoding. The encoder observes x and finds
p (y|x)
Y|X
y˜=argmax w · .
y
y pY(y)
It then finds k, the index of w in {w } sorted in descending order,and encodes k using an optimal prefix-freecode
y˜ y y∈Y
for the power-law distribution q(k)∝k−λ, where λ=1+1/(I(X;Y)+e−1loge+1). This is analogousto the case in
conventionalcodebookgenerationin whichwe find the closesty˜∈C to x andencodesit intoits indexin C.Here we use
a prefix-free code over the positive integers to encode the index into the description m because the number of possible
codewords y˜ (which is typically the entire Y) is large, but those with large w are more likely to be used so they are
y
assigned shorter descriptions.
Decoding. The decoder receives m, recovers k, then finds y˜ at the index k in {w } sorted in descending order.
y y∈Y
Thefiniteblocklengthvariable-lengthlossysourcecodingproblem[16]concernsthecaseinwhichthesourceismemoryless
andaveragepersymboldistortiond(xn,yn)=(1/n) d(x ,y ). In[25]itisshownthattheexpectedpersymboldescription
i i i
lengthR¯/n=R(D)+(1+o(1))(1/n)lognisachievableviad-semifaithfulcodes[26]withd(Xn,Y˜n)≤D surely.Applying
Theorem 2 to Xn, we have P
R¯/n=R(D)+(1/n)(log(nR(D)+1)+6)=R(D)+(1+o(1))(1/n)logn.
Hence we achieve the same redundancy as [25] albeit under the expected distortion constraint instead of the stronger sure
distortion constraint using the d-semifaithful codes.
We can use Theorem 2 to establish the achievability of Shannon’s (asymptotic) lossy source coding theorem [5], assuming
there exists a symbol y ∈ Y with finite d(x,y ) for all x. First note that the redundancy (1+o(1))(1/n)logn in the finite
0 0
blocklengthextensioncanbe madearbitrarilysmall,henceR¯/n canbemadearbitrarilyclose toR(D). Nowwe usethefinite
block length scheme over l blocks of n source symbols each of length n (for a total block length of nl). By the law of large
numbers,theprobabilitythatthetotaldescriptionlengthisgreaterthannl(R(D)+ǫ)tendsto0astheblocklengthapproaches
infinity. Hence, we can construct a fixed length code out of the variable-length code by simply discarding descriptions longer
than nl(R(D)+ǫ) and assigning the reconstruction sequence (y ,...,y ) to the discarded descriptions.
0 0
B. Multiple Description Coding
In this section, we use the SFRL to establish a one-shot inner bound for the variable-length multiple description coding
problem,whichyieldsanalternativeproofoftheElGamal-Coverinnerbound[6]andtheZhang-Bergerinnerbound[7],[27],
[28] in the asymptotic regime. The encoder observes X ∼ P and produces two prefix-free descriptions M ,M ∈ {0,1}∗.
X 1 2
Decoder 1 observes M and generates Y˜ with distortion d (X,Y˜ ). Similarly, Decoder 2 observes M and produces Y˜ with
1 1 1 1 2 2
distortion d (X,Y˜ ). Decoder 0 observes M and M and produces Y˜ with distortion d (X,Y˜ ). An expected description
2 2 1 2 0 0 0
length-distortiontuple (R¯ ,R¯ ,D ,D ,D ) is said to be achievable if there exists a scheme with expected description length
1 2 0 1 2
E[L(M )]≤R¯ and expected distortion E[d (X,Y˜)]≤D .
i i i i i
Theorem 3. The tuple (R¯ ,R¯ ,D ,D ,D ) is achievable if
1 2 0 1 2
R¯ ≥I(X;Y ,U)+2η,
1 1
R¯ ≥I(X;Y ,U)+2η,
2 2
R¯ +R¯ ≥I(X;Y ,Y ,Y |U)+2I(X;U)+I(Y ;Y |U)+5η,
1 2 0 1 2 1 2
D ≥E[d (X,Y )] for i=0,1,2
i i i
6
for some P , where
U,Y0,Y1,Y2|X
η =log I(X;Y ,Y ,Y ,U)+I(Y ;Y |U)+1 +7.
0 1 2 1 2
Note that the only difference between the(cid:0)above region and Zhang-Berger inner(cid:1)bound is the addition of η, which grows
like logn if we consider Xn and does not affect the asymptotic rate.
Proof: It suffices to prove the achievability of the corner point:
R¯ =I(X;Y |U)+I(X;U)+2η−1,
1 1
R¯ =I(X,Y ;Y |U)+I(X;Y |Y ,Y ,U)+I(X;U)+3η−1,
2 1 2 0 1 2
D =E[d (X,Y )] for i=0,1,2.
i i i
The desired rate region can be achievedby time sharing between this corner pointand the other cornerpointwhere Y ,Y are
1 2
flipped, resulting in a penalty of at most 1 bit (we can use the first bits of M and M to represent which corner point it is).
1 2
Applying the SFRL to X,U, we have U =g (X,Z ), where Z ⊥⊥X such that
X→U 3 3
H(U|Z )≤I(X;U)+log(I(X;U)+1)+4
3
≤I(X;U)+η−3.
Applying the SFRL to X,Y conditioned on U, we have Y =g (X,Z ,U), where Z ⊥⊥(X,U) such that
1 1 X→Y1|U 1 1
H(Y |U,Z )≤I(X;Y |U)+log(I(X;Y |U)+1)+4
1 1 1 1
≤I(X;Y |U)+η−3.
1
Applying the SFRL to (X,Y ),Y conditioned on U, we have Y =g (X,Y ,Z ,U), Z ⊥⊥(X,Y ,U) such that
1 2 2 XY1→Y2|U 1 2 2 1
H(Y |U,Z )≤I(X,Y ;Y |U)+log(I(X,Y ;Y |U)+1)+4
2 2 1 2 1 2
≤I(X,Y ;Y |U)+η−3.
1 2
Applying the SFRL to X,Y conditionedon (Y ,Y ,U), we have Y =g (X,Z ,Y ,Y ,U), Z ⊥⊥(X,Y ,Y ,U)
0 1 2 0 X→Y0|Y1Y2U 0 1 2 0 1 2
such that
H(Y |Y ,Y ,U,Z )≤I(X;Y |Y ,Y ,U)+log(I(X;Y |Y ,Y ,U)+1)+4
0 1 2 0 0 1 2 0 1 2
≤I(X;Y |Y ,Y ,U)+η−3.
0 1 2
Note that Z3 ⊥⊥X. Consider the convex hull of the 7-dimensional vectors
0
H(U|Z3 =z3)
0 0
H(Y |U,Z3 =z3)
1 0 0
H(Y2|U,Z03 =z03)
H(Y0|Y1,Y2,U,Z03 =z03)
E[d0(X,Y0)|Z03 =z03]
E[d1(X,Y1)|Z03 =z03]
E[d2(X,Y2)|Z03 =z03]
for different z3 ∈Z ×Z ×Z ×Z . By Carathéodory’s theorem, there exists a pmf p with cardinality |Q|≤7 and z˜3(q)
0 0 1 2 3 Q 0
such that
H(U|Q, Z3 =z˜3(Q))≤I(X;U)+η−3,
0 0
andsimilarlyfortheother6inequalities.TakeU˜ =g (X, z˜ (Q)),Y˜ =g (X, z˜ (Q),U˜),Y˜ =g (X,Y˜ ,
X→U 3 1 X→Y1|U 1 2 XY1→Y2|U 1
z˜ (Q),U˜) and Y˜ = g (X,z˜ (Q),Y˜ ,Y˜ ,U˜). Write C (y) ∈ {0,1}∗ for the Huffman codeword of y for the
2 0 X→Y0|Y1Y2U 0 1 2 pY
distribution p . We set M to be the concatenation of Q (3 bits), C (U˜) and C (Y˜ ), and M to be
Y 1 pU˜|Q(·|Q) pY˜1|U˜Q(·|U˜,Q) 1 2
the concatenation of Q, C (U˜), C (Y˜ ) and C (Y˜ ). The expected length of M is
pU˜|Q(·|Q) pY˜2|U˜Q(·|U˜,Q) 2 pY˜0|Y˜1Y˜2U˜Q(·|Y˜1,Y˜2,U˜,Q) 0 1
upper bounded by
3+(I(X;U)+η−3+1)+(I(X;Y |U)+η−3+1)
1
=I(X;Y |U)+I(X;U)+2η−1.
1
The bound on the expected length of M can be obtained similarly.
2
Decoder1 receivesM andrecoversQ, andthen recoversU˜ bydecodingthe Huffmancodeforthe distributionp (·|Q),
1 U˜|Q
and then recovers Y˜ similarly. Decoder 2 receives M and recovers Q,U˜ and Y˜ . Decoder 0 receives M ,M and recovers
1 2 2 1 2
Q,U˜,Y˜ ,Y˜ and Y˜ .
1 2 0
7
C. Lossy Gray–Wyner System
In this section, we use the SFRL to establish a one-shotinner boundfor the lossy Gray–Wynersystem [8], which yields an
alternativeproofoftheachievabilityoftherateregionintheasymptoticregime.Theencoderobserves(X ,X )∼P and
1 2 X1,X2
produces three prefix-free descriptions M ,M ,M ∈ {0,1}∗. Decoder 1 observes M ,M and generates Y˜ with distortion
0 1 2 0 1 1
d (X ,Y˜ ). Similarly, Decoder 2 observes M ,M and produces Y˜ with distortion d (X ,Y˜ ). An expected description
1 1 1 0 2 2 2 2 2
length-distortion tuple (R¯ ,R¯ ,R¯ ,D ,D ) is said to be achievable if there exists a scheme with expected description length
0 1 2 1 2
E[L(M )]≤R¯ and expected distortion E[d (X ,Y˜)]≤D .
i i i i i i
Theorem 4. The tuple (R¯ ,R¯ ,R¯ ,D ,D ) is achievable if
0 1 2 1 2
R¯ ≥I(X ,X ;U)+log(I(X ,X ;U)+1)+8,
0 1 2 1 2
R¯ ≥I(X ;Y |U)+log(I(X ;Y |U)+1)+5,
1 1 1 1 1
R¯ ≥I(X ;Y |U)+log(I(X ;Y |U)+1)+5,
2 2 2 2 2
D ≥E[d (X ,Y )] for i=1,2
i i i i
for some P , P , P .
U|X1,X2 Y1|X1,U Y2|X2,U
Note that the only difference between the above region and the lossy Gray–Wyner rate region [1, p. 357] is the addition of
the logarithm terms, which grows like logn if we consider Xn,Xn and does not affect the asymptotic rate.
1 2
Proof: Applying the SFRL to (X ,X ),U, we have U =g (X ,X ,Z ), where Z ⊥⊥(X ,X ) such that
1 2 X1X2→U 1 2 0 0 1 2
H(U|Z )≤I(X ,X ;U)+log(I(X ,X ;U)+1)+4.
0 1 2 1 2
Applying the SFRL to X ,Y conditioned on U, we have Y =g (X ,Z ,U), where Z ⊥⊥(X ,U) such that
1 1 1 X1→Y1|U 1 1 1 1
H(Y |U,Z )≤I(X ;Y |U)+log(I(X ;Y |U)+1)+4.
1 1 1 1 1 1
Applying the SFRL to X ,Y conditioned on U, we have Y =g (X ,Z ,U), where Z ⊥⊥(X ,U) such that
2 2 2 X2→Y2|U 2 2 2 2
H(Y |U,Z )≤I(X ;Y |U)+log(I(X ;Y |U)+1)+4.
2 2 2 2 2 2
Note that Z2 ⊥⊥(X ,X ). Consider the convex hull of the 5-dimensional vectors
0 1 2
H(U|Z2 =z2)
0 0
H(Y |U,Z2 =z2)
1 0 0
H(Y2|U,Z02 =z02)
E[d1(X1,Y1)|Z02 =z02]
E[d2(X2,Y2)|Z02 =z02]
for different z2 ∈Z ×Z ×Z . By Carathéodory’s theorem, there exists a pmf p with cardinality |Q|≤5 and z˜2(q) such
0 0 1 2 Q 0
that
H(U|Q, Z2 =z˜2(Q))≤I(X ,X ;U)+log(I(X ,X ;U)+1)+4,
0 0 1 2 1 2
and similarly for the other 4 inequalities. Take U˜ = g (X ,X , z˜ (Q)), Y˜ = g (X , z˜ (Q),U˜) and Y˜ =
X1X2→U 1 2 0 1 X1→Y1|U 1 1 2
g (X , z˜ (Q),U˜). Write C (y)∈{0,1}∗ for the Huffman codeword of y for the distribution p . We set M to be
X2→Y2|U 2 2 pY Y 0
the concatenation of Q (3 bits) and C (U˜), M = C (Y˜ ) and M = C (Y˜ ). The expected
pU˜|Q(·|Q) 1 pY˜1|U˜Q(·|U˜,Q) 1 2 pY˜2|U˜Q(·|U˜,Q) 2
length of M is upper bounded by
0
3+(H(U|Z )+1)
0
≤3+(I(X ,X ;U)+log(I(X ,X ;U)+1)+4+1)
1 2 1 2
=I(X ,X ;U)+log(I(X ,X ;U)+1)+8.
1 2 1 2
The bound on the expected length of M ,M can be obtained similarly.
1 2
Decoder 1 receives M ,M and recovers Q, and then recovers U˜ by decoding the Huffman code for the distribution
0 1
p (·|Q), and then recovers Y˜ by decoding the Huffman code for the distribution p (·|U˜,Q). Similar for Decoder 2.
U˜|Q 1 Y˜1|U˜Q
8
V. ACHIEVABILITY OF GELFAND–PINSKER
In this section, we use the SFRL to provethe achievabilitypart of the Gelfand-Pinskertheorem[9] for discrete memoryless
channels with discrete memoryless state p p , where the state is noncausally available at the encoder. The asymptotic
S Y|X,S
capacity of this setting is
C = max (I(U;Y)−I(U;S)).
GP
pU|S,x(u,s)
We show the achievability of any rate below C directly by using the SFRL to reduce the channel to a point-to-point
GP
memorylesschannel. Fix p and x(u,s) that attain the capacity. Applying the SFRL to S,U, there exists a random variable
U|S
V ⊥⊥S such that
H(U|V)≤I(U;S)+log(I(U;S)+1)+4.
Note that
I(V;Y)=I(U;Y)−I(U;Y|V)
≥I(U;Y)−H(U|V)
≥I(U;Y)−I(U;S)−log(I(U;S)+1)−4.
Hence we have constructed a memoryless point-to-pointchannel p with achievable rate close to I(U;Y)−I(U;S).
Y|V
For n channel uses, let Un|{Sn =sn}∼ p (u |s ). The SFRL applied to Sn,Un gives
i U|S i i
I(V;Yn)≥QnI(U;Y)−nI(U;S)−log(nI(U;S)+1)−4.
Now we use the channel p l times (for a total block length of nl). By the channel coding theorem, we can communicate
Yn|V
l(nI(U;Y)−nI(U;S)−log(nI(U;S)+1)−4)−o(l) bits with error probability that tends to 0 as l →∞. Letting n→∞
completes the proof.
In the above proof, we see that the SFRL can be used to convert a channel with state into a point-to-point channel by
“orthogonalizing”theauxiliaryinputU andthestateS.Thepoint-to-pointchannelcanbeconstructedexplicitlyviaexponential
functionalrepresentation.Thisconstructioncanbeusefulfordesigningcodesforchannelswith state basedoncodesforpoint-
to-pointchannels.ItisinterestingtonotethatthisreductionmakestheachievabilityprooffortheGelfand–Pinskerquitesimilar
to that for the causal case in which the channel is reduced to a point-to-point channel using the "Shannon strategy” (see [1,
p. 176]).
Note that Marton’s inner bound for the broadcast channels with private messages [29] can also be proved using the SFRL
in a similar manner.The idea is to “orthogonalize”the dependentauxiliary randomvariablesU ,U by applyingthe SFRL on
1 2
U ,U to produce two independentinput random variables, and treat them with Y ,Y as an interference channel, and finally
1 2 1 2
to treat interference as noise.
VI. GENERALDISTRIBUTIONS ANDPOISSON FUNCTIONAL REPRESENTATION
In this section, we prove Theorem 1 for general X and Y (over a Polish space with Borel probability measure). We first
extend the exponential functional representation to general distributions.
Definition 2 (Poisson functional representation). Fix any joint distribution P . Let 0 ≤ T ≤ T ≤ ··· be a Poisson point
XY 1 2
process with rate 1 (i.e., the incrementsT −T are i.i.d. Exp(1) for i=1,2,... with T =0), and Y˜ ,Y˜ ,... be i.i.d. with
i i−1 0 1 2
Y˜ ∼P . Take Z ={(T ,Y˜)} , i.e., a marked Poisson point process. Define
1 Y i i i=1,2,...
dP
Y
g (x, {(t ,y˜)})=y˜ argmin t · (y˜) ,
X→Y i i i dP (·|x) i
(cid:18) i Y|X (cid:19)
where we write y˜(k)=y˜ for readability.
k
NotethatifY isdiscrete,thePoissonfunctionalrepresentationreducestotheexponentialfunctionalrepresentationbyletting
Z =p (y)· min T .
y Y i
i:Y˜i=y
Now we proceed to prove Theorem 1 by showing that the Poisson functional representation satisfies the constraints.
Proof: Condition on the event {X =x}. First we show that g (x, {(T ,Y˜)}) follows the distribution P (·|x). By
X→Y i i Y|X
the markingtheoremof the Poisson pointprocess[30], {(T ,Y˜)} is a Poisson pointprocessoverthe productmeasure µ×P
i i Y
(where µ is the Lebesgue measure on [0,∞)). By the displacement theorem [30],
dP
T · Y (Y˜), Y˜
i i i
dP (·|x)
(cid:26)(cid:18) Y|X (cid:19)(cid:27)
9
is a Poisson point process over µ×P (·|x). Hence
Y|X
dP
minT · Y (Y˜) ∼ Exp(1),
i i
i dPY|X(·|x)
and
dP
Y˜ argminT · Y (Y˜) ∼ P (·|x),
i dP (·|x) i Y|X
(cid:18) i Y|X (cid:19)
where we write Y˜(k)=Y˜ . Now we bound H(Y |{(T ,Y˜)}). Let
k i i
dP
Θ=minT · Y (Y˜),
i i
i dPY|X(·|x)
dP
K =argminT · Y (Y˜),
i i
dP (·|x)
i Y|X
then H(Y |{(T ,Y˜)})≤H(K). Conditioned on Θ=θ, Y˜ ∼P (·|x) and {(T ,Y˜)} is a Poisson point process over
i i K Y|X i i i6=K
the semidirect product measure
dP (·|x)
ν(A×B)= µ A ∩ θ· Y|X (y), ∞ dP (y).
ˆ dP Y
B (cid:18) (cid:20) Y (cid:19)(cid:19)
Note that K −1 = |{i: T <T }|. Hence K −1 conditioned on Θ = θ and Y˜ = y˜ follows the Poisson distribution with
i K K
rate
dP (·|x)
Y|X
ν([0, T )×Y)=ν 0, θ· (y˜) ×Y
K
dP
(cid:18)(cid:20) Y (cid:19) (cid:19)
dP (·|x) dP (·|x)
= µ 0, θ· Y|X (y˜) ∩ θ· Y|X (y), ∞ dP (y)
ˆ dP dP Y
Y (cid:18)(cid:20) Y (cid:19) (cid:20) Y (cid:19)(cid:19)
dP (·|x) dP (·|x)
=θ max 0, Y|X (y˜)− Y|X (y) dP (y)
ˆ dP dP Y
Y (cid:26) Y Y (cid:27)
dP (·|x)
≤θ Y|X (y˜)·dP (y)
ˆ dP Y
Y Y
dP (·|x)
Y|X
=θ (y˜).
dP
Y
Therefore
∞
E[logK]=E e−θE logK|Θ=θ, Y˜ =Y dθ
Y∼PY|X(·|x) ˆ K
(cid:20) 0∞ h dP (·|x) i (cid:21)
≤E e−θlog θ Y|X (Y)+1 dθ
Y∼PY|X(·|x) ˆ dP
(cid:20) 0 (cid:18) Y (cid:19) (cid:21)
∞ dP (·|x)
≤E log e−θθ Y|X (Y)dθ+1
Y∼PY|X(·|x) ˆ dP
(cid:20) (cid:18) 0 Y (cid:19)(cid:21)
dP (·|x)
=E log Y|X (Y)+1
Y∼PY|X(·|x) dP
(cid:20) (cid:18) Y (cid:19)(cid:21)
dP (·|x)
≤E max log Y|X (Y),0 +1
Y∼PY|X(·|x) dP
(cid:20) (cid:26) Y (cid:27) (cid:21)
dP (·|x)
=D(P (·|x)kP )−E min log Y|X (Y),0 +1
Y|X Y Y∼PY|X(·|x) dP
(cid:20) (cid:26) Y (cid:27)(cid:21)
≤D(P (·|x)kP )+e−1loge+1,
Y|X Y
where the last line follows by the same arguments as in Appendix A in [4]. For X ∼P ,
X
E[logK]≤I(X;Y)+e−1loge+1.
By the maximum entropy distribution subject to a given E[logK] (see the Appendix), we have
H(K)≤I(X;Y)+e−1loge+2+log I(X;Y)+e−1loge+2
≤I(X;Y)+log(I(X;Y)+1)+e−1loge+2+log e−1loge+2
(cid:0) (cid:1)
<I(X;Y)+log(I(X;Y)+1)+4.
(cid:0) (cid:1)
10
VII. TIGHTNESSOF THESFRL
The SFRL states that I(X;Z|Y)≤log(I(X;Y)+1)+4 is achievable.A natural question to ask is whether this inequality
is tight for some X,Y. In this section, we show that the log term is necessary and that the SFRL is in general tight to within
5 bits. Define the excess functional information as
Ψ(X →Y)= inf I(X;Z|Y).
Z:Z⊥⊥X,H(Y|X,Z)=0
Then by the SFRL, Ψ(X →Y)≤log(I(X;Y)+1)+4. We now establish the following lower bound on Ψ(X →Y).
Proposition 1. For discrete Y,
1
Ψ(X →Y)≥− P p (y|X)≥t log P p (y|X)≥t dt−I(X;Y).
ˆ X Y|X X Y|X
y∈Y 0
X (cid:8) (cid:9) (cid:0) (cid:8) (cid:9)(cid:1)
Moreover for |Y| =2, equality holds in the above inequality, and the infimum in Ψ(X →Y) is attained via the exponential
functional representation.
Proof: Fix Z ⊥⊥ X such that Y = g(X,Z). For any y, let V = p (y|Z), U ∼ Unif[0,1], X˜ = p (y|X),
y Y|Z y Y|X
V˜ =P X˜ ≥U|U , then E[V ]=E[V˜ ]=p (y). We have
y y y y Y
n o
1
P{V ≥t}dt=E[max{V −v, 0}]
ˆ y y
v
=E max p (y|Z)−v, 0
Z Y|Z
=E [max{P {g(X,Z)=y|Z}−v, 0}]
Z(cid:2) (cid:8) X (cid:9)(cid:3)
=E max E P g(X,Z)=y Z,X˜ Z −v, 0
Z X˜y X y
=EZhmaxnEX˜yhPXng(X,Z)=y(cid:12)(cid:12)Z,X˜yo (cid:12)(cid:12)(cid:12)Zi−EX˜yoi1 X˜y >FX−˜y1(1−v) , 0
≤EZhEX˜y nmax hPXng(X,Z)=y(cid:12)(cid:12)Z,X˜yo−(cid:12)(cid:12)(cid:12) 1i X˜y >hFX−˜ny1(1−v) , 0 Zoi oi
=EZhEX˜yhPX ng(X,nZ)=y Z,X˜y(cid:12)(cid:12) ·1 oX˜y ≤nFX−˜y1(1−v) Z o o (cid:12)(cid:12)(cid:12) ii
=EX˜yh EZhPXng(X,Z)=y(cid:12)(cid:12)Z,X˜yo X˜ny ·1 X˜y ≤FX−˜y1(1o−(cid:12)(cid:12)(cid:12) v)ii
=EX˜yhEXhPZ{ng(X,Z)=y|(cid:12)(cid:12)X} X˜oy(cid:12)(cid:12)(cid:12) ·1i X˜yn≤FX−˜y1(1−v) oi
=E hE hp (y|X) X˜ ·1 X˜(cid:12) ≤iF−1n(1−v) oi
X˜y X Y|X y (cid:12)y X˜y
=E hX˜ ·h1 X˜ ≤F−(cid:12)1(1i−v)n oi
X˜y y y X˜(cid:12)y
=E hmax PnX˜ ≥U U −vo, i0
U y
=E mhax Vn˜ −nv, 0 (cid:12) o oi
y (cid:12)
h1 n oi
= P{V˜ ≥t}dt.
ˆ y
v
Hence V dominates V˜ stochastically in the second order. By the concavity of −tlogt, we have
y y
H(Y|Z)=− E p (y|Z)logp (y|Z)
Z Y|Z Y|Z
y
X (cid:2) (cid:3)
=− E[V logV ]
y y
y
X
(1)
≥− E V˜ logV˜
y y
Xy h i
1
=− P p (y|X)≥u log P p (y|X)≥u du.
ˆ X Y|X X Y|X
y 0
X (cid:8) (cid:9) (cid:0) (cid:8) (cid:9)(cid:1)
Therefore,
1
I(X;Z|Y)≥− P p (y|X)≥t log P p (y|X)≥t dt−I(X;Y).
ˆ X Y|X X Y|X
y 0
X (cid:8) (cid:9) (cid:0) (cid:8) (cid:9)(cid:1)
