On the Renyi Divergence, Joint Range of Relative Entropies, and a Channel Coding Theorem

0.28 MB·English

by  Igal Sason

On the Re´nyi Divergence, Joint Range of Relative
Entropies, and a Channel Coding Theorem
Igal Sason
5
1
0
2

t
c
O
Abstract

6
2
This paper starts by considering the minimization of the Re´nyi divergence subject to a constraint on

]
T the total variation distance. Based on the solution of this optimization problem, the exact locus of the
I
. points D(Q P1),D(Q P2) is determinedwhen P1,P2,Q are arbitraryprobabilitymeasures which are
s k k
c mutual(cid:0)ly absolutely continuo(cid:1)us, and the total variation distance between P1 and P2 is not below a given
[

value. It is further shown that all the points of this convex region are attained by probability measures
4
v which are defined on a binary alphabet. This characterization yields a geometric interpretation of the
6
1 minimal Chernoff information subject to a constraint on the variational distance.
6
3 This paper also derives an exponentialupper bound on the performance of binary linear block codes
0
(or code ensembles) under maximum-likelihood decoding. Its derivation relies on the Gallager bounding
.
1
0 technique,anditreproducestheShulman-Federboundasaspecialcase.Theboundisexpressedintermsof
5
theRe´nyidivergencefromthenormalizeddistancespectrumofthecode(ortheaveragedistancespectrum
1
:
v of the ensemble) to the binomially distributed distance spectrum of the capacity-achieving ensemble
i
X of random block codes. This exponential bound provides a quantitative measure of the degradation in
r
a performance of binary linear block codes (or code ensembles) as a function of the deviation of their
distance spectra from the binomial distribution. An efficient use of this bound is considered.
Keywords: Chernoff information, distance spectrum, error exponent, maximum-likelihood
decoding, relative entropy, Re´nyi divergence, total variation distance.
I. Sason is with the Department of Electrical Engineering, Technion–Israel Institute of Technology, Haifa 32000,
Israel(e-mail:[email protected]).ThisworkhasbeensupportedbytheIsraeliScienceFoundation,grant12/12.
The paper has been submitted to the IEEE Transactions on Information Theory in February 17, 2015.
This work has been presented at the 2015 IEEE International Symposium on Information Theory, Hong Kong,
June 14–19, 2015.
2
I. INTRODUCTION
The Re´nyi divergence, introduced in [30], has been studied so far in various information-
theoretic contexts (and it has been actually used before it had a name [37]). These include
generalized cutoff rates and error exponents for hypothesis testing ([1], [6], [38]), guessing
moments ([2], [9]), source and channel coding error exponents ([2], [12], [22], [27], [37]),
strong converse theorems for classes of networks [11], strong data processing theorems for
discrete memoryless channels [28], bounds for joint source-channel coding [41], and one-shot
bounds for information-theoretic problems [46].
In [14], Gilardoni derived a Pinsker-type lower bound on the Re´nyi divergence D (P Q)
α
k
for α (0,1). In view of the fact that this lower bound is not tight, especially when the total
∈
variation distance P Q is large, this paperstarts by consideringthe minimization ofthe Re´nyi
| − |
divergence D (P Q), for an arbitrary α > 0, subject to a given (or minimal) value of the total
α
k
variation distance. Note that the minimization here is taken over all probability measures with
a total variation distance which is not below a given value; this problem differs from the type
of problems studied in [3] and [24], in connection to the minimization of the relative entropy
D(P Q) subject to a minimal value of the total variation distance with a fixed probability
k
measure Q. The solution of this problem generalizes the problem of minimizing the relative
entropy D(P Q) subject to a given value of the total variation distance where the latter is a
k
special case with α = 1 (see [10], [13], [29]).
One possible way to deal with this problem stems from the fact that the Re´nyi divergence is a
one-to-one transformation of the Hellinger divergence H (P Q) where for α (0,1) (1, ):
α
k ∈ ∪ ∞
1
D (P Q) = log(1+(α 1)H (P Q)) (1)
α α
k α 1 − k
−
and H (P Q) is an f-divergence; since the total variation distance is also an f-divergence,
α
k
this problem can be viewed as a minimization of an f-divergence subject to a constraint on
another f-divergence. The numerical optimization of an f-divergence subject to simultaneous
constraints on f -divergences (i = 1,...,L) was recently studied in [15], where it has been
i
shown that it suffices to restrict attention to alphabets of cardinality L+2. In fact, as shown
in [44, (22)], a binary alphabet suffices if there is a single constraint (i.e., L = 1) which is on
the total variation distance. In view of (1), the same conclusion also holds when minimizing
the Re´nyi divergence subject to a constraint on the total variation distance. To set notation, the
divergences D(P Q), P Q , H (P Q), D (P Q) are defined at the end of this section,
α α
k | − | k k
October27,2015 DRAFT
3
being consistent with the notation in [35] and [45].
This paper treats this minimization problem of the Re´nyi divergence in a different way. We
first generalize the analysis in [10], which was used for the minimization of the relative entropy
subject to a constraint on the variational distance, for proving that it suffices to restrict attention
to probability measures which are defined on a binary alphabet. Furthermore, the continuation
of the analysis in this paper relies on the Lagrange duality, and a solution of the Karush-Kuhn-
Tucker (KKT) equations while asserting strong duality for the studied problem. The use of
Lagrange duality further simplifies the computational task of the studied minimization problem.
As complementary results to the minimization problem studied in this paper, the reader is
referred to [35, Section 8] which provides upper bounds on the Re´nyi divergence D (P Q) for
α
k
an arbitrary α (0, ) as a function of either the total variation distance or relative entropy in
∈ ∞
case that the relative information is bounded.
The solution of the minimization problem of the Re´nyi divergence, subject to a constraint
on the total variation distance, provides an elegant way for the characterization of the exact
locus of the points D(Q P ),D(Q P ) where P and P are probability measures whose
1 2 1 2
k k
total variation distan(cid:0)ce is not below a give(cid:1)n value ε, and Q is an arbitrary probability measure.
It is further shown in this paper that all the points of this convex region can be attained by a
triple of probability measures (P ,P ,Q) which are defined on a binary alphabet.
1 2
In view of the characterization of the exact locus of these points, a geometric interpretation
is provided in this paper for the minimal Chernoff information between P and P , denoted
1 2
by C(P ,P ), subject to an ε-separation constraint on the variational distance between P
1 2 1
and P . It is demonstrated in the following that the intersection point at the boundary of the
2
locus of D(Q P ),D(Q P ) and the straight line D(Q P ) = D(Q P ) is the point whose
1 2 1 2
k k k k
coordinat(cid:0)es are equal to the mi(cid:1)nimal value of C(P ,P ) under the constraint P P ε. The
1 2 1 2
| − | ≥
reader is referred to [48], which relies on the closed-form expression in [31, Proposition 2] for
the minimization of the constrained Chernoff information, and which analyzes the problem of
channel-codedetectionbyathird-partyreceiverviathelikelihoodratiotest.Inthelatterproblem,
a third-party receiverhas to detect the channelcode used by the transmitter by observing a large
number of noise-affected codewords; this setup has applications in security or cognitive radios,
or in link adaptation in some wireless technologies.
Sincethe Re´nyidivergenceD (P Q) forms ageneralizationofthe relativeentropyD(P Q),
α
k k
where the latter corresponds to α = 1, the approach suggested in this paper for the characteri-
October27,2015 DRAFT
4
zation of the exact locus of pairs of relative entropies in view of a solution to a minimization
problem of the Re´nyi divergence is analogous to the usefulness of complex analysis in solving
real-valued problems. We consider the analysis of the considered problem as mathematically
pleasing in its own right. Note, however, that an operational meaning of a special point at the
boundary of this locus has an operational meaning in view of [48] (see the previous paragraph).
The studied problem considered here differs from the study in [17] which considered the joint
range of f-divergences for pairs (rather than triplets) of probability measures.
The performance analysis of linear codes under maximum-likelihood (ML) decoding is of
interest for studying the potential performance of these codes under optimal decoding, and for
the evaluation of the degradation in performance that is incurred by the use of sub-optimal and
practical decoding algorithms. The reader is referred to [32] which is focused on this topic.
The second part of this paper derives an exponential upper bound on the performance of ML
decoded binary linear block codes (or code ensembles). Its derivation relies on the Gallager
bounding technique(see [32, Chapter 4], [36]), and it reproducesthe Shulman-Federbound [40]
as a special case. The new exponential bound derived in this paper is expressed in terms of
the Re´nyi divergence from the normalized distance spectrum of the code (or average distance
spectrum of the ensemble) to the binomial distribution which characterizes the average distance
spectrum of the capacity-achieving ensemble of fully random block codes. This exponential
bound provides a quantitative measure of the degradation in performance of binary linear block
codes (or code ensembles)as a function of the deviation of their (average)distance spectra from
the binomial distribution, and its use is exemplified for an ensemble of turbo-block codes.
This paper is structured as follows: Section II solves the minimization problem for the Re´nyi
divergence under a constraint on the total variation distance, Section III uses the solution of
this minimization problem to obtain an exact characterization of the joint range of the relative
entropies in the considered setting above. Section IV provides a new exponential upper bound
on the block error probability of ML decoded binary linear block codes, which is expressed in
terms of the Re´nyi divergence, suggests an efficient way to apply the bound to the performance
evaluationof binary linear block codes(or code ensembles),and exemplifiesits use. Throughout
this paper, logarithms are to the base e.
We end this section by introducing the definitions and notation used in this work, which are
consistent with [35], [45], and are included here for the convenience of the reader.
October27,2015 DRAFT
5
Definitions and Notation
We assume throughout that the probability measures P and Q are defined on a common
measurable space ( ,F), and P Q denotes that P is absolutely continuous with respect
A ≪
to Q, namely there is no event F such that P( ) > 0 = Q( ). Let dP denote the
F ∈ F F dQ
Radon-Nikodym derivative (or density) of P with respect to Q.
Definition 1 (Relative entropy): The relative entropy is given by
dP
D(P Q) = dP log . (2)
k dQ
ZA (cid:18) (cid:19)
Definition 2 (Total variation distance): The total variation distance is given by
dP
P Q = 1 dQ. (3)
| − | dQ −
ZA(cid:12) (cid:12)
Definition 3 (Hellinger divergence): The Hell(cid:12)inger div(cid:12)ergence of order α (0,1) (1, ) is
(cid:12) (cid:12)
∈ ∪ ∞
(cid:12) (cid:12)
given by
1 dP α
H (P Q) = dQ 1 . (4)
α
k α 1 dQ −
− (cid:18)ZA (cid:18) (cid:19) (cid:19)
The analytic extension of H (P Q) at α = 1 yields H (P Q) = D(P Q) (nats).
α 1
k k k
Definition 4 (Re´nyi divergence): The Re´nyi divergence of order α 0 is given as follows:
≥
• If α (0,1) (1, ), then
∈ ∪ ∞
1 dP α
D (P Q) = log dQ . (5)
α
k α 1 dQ
− (cid:18)ZA (cid:18) (cid:19) (cid:19)
• If α = 0, then
1
D (P Q) = max log . (6)
0
k F∈F:P(F)=1 Q( )
(cid:18) F (cid:19)
• D1(P Q) = D(P Q) which is the analytic extension of Dα(P Q) at α = 1.
k k k
• If α = + then
∞
dP
D (P Q) = log esssup (Y) (7)
∞
k dQ
(cid:18) (cid:19)
with Y Q.
∼
Definition 5 (Chernoff information): The Chernoff information between probability measures
P and P is expressed as follows in terms of the Re´nyi divergence:
1 2
C(P ,P ) = max (1 α)D (P P ) (8)
1 2 α 1 2
α∈[0,1] − k
n o
and it is the best achievable exponent in the Bayesian probability of error for binary hypothesis
testing (see, e.g., [5, Theorem 11.9.1]).
October27,2015 DRAFT
6
II. MINIMIZATION OF THE RE´NYI DIVERGENCE WITH A CONSTRAINED TOTAL VARIATION
DISTANCE
In this section, we derive a tight lower bound on the Re´nyi divergence D (P P ) subject
α 1 2
k
to an equality constraint on the total variation distance P P = ε where ε [0,2) is
1 2
| − | ∈
fixed; alternatively, it can regarded as a minimization problem under the inequality constraint
P P ε. It is first shown that this lower bound is attained for probability measures defined
1 2
| − | ≥
on a binary alphabet, and Lagrange duality is used to further simplify the computational task
of this bound. The special case where α = 1, which is specialized to the minimization of the
relative entropy subjectto a fixedtotal variation distance,has beenstudied extensively,andthree
equivalent forms of the solution to this optimization problem were derived in [10], [13], [29].
In [14, Corollaries 6 and 9], Gilardoni derived two Pinsker-type lower bounds on the Re´nyi
divergence of order α (0,1), expressed in terms of the total variation distance. Among these
∈
two bounds, the improved lower bound is given (in nats) by
D (P Q) 1αε2 + 1α(1+5α 5α2)ε4, α (0,1) (9)
α k ≥ 2 9 − ∀ ∈
where P Q = ε denotes the total variation distance between P and Q. Note that in the limit
| − |
where ε tend to 2 (from below), this lower bound converges to a finite value which is at most
22; it is, however, an artifact of the lower bound in view of the next lemma.
9
Lemma 1:
lim inf D (P Q) = , α> 0. (10)
α
ε↑2 P,Q:|P−Q|=ε k ∞ ∀
Proof: See Appendix I-A.
Inthefollowing,wederiveatightlowerboundwhichisshowntobeachievablebyarestriction
of the probability measures to a binary alphabet. For α> 0, let
g (ε) , min D (P P ), (11)
α α 1 2
P1,P2:|P1−P2|=ε k
= min D (P P ), ε [0,2). (12)
α 1 2
P1,P2:|P1−P2|≥ε k ∀ ∈
In the following, we evaluate the function g . In view of [10, Section 2] which characterizes
α
the minimum of the relative entropy in terms of the total variation distance, we first extend the
argument in [10] to prove the next lemma.
Lemma 2: Foranarbitraryα> 0,theminimizationin(11)isattainedbyprobabilitymeasures
which are defined on a binary alphabet.
October27,2015 DRAFT
7
Proof: See Appendix I-B.
The following proposition enables to calculate g for an arbitrary positive α.
α
Proposition 1: Let α (0,1) (1, ) and ε [0,2). The function g in (11) satisfies
α
∈ ∪ ∞ ∈
g (ε) = min d (p q) (13)
α α
p,q∈[0,1]:|p−q|≥ε k
2
where
log pαq1−α+(1 p)α(1 q)1−α
d (p q) , − − (14)
α k (cid:16) α 1 (cid:17)
−
denotes the binary Re´nyi divergence.
Proof: This directly follows from Lemma 2.
Proposition 2:
g (ε) = log 1 1 ε2 , ε [0,2) (15)
21 − − 4 ∀ ∈
(cid:0) (cid:1)
and
log(1+ε2), if ε [0,1],
g (ε) = ∈ (16)
2

log 1 1ε , if ε (1,2).
 − − 2 ∈
Furthermore, for α (0,1) and ε [0,2),(cid:0) (cid:1)
∈ ∈
α
g (ε) = g (ε), (17)
α 1−α
1 α
(cid:18) − (cid:19)
and
1
g (ε) c (α) log +c (α), (18)
α ≥ 1 1 1ε! 2
− 2
where
α log 2
c (α) , min 1, , c (α) , . (19)
1 2
1 α −1 α
(cid:26) − (cid:27) −
Proof: See Appendix II.
Remark 1: The lower bound on g () in (18) providesanother proof of Lemma 1 since it first
α
·
yields that lim g (ε) = for α (0,1); this lemma also holds for α 1 since D (P Q)
ε↑2 α α
∞ ∈ ≥ k
is monotonically increasing in its order α.
In the following, we use Lagrange duality to obtain an alternative expression as a solution of
theminimizationproblemforg .RecallthatProposition1appliestoeveryα > 0.Thefollowing
α
enables to simplify considerably the computational task in calculating g , for α (0,1).
α
∈
October27,2015 DRAFT
8
Lemma 3: Let α (0,1) and ε′ (0,1). The function
∈ ∈
α−1 α−1
1 ε′ 1+ ε′
fα,ε′(q) , (cid:16) −1+1−εq′(cid:17)α −1(cid:16) ε′q(cid:17)α , ∀q ∈ (0,1−ε′) (20)
q − − 1−q
(cid:16) (cid:17) (cid:16) (cid:17)
is strictly monotonically increasing, positive, continuous, and
lim fα,ε′(q) = 0, lim fα,ε′(q) =+ . (21)
q→0+ q→(1−ε′)− ∞
Proof: See Appendix III.
Corollary 1: For α (0,1) and ε′ (0,1), the equation
∈ ∈
fα,ε′(q) = 1−αα (22)
has a unique solution q (0,1 ε′).
∈ −
Proof: It follows from Lemma 3, and the mean value theorem for continuous functions.
Remark 2: Sincefα,ε′: (0,1 ε′) (0, )isstrictlymonotonicallyincreasing(seeLemma3),
− → ∞
the numerical calculation of the unique solution of equation (22) is easy.
An alternative simplified form for the optimization problem in Proposition 1 is next provided
for orders α (0,1). Hence, Proposition 1 applies to every α > 0, whereas the following is
∈
restricted to α (0,1). This, however, proves to be very useful in the next section in terms of
∈
obtaining a significant reduction in the computationalcomplexity of g () where only α (0,1)
α
· ∈
is of interest there.1
Proposition 3: Let α (0,1), ε (0,2), and let ε′ = ε. A solution of the minimization
∈ ∈ 2
problem for g (ε) in Proposition 1 is obtained by calculating the binary Re´nyi divergence
α
d (p q) in (14) while taking the unique solution q (0,1 ε′) of (22), and setting p = q+ε′.
α
k ∈ −
Proof: See Appendix IV.
In view of Proposition 3, the plots in Figures 1 and 2 provide numerical results.
1This saving in the computational complexity accelerated the running time of the numerical calculations in our
computer by two orders of magnitude.
October27,2015 DRAFT
9
7
6
5
)2
P
|| 1 4
P
(α
D
n 3
mi
2
1
0
0 0.5 1 1.5 2
ε
Fig.1. A plot of theminimum of the Re´nyi divergence Dα(P1kP2) subject tothe constraint |P1−P2|≥εwhere
ε∈[0,2). Thecurvesinthisplot correspond toα=0.25 (thicksolidcurve),α=0.50 (thinsolidcurve), α=0.75
(thick dashed curve), and α=1.00 (thin dashed curve, referring to the relative entropy).
III. THE LOCUS OF (D(Q P1),D(Q P2)) WITH A CONSTRAINED TOTAL VARIATION
k k
DISTANCE
In this section, we address the following question:
Question 1: What is the locus of the points D(Q P ),D(Q P ) if P ,P ,Q are arbitrary
1 2 1 2
k k
probability measures which are mutually absolu(cid:0)tely continuous, and(cid:1)P P ε for a given
1 2
| − | ≥
value ε (0,2) ? (none of the three probability measures is fixed).
∈
The present section provides an exact characterization of this locus in view of the solution to
the minimization problem in Section II, and the following lemma:
Lemma 4: Let P ,P ,Q be pairwise mutually absolutely continuous probability measures
1 2
defined on a measurable space ( ,F). Then, for α (0,1) (1, ),
A ∈ ∪ ∞
D (P P ) =D(Q P )+ α D(Q P )+ 1 D(Q Q ) (23)
α 1k 2 k 2 1−α · k 1 α−1 · k α
October27,2015 DRAFT
10
7
6
5
)2
P
|| 1 4
P
(α
D
n 3
mi
2
1
0
0 0.4 0.8 1.2 1.6 2
|P − P |
1 2
Fig. 2. A plot of the minimum of the Re´nyi divergence Dα(P1kP2) of order α = 0.90 subject to the constraint
|P1−P2| ≥ ε ∈ [0,2). The exact minimum (thick solid curve) is compared with the Pinsker-type lower bound in
[14, Corollary 9] (the thin solid curve), and its weaker version in [14, Corollary 6] (the dashed curve).
where the probability measure Q is given by
α
α 1−α
dP1 (x) dP2 (x)
dQα dQ dQ
(x) = , x . (24)
dQ d(cid:16)P α(cid:17) (cid:16)dP (cid:17)1−α ∀ ∈ A
1 2
(u) (u) dQ(u)
dQ dQ
Z A(cid:18) (cid:19) (cid:18) (cid:19)
Proof: See Appendix V.
As a corollary of Lemma 4, the following tight inequality holds, which is attributed to van
Erven [7, Lemma 6.6] and Shayevitz [39, Section IV.B.8]). It will be useful for the continuation
of this section, jointly with the results of Section II.
Corollary 2: Let P P be mutually absolutely continuousdiscreteprobability measures
1 2
≪≫
defined on a common set . If α (0,1) then
A ∈
α D(Q P )+D(Q P ) D (P P ) (25)
1−α · k 1 k 2 ≥ α 1k 2
October27,2015 DRAFT