A Variational Characterization of R\'enyi Divergences

0.21 MB·

by  Venkat Anantharam

´
A VARIATIONAL CHARACTERIZATION OF RENYI
DIVERGENCES
VENKATANANTHARAM1
7
1
0
2

n
a
J

6
2 ABSTRACT:Atar,ChowdharyandDupuishaverecentlyexhibitedavariationalformulaforexponen-

tial integrals of bounded measurable functions in terms of Re´nyi divergences. We develop a variational
]
T
characterization of the Re´nyi divergences between two probability distributions on a measurable space
I
. in terms of relative entropies. When combined with the elementary variational formula for exponential
s
c integralsofboundedmeasurablefunctionsintermsofrelativeentropy,thisyieldsthevariationalformula
[
ofAtar, Chowdharyand Dupuisas acorollary.

1
WealsodevelopananalogousvariationalcharacterizationoftheRe´nyidivergenceratesbetweentwo
v
6 stationary finitestateMarkov chains in termsof relativeentropy rates. When combinedwith Varadhan’s
9
variationalcharacterization ofthespectral radius ofsquarematrices withnonnegativeentries interms of
7
7 relative entropy, this yields an analog of the variational formula of Atar, Chowdary and Dupuis in the
0 framework offinitestateMarkovchains.
.
1
0
Key words: Markovchains;Relativeentropy;Re´nyidivergence;Variationalformulas.
7
1
:
v
i
X
r
a
1EECS Department, University of California, Berkeley, CA 94720, USA. Research supported in part by the National
ScienceFoundationgrantsECCS-1343398,CNS-1527846,CCF-1618145,theNSFScience&TechnologyCentergrantCCF-
0939370 (Science of Information), and the William and Flora Hewlett Foundation Center for Long Term Cybersecurity at
Berkeley.
1
1 Introduction
Evaluatinghowfarawayagivenprobabilitydistributionisfromanothercan bedoneinmanyways. The
Kullback-Leibler divergence or relative entropy, which is closely tied to Shannon’s notion of entropy,
is one such measure prominent in statistical applications. It belongs to a larger family of divergences,
the so-called Re´nyi divergences, which are closely tied to Re´nyi’s notion of entropy. Re´nyi divergences
also havenumerousapplicationsinproblemsofinterestin statisticsand informationtheory,see[4]fora
surveyofsomeoftheirbasicpropertiesandsomeindicationoftheirapplications. TheRe´nyidivergences,
with a minor change in scaling relative to the definition in [4], are the topic of this article. We treat the
Re´nyi divergencesas parametrized byareal numberα R, α 0,α 1.
∈ ≠ ≠
Wewere promptedtowritethisdocumentby reading arecent paperofAtar, Chowdhary and Dupuis
[1], which provides a variational formula for exponential integrals of bounded measurable functions in
termsofRe´nyidivergences. Weshowthatthevariationalcharacterization in[1]isasimpleconsequence
of a variational characterization for Re´nyi divergences in terms of relative entropies, which we also
develop. For the case of probability distributions on a finite set, and in the range α 0, α 1, our
> ≠
variational characterization for Re´nyi divergences was developed by Shayevitz, [10] and [11, Thm. 1].
More recently, for mutually absolutely continuous probability distributions on a measurable space, in
the case α 0, α 1, parts of this variational characterization appear in a paper of Sason, see [8,
> ≠
Lem. 4 and Cor. 2]. The ability to derive the variational formula of [1] from inequalities for the Re´nyi
divergencesin termsofrelativeentropies,in thecaseα 1, is alsoremarked on ina recent paperofLiu,
>
Courtade, Cuff, and Verdu´ [6, Sec. II-A]. To thebest of ourknowledge,however,a full treatment of this
variationalcharacterizationofRe´nyidivergencesintermsofrelativeentropies,coveringanarbitrarypair
of probability distributionson a measurable space and all possiblevalues for α, does not appear to be in
the literature and so it seems worth writing down. It is also worth noting how easily the full variational
formulaof[1], inall cases, fallsoutofthisvariationalcharacterization ofRe´nyidivergences.
Section 2 presents the notational conventions and the definitions of the main quantities used in this
document in the i.i.d. case. The main result in the i.i.d. case, Theorem 1, is stated in Section 3. The
result of [1] that prompted this paper is presented in Section 4, and is derived there as a consequence of
Theorem 1 and the elementary variational formula for exponential integrals in (2). Theorem 1 itself is
provedinSection 5.
We then turn to a development of analogs of the preceding results in the case of stationary finite
state Markov chains. Section 6 makes the necessary definitions and gathers some standard facts about
the asymptoticproperties of iterated powers ofa square matrix with nonnegativeentries, which weneed
for our discussion. It also contains the analog of the elementary variational formula in the context of
finite state Markov chains, in (17), which is Varadhan’s variational characterization in terms of relative
entropy of the spectral radius of square matrices with nonnegative entries. The main results in the case
of stationary finite state Markov chains are stated in Section 7. These are Theorem 2, which gives a
variational characterization of each Re´nyi divergence rate between two stationary finite state Markov
chainsintermsofrelativeentropyrates,andTheorem3,whichgivesananalogofthevariationalformula
of [1] in thecontextof finitestate Markovchains. A proofof Theorem 3 assumingthe truthof Theorem
2, and using (17), is also provided in this section. The proof of Theorem 2 is provided in Section 8. We
end thepaperinSection 9 withsomethoughtsabout directionsforfuturework.
In order to maintain the flow of the main exposition, the details of several proofs are relegated to
appendices.
2
2 Setup
Let S, be a measurable space. S denotes the set of bounded measurable real-valued functions
( F) B( )
and S the set of probability measures on S, . For ν,θ S , ν θ is notation for ν being
P( ) ( F) ∈ P( ) ⪯
absolutely continuous with respect to θ, see [2, pg. 442] for the definition. If ν θ, then dν denotes the
⪯ dθ
Radon-Nikodym derivative of ν with respect to θ; any two choices of Radon-Nikodym derivative differ
onlyon aθ-nullset, see[2,Thm. 32.2]. TherelativeentropyD ν θ ofν withrespect toθ isdefined by
( ∥ )
⎧ log dν dν , ifν θ,
D ν θ ⎪⎪∫S( dθ) ⪯ (1)
( ∥ ) ∶=⎨ ifν θ.
⎪⎪∞ ⪯̸
⎩
From theconvexityofthexlogxfunctionfornonnegativex, onecan check that D ν θ 0.
( ∥ )≥
Here,andintherestofthepaper, isnotationforequalitybydefinition. Logarithmscanbeassumed
∶=
to beto the natural base. For two measurablefunctionsf and g on S, , not necessarily bounded, and
( F)
η S , f g denotes equality of f and g except possibly on an η-null set. Similarly, for C,D ,
η
∈ P( ) = ∈ F
C D denotesequalityofC andD uptoη-nullsetsandC D denotesthecontainmentofC inD up
η η
= ⊆
to η-nullsets.
The variational characterization in (2) below of exponential integrals of bounded measurable func-
tionsiselementary. Foranyµ S and g S wehave
∈P( ) ∈B( )
log egdµ sup gdθ D θ µ sup gdθ D θ µ . (2)
∫S =θ∈P(S)(∫S − ( ∥ ))=θ∈P(S)∶ θ⪯µ(∫S − ( ∥ ))
We providea proofin AppendixA.
For any α R 0,1 , and ν,θ S , the Re´nyi divergence R ν θ is defined as in eqn. (2.1) of
α
∈ /{ } ∈ P( ) ( ∥ )
[1], by first definingit forα 0,α 1, by
> ≠
ifα 1and ν θ
⎧
Rα(ν∥θ)∶=⎪⎪⎨∞1 log ν′ αdθ other>wise, ⪯̸ (3)
⎪⎪α(α−1) ∫{ν′θ′>0}(θ′)
⎩
where ν′ dν and θ′ dθ, where η S is an arbitrary probability distribution such that ν η and
∶= dη ∶= dη ∈ P( ) ⪯
θ η. It is straightforward to check that every choice of η, subject to the absolutecontinuity conditions,
⪯
resultsin thesamevalueoftheRe´nyientropy. Then, forα 0,weusethedefinition
<
R ν θ R θ ν . (4)
α 1−α
( ∥ ) ∶= ( ∥ )
Remark1. EventhoughthedefinitionofR ν θ isbrokenupintocasesabove,asingleformulawould
α
( ∥ )
work, ifsuitablyinterpreted. Onecouldwrite
1
R ν θ log ν′ α θ′ 1−αdη , forallα R 0,1 .
α
( ∥ ) = α α 1 ∫S( ) ( ) ∈ /{ }
( − )
In this formula, if η ν′ 0,θ′ 0 0 and α 1, then because ν′ α θ′ 1−α (ν′)α on this event,
( > = ) > > ( ) ( ) = (θ′)α−1 = ∞
we are forced to intepret R ν θ as being . A similar argument forces us to interpret R ν θ as
α α
( ∥ ) ∞ ( ∥ ) ∞
if η ν′ 0,θ′ 0 0 and α 0. Rather than requiring of the reader the mental gymnastics needed to
( = > ) > <
keep trackofsuchinterpretations,weprefer to breakthediscussionup intocases.
3
Remark 2. It is clear that R ν θ 0 (possibly ) if α 1 or α 0. For 0 α 1, an applicationof
α
Ho¨lder’s inequalitywithp 1(a∥nd)q≥ 1 (so 1 ∞1 1)g>ives < < <
∶= α ∶= 1−α p + q =
ν′
αdθ ν′ α θ′ 1−αdη
∫{ν′θ′>0}(θ′) = ∫{ν′θ′>0}( ) ( )
ν′ α θ′ 1−αdη
= ∫S( ) ( )
α 1−α
≤ ν′dη θ′dη
(∫S ) (∫S )
= 1.
Hence we also have R ν θ ≥ 0 (possibly ) if 0 < α < 1. Note in particular that if η ν′θ′ > 0 = 0,
α
then R ν θ = fora(llα∥ ∈)R 0,1 . ∞ ( )
α
( ∥ ) ∞ /{ }
3 Statement of the main result in the i.i.d. case
Ourmainresult inthei.i.dcaseisthefollowingvariationalcharacterization ofRe´nyi divergence.
Theorem 1. Let α ∈R 0,1 and ν,θ ∈ S . Then, ifα >1, wehave
/{ } P( )
1 1
R ν θ = sup D µ θ D µ ν , (5)
α
( ∥ ) (α ( ∥ )− α 1 ( ∥ ))
{µ∈P(S)∶ µ⪯ν}
−
while, if0<α<1,we have
1 1
R ν θ = inf D µ θ D µ ν , (6)
α
( ∥ ) {µ∈P(S)∶ µ⪯ν,µ⪯θ}(α ( ∥ )− α 1 ( ∥ ))
−
and, ifα <0, wehave
1 1
R ν θ = sup D µ θ D µ ν . (7)
α
( ∥ ) (α ( ∥ )− α 1 ( ∥ ))
{µ∈P(S)∶ µ⪯θ}
−
Further, when 0 < α < 1, one can find µ ∈ S , µ ⪯ ν, µ ⪯ θ, achieving the infimum on the RHS of (6),
whenever µ∈ S µ⪯ν,µ⪯θ is nonPe(mp)ty. ◻
{ P( ) ∶ }
Remark 3. The case by case structure of this result is partly a consequence of the normalization cho-
sen for the Re´nyi divergences (which is necessary to make Re´nyi divergence nonnegative) and partly a
consequenceof theneed toapplythecorrect absolutecontinuityconditions. If itconsidered desirableto
write a singeformulacovering allcases, this can bedonebyconsideringΛ ν θ ∶=α α−1 R ν θ ,
α α
for α∈R 0,1 . Then onehasthesingleformula ( ∥ ) ( ) ( ∥ )
/{ }
Λ ν θ = sup α−1 D µ θ −αD µ ν ,
α
( ∥ ) or (( ) ( ∥ ) ( ∥ ))
{µ∈P(S)∶ µ⪯ν µ⪯θ}
for all α ∈ R 0,1 . Note, however, that the set over which the supremum is being taken need not be
convex ingen/e{ral. T}his isessentialtoavoidencounteringexpressionsoftheform∞−∞.
4
4 Discussion
Atar,ChowdharyandDupuis[1]haverecentlyestablishedavariationalformulaforexponentialintegrals
ofbounded measurablefunctions. Thisis establishedintwo forms. Foranyα ∈R 0,1 ,ν ∈ S , and
/{ } P( )
g ∈ S , eqn. (2.6)of[1]statesthat
B( )
1 1
log e(α−1)gdν = inf log eαgdθ+R ν θ , (8)
α−1 ∫S θ∈P(S)(α ∫S α( ∥ ))
whileeqn. (2.7)of[1]statesthat foranyα ∈R 0,1 ,θ ∈ S , and g ∈ S wehave
/{ } P( ) B( )
1 1
log eαgdθ = sup log e(α−1)gdν−R ν θ . (9)
α ∫S ν∈P(S)(α−1 ∫S α( ∥ ))
Itisstraightforwardtoexhibittheequivalenceofthesetwoforms. Forinstance,assuming(8),letβ ∶=1−α
and h∶=−g, and concludethatfor allβ ∈R 0,1 ,ν ∈ S , and h∈ S wehave
/{ } P( ) B( )
1 1
−β log∫Seβhdν =θ∈iPn(fS)(1−β log∫Se(β−1)hdθ+R1−β(ν∥θ)) ,
orequivalentlythat
1 1
log eβhdν = sup log e(β−1)hdθ−R θ ν ,
β ∫S θ∈P(S)(β−1 ∫S β( ∥ ))
which is (9). One can similarlygo in theoppositedirection. We willtherefore focus onlyon theform in
(9). AsobservedinRemark2.3of[1],takingthelimitasα→1in(9)recoverstheelementaryvariational
formulaforexponentialintegralsofboundedmeasurablefunctionsin(2).
The structure of Theorem 1 is motivated by the variational characterization in (9). We will now
demonstratethatTheorem1isatleastasstrongas(9)byderiving(9)fromTheorem1andtheelementary
variationalformula(2).
First of all, we show that for any α ∈ R 0,1 , θ ∈ S , and g ∈ S one can find ν ∈ S
/{ } P( ) B( ) P( )
achievingthesupremumin(9). ThisproofdoesnotdependonTheorem1and(2). Infact,thesupremum
isachievedbythechoice 1e−gdν =dθ,whereZ isthenormalizationfactor, anditiselementarytoprove
Z
this. Forcompleteness,aproofis includedinAppendixB.
It remainstoprovethatforany α∈R 0,1 ,g ∈ S , and θ,ν ∈ S , wehave
/{ } B( ) P( )
1 1
log eαgdθ ≥ log e(α−1)gdν−R ν θ . (10)
α ∫S α−1 ∫S α( ∥ )
AssumingthetruthofTheorem 1, and using(2), thisisprovedinAppendixC.
5 Proof of Theorem 1
We nowproveTheorem 1.
5
Consider first the case α > 1. Suppose ν ⪯̸ θ. Then the LHS of (5) is ∞. Also, in this case, we can
choose µ ∈ S such that µ ⪯ ν but µ ⪯̸ θ, which makes the RHS of (5) also equal to ∞. Thus we may
P( )
assumethatν ⪯θ. GivenK >0sufficientlylarge, define µ ∈ S by
K
P( )
1
µ′ ∶= ν′ α θ′ 1−α1 ν′ α θ′ 1−α ≤K
K Z ( ) ( ) (( ) ( ) )
K
where η ∈ S ×S is chosensuch thatθ ⪯η,and wedefineν′ ∶= dν, θ′ ∶= dθ, and µ′ ∶= dµK. Further,
M( ) dη dη K dη
Z ∶= ν′ α θ′ 1−αdη ,
K
∫{(ν′)α(θ′)1−α≤K}( ) ( )
and K sufficientlylargemeans thatZ >0. Wenotethatµ ⪯ν (and soµ ⪯θ). Then
K K K
1 1
D µ θ − D µ ν
α ( K∥ ) α−1 ( K∥ )
1 µ′ 1 µ′
= log K dµ − log K dµ
α ∫{µ′ >0}( θ′ ) K α−1∫{µ′ >0}( ν′ ) K
K K
1 ν′ α 1 θ′ 1−α
= log ( ) dµ − log ( ) dµ
α ∫{µ′K>0}( ZK(θ′)α) K α−1 ∫{µ′K>0}( ZK(ν′)1−α) K
1
= logZ ,
α α−1 K
( )
which, as K →∞, converges to
1 ν′ α 1
R ν θ = log dθ = log ν′ α θ′ 1−αdη .
α( ∥ ) α α−1 ∫{ν′θ′>0}(θ′) α α−1 ∫S( ) ( )
( ) ( )
It remains to show that, in the case α > 1, for all ν,θ ∈ S such that ν ⪯ θ, we have, for all µ ∈ S
P( ) P( )
such thatµ⪯ν, theinequality
1 1
R ν θ ≥ D µ θ − D µ ν . (11)
α( ∥ ) α ( ∥ ) α−1 ( ∥ )
Pick η ∈ S such that θ ⪯ η (so we also have ν ⪯ η and µ ⪯ η), and let ν′ ∶= dν, θ′ ∶= dθ, and µ′ ∶= dµ.
P( ) dη dη dη
MultiplyingtheRHS of(11) byα α−1 gives
( )
µ′ µ′ ν′ α θ′ 1−α
α−1 log dµ−α log dµ= log ( ) ( ) dµ.
( )∫{ν′θ′µ′>0} θ′ ∫{ν′θ′µ′>0} ν′ ∫{ν′θ′µ′>0} µ′
On theotherhand, wehave
ν′ ν′ θ′
α α−1 R ν θ =log αdθ ≥log α dµ ,
α
( ) ( ∥ ) ∫{ν′θ′>0}(θ′) ∫{ν′θ′µ′>0}(θ′) µ′
so (11)followsfrom theconcavity ofthelogarithm.
Next,considerthecasewhen 0 <α <1. Pick η ∈ S such that ν ⪯η and θ ⪯η, and let ν′ ∶= dν and
P( ) dη
θ′ ∶= dθ. If ν′θ′ >0 = ∅, then ν′ αdθ =0, and so
dη { } η ∫{ν′θ′>0}(θ′)
1 ν′
R ν θ ∶= log αdθ =∞.
α( ∥ ) α α−1 ∫{ν′θ′>0}(θ′)
( )
6
But wealsohave µ∈ S ∶ µ⪯ν,µ ⪯θ =∅, sotheRHSof(6)equals∞. Wemaythereforeassume
that η ν′θ′ > 0 >{0. NPo(w,)an application o}f Ho¨lder’s inequality with p ∶= 1 and q ∶= 1 (so 1 + 1 = 1)
( ) α 1−α p q
gives
ν′
αdθ = ν′ α θ′ 1−αdη
∫{ν′θ′>0}(θ′) ∫{ν′θ′>0}( ) ( )
= ν′ α θ′ 1−αdη
∫S( ) ( )
α 1−α
≤ ν′dη θ′dη
(∫S ) (∫S )
= 1
Let µ ∈ S be defined by µ′ ∶= 1 ν′ α θ′ 1−α, where Z ∶= ν′ α θ′ 1−αdη. Note that R ν θ =
1 loPgZ( .)Wehaveµ⪯ν and µ⪯Zθ(, a)s r(equ)ired ontheRHSo∫fS((6).)N(ow), α( ∥ )
α(α−1)
1 1
D µ θ − D µ ν
α ( ∥ ) α−1 ( ∥ )
1 µ′ 1 µ′
= log dµ− log dµ
α ∫{µ′>0}( θ′) α−1∫{µ′>0}( ν′)
1 ν′ α 1 θ′ 1−α
= log ( ) dµ− log ( ) dµ
α ∫{µ′>0}( Z θ′ α) α−1 ∫{µ′>0}( Z ν′ 1−α)
( ) ( )
1
= logZ ,
α α−1
( )
which equals R ν θ . It remains to show that, in the case 0 < α < 1, for all ν,θ ∈ S such that
α
( ∥ ) P( )
η ν′θ′ >0 >0, wehave, forallµ∈ S suchthat µ⪯ν and µ⪯θ, theinequality
( ) P( )
1 1
R ν θ ≤ D µ θ − D µ ν . (12)
α( ∥ ) α ( ∥ ) α−1 ( ∥ )
To seethis,notethat
1 1
α 1−α D µ θ − D µ ν
( )(α ( ∥ ) α−1 ( ∥ ))
= 1−α D µ θ +αD µ ν
( ) ( ∥ ) ( ∥ )
µ′ µ′
= 1−α log dµ+α log dµ
( )∫{µ′>0}( θ′) ∫{µ′>0}( ν′)
µ′
= log dµ
∫{µ′>0}( ν′ α θ′ 1−α)
( ) ( )
ν′ α θ′ 1−α
= f ( ) ( ) µ′dη
∫S ( µ′ )
≥f ν′ α θ′ 1−αdη
(∫S( ) ( ) )
=−log ν′ α θ′ 1−αdη
∫S( ) ( )
=α 1−α R ν θ .
α
( ) ( ∥ )
7
where f ⋅ is the negative logarithm function, which is decreasing and convex. This establishes (12).
()
Note that we have also estabished the claim in Theorem 1 that when 0 < α < 1 one can find µ realizing
theinfimumin(6)whenever µ∈ S ∶ µ⪯ν,µ ⪯θ isnonempty.
{ P( ) }
It remains to consider the case where α < 0. Let β ∶= 1−α. Then β > 1. By definition R ν θ =
α
( ∥ )
R θ ν . However,wehavealready provedthat
β
( ∥ )
1 1
R θ ν = sup D µ ν − D µ θ .
β( ∥ ) (β ( ∥ ) β−1 ( ∥ ))
{µ∈P(S)∶ µ⪯θ}
Thisreads
1 1
R ν θ = sup D µ θ − D µ ν ,
α( ∥ ) (α ( ∥ ) α−1 ( ∥ ))
{µ∈P(S)∶ µ⪯θ}
which establishes(7)in thiscasealso andcompletestheproofofTheorem 1.
6 Re´nyi divergencerate between stationary finite state Markov chains
In this section we set the stage to present analogs of the preceding results involving the Re´nyi diver-
gence rates between twostationaryfinitestateMarkovchains. Extensionstogeneral statespace Markov
processes in both discrete and continuous time of a form similar to those we will present for stationary
finite state Markov chains no doubt exist, under suitable conditions on the transition kernel, but may be
considered topicsforfuturework.
From this point onwards in this document we take S = 1,...,d and to be comprised of all the
{ } F
subsetsofS. Let S×S denotethesetofMarkovprobabilitydistributionson S×S, × ,where
M( ) ( F F)
ν ∈ S ×S if ν i,j ≥ 0 for all i,j ∈ S ×S, ∑ ν i,j = 1, and ν k,∗ = ν ∗,k for all k ∈ S,
i,j∈S
M( ) ( ) ( ) ( ) ( ) ( )
where ν k,∗ ∶= ∑ ν k,j and ν ∗,k ∶= ∑ ν i,k . Here × is comprised of all the subsets of
j∈S i∈S
( ) ( ) ( ) ( ) F F
S ×S.
Given ν ∈ S ×S , let S ∶= k ∶ ν k,∗ > 0 . S is a subset of S, and is called the support
ν ν
M( ) { ( ) }
of ν. For i ∈ S and j ∈ S, we define ν j∣i) ∶= ν(i,j). Note that ν(j∣i) = 0 if i ∈ S and j ∉ S , and
ν ( ν(i,⋅) ν ν
∑ ν(j∣i) = 1. For i ∉ S , we define ν(j∣i) = 0 for all j. This may seem strange, but is an important
j∈S ν
notationalconventionfortheequationswearegoingtowrite. Notethat ∑ ν(j∣i)=0 fori∉S .
j∈S ν
Given ν,θ ∈ (S × S) we say ν is absolutely continuous with respect to θ, denoted ν ⪯ θ, if
θ(i,j) = 0 ⇒ ν(iM,j) = 0 for all (i,j) ∈ S ×S. The relative entropy D(ν∥θ) of ν with respect to θ is
defined by
⎧
⎪⎪∑ ν(i,j)log ν(j∣i) , ifν ⪯θ,
D(ν∥θ)∶=⎨ i,j∈Sν θ(j∣i) (13)
⎪⎪∞ ifν ⪯̸θ.
⎩
It can bechecked thatD(ν∥θ)≥0.
We need certain basic facts about the asymptotic properties of iterated powers of square matrices
withnonnegativeentries. Wewillstatethesefactsinnarrativeform. Proofscanbeextractedfromseveral
booksthatprovidestandardtreatmentsofthetheoryofnonnegativematricesorfinitestateMarkovchains,
see e.g. [9, Chap. 1].
LetM = m bead×d matrixwithnonnegativeentries. Then thelimit
ij
[ ]
1
ρ M ∶= lim log∑m(n) i,j , (14)
( ) n→∞n ( )
i,j
8
exists, where m(n) i,j denotes the i,j entry of Mn. We can associate to M a directed graph on the
( ) ( )
vertex set 1,...,d , where we have a directed edge from i to j iff m > 0. This graph may have self
ij
{ }
loops. Thenρ M =−∞iffthisdirectedgraphdoesnothaveadirectedcycle. Otherwiseρ M isfinite.
( ) ( )
We callρ M thegrowthrateofM.
( )
Supposeρ M isfinite. Wesayµ∈ S×S isabsolutelycontinuouswithrespecttoM ifµ i,j >
0⇒m i,j >(0fo)ralli,j ∈S Letµ1,µM2 ∈( S)×S beabsolutelycontinuouswithrespect toM(. Th)en
so is 21(µ1 +) µ2 . Thus there is a maximumMe(lement)µ ∈ S ×S that is absolutely continuous with
( ) M( )
respecttoM, inthesensethateveryotherν ∈ S×S thatisabsolutelycontinuouswithrespecttoM
M( )
satisfies ν ⪯ µ. This maximum element need not be unique. Pick any such maximum element, call it τ.
Let M′ ∶= m i,j 1 i,j ∈S . Thenρ M′ =ρ M .
τ
[ ( ) ( )] ( ) ( )
Let µ ∈ S × S , which we also think of as a nonnegative d × d matrix. The support of µ can
M( )
be uniquely written as a disjoint union of subsets, called classes, S = ⊍l C , for some l ≥ 1, such that
µ k=1 k
µ i,j =0ifi,j ∈S areindistinctclasses,andsuchthat,foreach1≤k ≤l,ifweconsidertherestriction
µ
( )
ofthedirectedgraphassociatedtoµtotheverticesintheclassC ,thenthisdirectedgraphisirreducible,
k
in thesensethat thereisadirected pathin thegraph between anypairofverticesin C .
k
Given µ ∈ S ×S and a d×d matrix M with nonnegative entries, we say M is compatible with
M( )
µ if m i,j > 0 ⇔ µ i,j > 0. Let S = ⊍l C be the decomposition of the support of µ into classes.
( ) ( ) µ k=1 k
For each 1 ≤ k ≤ l, the restriction of M to the coordinates in C defines a C × C irreducible matrix
k k k
∣ ∣ ∣ ∣
with nonnegative entries. This matrix has an associated Perron-Frobenius eigenvalue, which we denote
by λk M . We have λk M > 0 for all 1 ≤ k ≤ l. We have ρ M = logmax1≤k≤lλk M . Also, for
( ) ( ) ( ) ( )
each 1 ≤ k ≤ l, the restriction of M to the coordinates in C has a left eigenvector associated to the
k
eigenvalue λ M , which has all its coordinates strictly positive and is unique up to scaling, and also a
k
( )
right eigenvector associated to the eigenvalue λ M , which has all its coordinates strictly positive and
k
( )
is uniqueupto scaling.
Given ν ∈ S ×S , what we mean by the stationary Markov chain defined by ν is the following:
M( )
for each n ≥ 1 define a probability distribution ν on Sn, , where is comprised of all subsets of
n n n
( F ) F
Sn, by setting
ν1 k = ν k,∗ , forallk ∈S ,
( ) ( )
ν2 i,j = ν i,j , foralli,j ∈S ,
( ) ( )
⋮
n−1
νn i1,...,in = ν i1,i2 ∏ν ik+1 ik , forall i1,...,in ∈Sn ,
( ) ( ) ( ∣ ) ( )
k=2
⋮ .
It is straightfowardtocheck thatforall n≥2 and ν,θ ∈ S ×S wehave
M( )
ν ⪯θ ⇔ν ⪯θ . (15)
n n
The followingfact, which will be very useful later, is easy to verify from thedefinitions. It holdsfor
all ν,θ ∈ S ×S .
M( ) 1
D ν θ = lim D ν θ , (16)
( ∥ ) n→∞n ( n∥ n)
9
where on the RHS of this defintion the notation D ν θ refers to the relative entropy between proba-
n n
( ∥ )
bilitydistributionson Sn, .
n
( F )
We are now in a position where we can state the analog for stationary finite state Markov chains
of the elementary variational formula (2). Let G = g i,j ∈ Rd×d and µ ∈ S × S . We have
[ ( )] M( )
the following variational characterization of the growth rate of the exponential integral of G along the
stationaryMarkovchain defined by µ.
ρ eg(i,j)µ j i = sup ∑ g i,j θ i,j −D θ µ (17)
([ ( ∣ )]) θ∈M(S×S)(i,j∈S ( ) ( ) ( ∥ ))
= sup ∑ g i,j θ i,j −D θ µ .
θ∈M(S×S)∶ θ⪯µ(i,j∈S ( ) ( ) ( ∥ ))
The proof is in Appendix D. The result is standard, being Varadhan’s characterization of the spectral
radius ofnonnegativematrices,seee.g. [3, Exer. 3.1.19].
We are also in a position to define the Re´nyi divergence rates between two stationary finite state
Markov chains. This definition is classical, see e.g. the paper of Rached, Alajaji, and Campbell [7],
which also considers the nonstationary case, and the references therein. Given ν,θ ∈ S × S and
α ∈R 0,1 ,we definetheRe´nyidivergencerateofν withrespect toθ, denoted R ν θM, b(y )
α
/{ } ( ∥ )
1
R ν θ ∶= lim R ν θ , (18)
α( ∥ ) n→∞n α( n∥ n)
whereontheRHSofthisdefintionthenotationR ν θ refers totheRe´nyidivergencebetweenprob-
α n n
( ∥ )
abilitydistributionson Sn, defined as in(3)and (4). Theproofsoftheexistenceofthelimitin(18)
n
( F )
aswellasofthepropertiesoftheRe´nyidivergencerateofinteresttous,whicharestatedinthefollowing
proposition,are inAppendixE.
Proposition1. Givenν,θ ∈ S×S ,theRe´nyidivergencerate,asdefinedin(18),satisfiesthefollowing
M( )
properties:
⎧∞ ifα >1 andν ⪯̸θ ,
Rα(ν∥θ) =⎪⎪⎨ 1 ρ ν j i αθ j i 1−α if0<α <1 orif α >1 andν ⪯θ ,
⎪⎪α(α−1) ([ ( ∣ ) ( ∣ ) ])
⎩
and
Rα ν θ =R1−α θ ν , ifα <0 .
( ∥ ) ( ∥ )
7 Main results in the Markov case
Our first main result in the Markov case is the following variational characterization of the Re´nyi diver-
gence rate, whichis adirectanalog ofTheorem1.
Theorem 2. Let α ∈R 0,1 and ν,θ ∈ S ×S . Then, if α>1,we have
/{ } M( )
1 1
R ν θ = sup D µ θ − D µ ν , (19)
α( ∥ ) {µ∈M(S×S)∶ µ⪯ν}(α ( ∥ ) α−1 ( ∥ ))
10