A Tight Lower Bound on the Mutual Information of a Binary and an Arbitrary Finite Random Variable in Dependence of the Variational Distance

0.08 MB·English

by  A. G. Stefani

A Tight Lower Bound on the Mutual Information of
a Binary and an Arbitrary Finite Random Variable
in Dependence of the Variational Distance
Arno G. Stefani, Johannes B. Huber Christophe Jardin, Heinrich Sticht
Institute for Information Transmission (LIT) Bioinformatics, Institute for Biochemistry
FAU Erlangen-Nuremberg FAU Erlangen-Nuremberg
Erlangen, Germany Erlangen, Germany
Email: {stefani, huber}@LNT.de Email: {christophe.jardin, h.sticht}@biochem.uni-erlangen.de
3 Abstract—”THISPAPERISELIGIBLEFORTHESTUDENT In the next section the notation is fixed. In section III the
1 PAPER AWARD”. details of the method are given. In section V some numerical
0
In this paper a numerical method is presented, which finds a examples are shown.
2
lower bound for the mutual information between a binary and
n an arbitrary finite random variable with joint distributionsthat II. NOTATIONALSETUP
a have a variational distance not greater than a known value to LetX,Y bea pairoffinite discreterandomvariables,with
J a known joint distribution. This lower bound can be applied to
mutual information estimation with confidence intervals. joint probability distribution
8
2 pXY ={pXY(i,j):i=1,2,...,Mx; j =1,2,...,My}.
I. INTRODUCTION Here X ∈ X and Y ∈ Y and it is w.l.o.g. assumed that
]
T X = {1,2,...,M } and that Y = {1,2,...,M }. The
A tight lower bound for the mutual information between x y
I marginal probability distributions are p = {p (i) : i =
. a binary and an arbitrary finite random variable with joint X X
s 1,2,...,M } and p = {p (j) : j = 1,2,...,M }. They
c distributions that have a variational distance not greater than x Y Y y
are calculated from the joint probalility distributionsas usual.
[ a known value to a known joint distribution can be found by
The conditional probability distributions are
minimizing over this set of joint distributions. Unfortunately,
2
v in general this minimization problem is hard to solve, since pY|X ={pY|X(j|i):i=1,2,...,Mx; j =1,2,...,My},
7 the mutual information is not convex in the joint distribution. p ={p (i|j):i=1,2,...,M ; j =1,2,...,M }.
3 X|Y X|Y x y
Therefore this minimization problem is split up into two
9 Itisdefinedthatp p =p p =p p =p p =
subproblems. Y|X X X Y|X X|Y Y Y X|Y
5 p . The product of the marginal distributions is denoted as
. If the marginal probability of the binary random variable XY
1
0 is fixed, then the mutual informationcan easily be minimized pXpY ={pX(i)pY(j):i=1,2,...,Mx; j =1,2,...,My}.
3 over the conditional probabilities of the second random vari- For any two joint probability distributions p , q the
XY XY
1 able,sincethemutualinformationisconvexintheconditional relative entropy or Kullback-Leiblerdistance [1] is defined as
: probabilities [1, Theorem 2.7.4] and the set of conditional
Xiv probabilities is convex (see Theorem 1) and therefore this D(pXYkqXY)= XMx XMy pXY(i,j)logpXY(i,j) (1)
optimization problem is convex. This constitutes the first q (i,j)
XY
i=1j=1
r subproblem which can easily be solved by standard methods
a and the mutual information between X and Y [1] as the
for convex optimization.
relative entropy between the joint probability distribution and
In the second subproblem, having a closer look on the
product of the marginal probability distributions of X and Y
marginal probability distribution of the binary random vari-
able, one first recognizes that this is only one-dimensional I(X;Y)=I(pXY)=D(pXYkpXpY). (2)
since the two probabilities have to sum up to 1. Next, the
All logs are assumed to be natural if not stated otherwise.
variational distance between the joint probabilities is greater
The variational distance between two joint probability dis-
or equal than the variational distance of the marginal proba-
tributions is defined as
bilities, asis shownin (5).Thereforeonecan simplygenerate
V(p ,q )=kp −q k
sufficiently many marginal probability distributions equidis- XY XY XY XY 1
tantlyintheonedimensionleft,solvethefirstsubproblemfor Mx My
everyofthesemarginalprobabilitydistributionsandreturnthe =XX|pXY(i,j)−qXY(i,j)|,
smallest mutual information calculated that way. i=1j=1
and similarly for the marginal distributions. It can be easily probability is fixed. Therefore the relative entropy, which is
seen, thatV(·,·)∈[0,2] forany two probabilitydistributions. a built in function in CVX and is convex in its two input
arguments, is used. Then it can be seen that
III. RESULTS
I(X;Y)=I(q q )=D(q q kq q ),
First it is shown that set of all conditional probability X Y|X X Y|X X Y
distributions constrained by a maximal variational distance is and q (i)q (j|i) are affine functions of q (j|i) as
X Y|X Y|X
convex. q (i)q (j) = q (i)( q (j|i)q (i)) are. Hence, the
X Y X Pi Y|X X
Theorem 1: Let p = p p be any fixed joint prob- convexity of D(·,·) is preserved [2, section 2.3.2], and it
XY X Y|X
ability distribution of any two two discrete finite random is straightforward to implement the minimization problem in
variables X, Y, let q be any fixed probability distribution Corollary 1 with CVX with this knowledge.
X
of X and let ǫ be any fixed number ∈ [0,2]. Then the set Next the second subproblem, namely the minimization of
Q={q | V(q q ,p )≤ǫ} is convex. the mutualinformationoverthe marginalprobabilitydistribu-
Y|X X Y|X XY
Proof:Letq1 ,q2 beanytwoconditionalprobability tion q , is solved. Herefore it is first shown that
Y|X Y|X X
distributions ∈Q. Then one only has to show that the convex
V(q ,p )=kq −p k
combination qλ = λq1 +(1−λ)q2 , with λ ∈ [0,1] X X X X 1
Y|X Y|X Y|X
is also in Q. Before this is done, it is defined that q1 = Mx
q q1 , q2 =q q2 and qλ =λq1 +(1−λ)qX2Y = =X|qX(i)−pX(i)|
X Y|X XY X Y|X XY XY XY i=1
q qλ . Now, to proof that qλ ∈Q, one only has to show
thXatYV|X(qXqYλ|X,pXY)≤ǫ. HeYr|eXfore =XMx (cid:12)(cid:12)(cid:12)XMy(qXY(i,j)−pXY(i,j))(cid:12)(cid:12)(cid:12)
(cid:12) (cid:12)
V(qXqYλ|X,pXY)=V(qXλY,pXY) i=1(cid:12)(cid:12)j=1 (cid:12)(cid:12)
=(cid:13)(cid:13)qXλY −pXY(cid:13)(cid:13)1 ≤ǫ, (3) ≤XMx XMy |qXY(i,j)−pXY(i,j)|
where the fact that any norm ball is convex[2, Section 2.2.3] i=1j=1
has been used in (3). Also, the further constraints implied by =V(qXY,pXY)
theprobabilitysimplex(whichisconvex)arenoproblemsince ≤ǫ. (5)
an intersection of convex sets is always convex [2, Section
Thereforeonlyq withV(q ,p )≤ǫhavetobeconsidered.
2.3.1]. X X X
Untilhereallresultsareapplicableto anyfinite M , butfrom
Since the empty set is convex, no restriction on V(p ,q ) x
X X here the restriction M = 2 applies. In this case q is one
(e.g. V(p ,q )≤ǫ) is necessary. x X
X X dimensionalobviously,and the set of all q is simply {q =
Corollary 1: Let p be any fixed joint probability distri- X X
XY {min(p (1) + γ,1),max(p (2) − γ,0)} | γ ∈ [−ǫ, ǫ]}.
bution of any two two discrete finite random variables X, Y, X X 2 2
Practically, the minimization problem
let q be any fixed probability distribution of X and let ǫ be
X
any fixed number ∈[0,2]. Then, the optimization problem min I(q ) (6)
XY
qXY : V(qXY,pXY)≤ǫ
min I(q q ) (4)
X Y|X is then simply solved by generating sufficiently many q
qY|X : V(qXqY|X,pXY)≤ǫ X
equidistantly in γ, solve the optimization problem of Corol-
is convex.
lary1foreveryq andreturnthesmallestmutualinformation
X
Proof: The mutual information I(qXqY|X) is a convex calculated that way. Here the number of qXs is considered
function of the conditional probabilities q when q is
Y|X X to be sufficient if one gets a smooth graph for the mutual
fixed, and the set {pY|X | V(qXqY|X,pXY) ≤ ǫ} is convex. informationminimizedovertheconditionalprobabilitiesqY|X
as a function of γ.
Corollary 1 basically says that the optimization problem
given is practically solvable. However, since it is a general IV. DISCUSSION
convex optimization problem, it can still be cumbersome to Together with the bound on the probability of a maximal
find a suitable algorithm with the correct parameters. Fortu- variational distance between the true joint distribution and an
nately the problem can be restated in such a way, that it can empirical joint distribution (see [6], and especially an refine-
be handled by disciplined convex programming (DCP) [3], mentofitwhichdropsthedependenceonthetruedistribution
which worksperfectlywellforthisproblemas can be seen in [4, Lemma 3]) the given bound can be used to construct a
section V. reasonably tight lower bound of the confidence interval for
TheminimizationprobleminCorollary1 cannotbesolved mutual information. Such an application can be found in [8].
in a straightforward manner with DCP, since this would Inmutualinformationestimationwithconfidenceintervals,the
violate the no product rule of DCP (see (1), (2)), also there boundgivenisespeciallyuseful,whenthemarginalprobability
is no built function in CVX (which is the software which distribuition is far from being uniform. Such a situation can
implements DCP) for the mutual informationas a functionof be found in [7]. In the case of two binary random variables
the conditionalprobabilitieswhen thecorrespondingmarginal the results seem to coincide with lower bound of [5].
V. NUMERICAL EXAMPLES
0.064
In the first example (Fig. 1) a distribution p and a
XY
maximal variational distance ǫ was handpicked to show that
0.062
the mutual information minimized over the transitional prob-
abilies q as a function of γ is neither convexnor concave
Y|X
0.06
(even for two binary random variables) and seems to be
not differentiable at γ = 0, as can be seen in Fig. 1. The
parameters chosen therefore are I0.058
p (1,1)=0.017,p (1,2)=0.285
XY XY
0.056
p (2,1)=0.424,p (2,2)=0.274
XY XY
and ǫ=0.3. 0.054
Then,
0.052
I(pXY)≈0.2210 and −0.04 −0.02 γ0 0.02 0.04
min I(q )≈0.0019.
XY
qXY : V(qXY,pXY)≤ǫ Fig.2.
InallfiguresI isequaltotheminimumofI(q q )over
X Y|X
q forfixedq ={min(p (1)+γ,1),max(p (2)−γ,0)},
Y|X X X X
constrained by V(q q ,p ) ≤ ǫ, and 1000 points were
X Y|X XY
generated equidistantly for γ ∈[−ǫ, ǫ]. In the last example (Fig. 3) My = 10 and the following
2 2
joint distribution was chosen at random (rounded for easier
reproducibility)
0.07
p (1,1)=0.101,p (1,2)=0.062,p (1,3)=0.025,
XY XY XY
0.06 p (1,4)=0.088,p (1,5)=0.005,p (1,6)=0.007,
XY XY XY
p (1,7)=0.069,p (1,8)=0.059,p (1,9)=0.080,
XY XY XY
0.05
p (1,10)=0.074,
XY
0.04 p (2,1)=0.103,p (2,2)=0.006,p (2,3)=0.038,
XY XY XY
I p (2,4)=0.002,p (2,5)=0.018,p (2,6)=0.079,
XY XY XY
0.03
p (2,7)=0.049,p (2,8)=0.032,p (2,9)=0.020,
XY XY XY
0.02 p (2,10)=0.020,
XY
and ǫ=0.1.
0.01
Then,
0
−0.1 −0.05 0 0.05 0.1 0.15
γ I(pXY)≈0.1311 and
min I(q )≈0.0369.
XY
Fig.1. qXY : V(qXY,pXY)≤ǫ
In the second example (Fig. 2) M = 5 and the following
y
ACKNOWLEDGMENT
joint distribution was chosen at random (rounded for easier
reproducibility) The authors would like to thank the DFG for supporting
their research with SPP1395 in the projects HU634 7 and
p (1,1)=0.090,p (1,2)=0.098,p (1,3)=0.207,
XY XY XY STI155 3.
p (1,4)=0.064,p (1,5)=0.026,
XY XY
REFERENCES
p (2,1)=0.239,p (2,2)=0.030,p (2,3)=0.104,
XY XY XY
p (2,4)=0.107,p (2,5)=0.035, [1] T.M.CoverandJ.A.Thomas,ElementsofInformationTheory,2nded.
XY XY NewYork:Wiley,2006.
and ǫ=0.1. [2] S.BoydandL.Vandenberghe,ConvexOptimization.CambridgeUniver-
sityPress,2004.
Then, [3] M. C. Grant, S. Boyd, and Y. Ye, CVX: Matlab Software
for Disciplined Convex Programming, 2005. Available at
I(p )≈0.1112 and http://www.stanford.edu/˜boyd/cvx/
XY
[4] S.-W. Ho and R. W. Yeung, “The interplay between entropy and varia-
min I(qXY)≈0.0524. tionaldistance,” IEEETrans.Inform.Theory,vol.56,no.12,pp.5906–
qXY : V(qXY,pXY)≤ǫ 5929,Dec.2010.
0.085
0.08
0.075
0.07
0.065
I 0.06
0.055
0.05
0.045
0.04
−0.04 −0.02 0 0.02 0.04
γ
Fig.3.
[5] A.G.Stefani,J.B.Huber,C.JardinandH.Sticht,“Towardsconfidence
intervals for the mutual information between two binary random vari-
ables,”InProc.WorkshopComputationalSystemsBiology(WCSB2012),
Ulm,Germany,Jun.4–6,2012.
[6] T. Weissman, E. Ordentlich, G. Seroussi, S. Verdu´ and M.J. Weiberger,
“Inequalities for the L1 Deviation of the Empirical Distribution,” Tech.
Rept.,HPLaboratories PaloAlto,HPL-2003-97(R.1),Jun.2003.
[7] O. G. Othersen, A. G. Stefani, J. B. Huber and H. Sticht, “Application
ofInformation TheorytoFeature Selection inProtein Docking,” JMol
Model.,vol.18,no.4,pp.1285–1297,Jul.2012.
[8] A.G.Stefani,J.B.Huber,C.JardinandH.Sticht,“Confidenceintervals
forthemutualInformation,” available athttp://arxiv.org/abs/1301.5942.