Signal Enhancement as Minimization of Relevant Information Loss

0.17 MB·English

by  Bernhard C. Geiger

Signal Enhancement as Minimization of Relevant
Information Loss
Bernhard C. Geiger∗, Gernot Kubin∗
∗Signal Processing and Speech Communication Laboratory, Graz University of Technology, Austria
geiger,gernot.kubin @tugraz.at
{ }
Abstract—Weintroducethenotionofrelevantinformationloss notion of relevance has a long history in machine learning
for the purpose of casting the signal enhancement problem in and neural networks: Plumbley, essentially using the same
information-theoreticterms.Weshowthatmanyalgorithmsfrom
definitionaswedo,explicitlyusedrelevantinformationlossin
machinelearningcanbereformulatedusingrelevantinformation
analyzing the properties of principle component analysis [4],
loss, which allows their application to the aforementioned prob-
3
lem. As a particular example we analyze principle component [5]. The information bottleneck (IB) method and its variants
1
analysis for dimensionality reduction, discuss its optimality, and directly maximize relevant information while minimizing the
0
2 show that the relevant information loss can indeed vanish if complexity of its signal representation [6]. Relevance w.r.t. a
the relevant information is concentrated on a lower-dimensional
specific goal was the motivation for defining an information
n subspace of the input space.
processingmeasureforneuralnetworksin[7],[8].Finally,the
a
J principle of relevant information,being structurally similar to
I. INTRODUCTION
6 the information bottleneck formulation, builds on minimizing
1 It is a widely known fact from information theory, that the relative entropybetween the relevantrandomvariable and
processing a random variable, or a signal, cannot increase its representation [9].
]
T the amount of information it represents. In fact, as the data Conversely, in signal processing the notion of relevant
I processing inequality states, information can only be lost informationhasnotgainedfootholdyet.In thispaperwe thus
s. by passing a signal through a deterministic system. This formulatea definitionof relevantinformationloss(SectionII)
c information loss can – very loosely speaking – be interpreted following the ideas of [4] and discuss its properties from
[
as the differencebetween the informationat the inputand the the view-point of system theory (Section III). On the basis

2 informationattheoutputofthesystem.Incaseofcontinuous- of these properties we then discuss the problem of signal
v valued random variables(which contain an infinite amountof enhancement in Section IV and investigate the connections
5
information) the authors developed a theory quantifying the to the machine learning literature. As a special example of
3
information loss both in absolute and relative terms [1], [2]. signal enhancement we analyze principle component analysis
9
6 We now look at the data processing inequality (DPI) from (Section V), generalizing the results of [4]. Finally, we give
. a different point of view by asking the following question: a brief example of the application of our results to a simple
5
0 How can we justify signal processing, knowing that it can digital communication system (Section VI).
2 onlyreduceinformation?Whenevertechnicalsystemsprepare
1 a physical, information-carryingsignal for human perception, II. A DEFINITION OFRELEVANT INFORMATIONLOSS
v: processing occurs and, presumably, a significant amount of We start with recalling the definition given in [1], where
i information is lost. The only justification for this approach the information loss induced by transforming a (possibly
X
is that the information which is lost is actually not the multidimensional) random variable (RV) X to another RV Y
ar information we are interested in, but rather some nuisance. by a static function g: , , RN was given as
We hope to preserve all the information relevant to us while X →Y X Y ⊆
minimizingthenuisance.Theproblemofsignalenhancement, L(X Y)=sup I(Xˆ;X) I(Xˆ;Y) =H(X Y) (1)
→ − |
as we will argue later, is an optimization problem which can
P (cid:16) (cid:17)
be cast in exactly these terms. Aside from that, each block in where the supremum is over all partitions of the sample
the signal processing chain has the goal of representing the space of X, and where Xˆ is obtainePd by quantizing
X
relevant information such that as little as possible is lost in X accordingly (see Fig. 1(a)). This definition of absolute
the subsequent blocks; as in the sense of [3], preprocessing information loss is accompanied by an expression for the
can improve performance, while postprocessing cannot – we relative information loss in [2].
cannotrecoverlostinformation,butwecanpreventloosing it. As the examples in [2] show, both the absolute and the
It is our goal to make these statements precise by supporting relativedefinitionofinformationlosshavetheirshortcomings,
them with a mathematical framework of relevant information especiallywhenitcomestosystemsgusedforsignalenhance-
loss. ment: Since the expressions only consider the RV X at the
Relevant information and its counterpart, relevant infor- input of the system, they do not take into account that not all
mation loss, are concepts not altogether new. Indeed, the oftheinformationcontainedin X isrelevant,oftenleadingto
P
S Q() Sˆ
·
I(Xˆ;Y) I(Sˆ;X) I(Sˆ;Y)
I(Xˆ;X)
Xˆ Q() X g() Y X g() Y
· · ·
P
(a) Computingthe(absolute) information lossL(X →Y) (b) Computingtherelevant information loss LS(X →Y)
Fig.1. Modelforcomputing theinformation lossofamemorylessinput-output systemg.Qisaquantizer withpartition P.
counter-intuitive results. The main contribution of this work III. PROPERTIES OFRELEVANTINFORMATIONLOSS
lies thus in analyzing the implications of the following
We now analyze the elementary properties of relevant
information loss:
Definition1(RelevantInformationLoss). LetX beanRVon
thesamplespace ,andletY beobtainedbytransformingX Proposition1 (ElementaryProperties). Therelevantinforma-
X
with a static function g. Let S be another RV on the sample tion loss from Definition 1 satisfies the following properties:
space representing relevant information. The information 1) L (X Y) I(S;X) H(S)
loss reSlevant w.r.t. S is defined as 2) LS(X →Y)≤L (X ≤Y)=0
S Y
→ ≥ →
3) L (X Y) L (X Y) = L(X Y), with
S X
→ ≤ → →
equality if X is a function of S.
LS(X →Y)=sup I(Sˆ;X)−I(Sˆ;Y) =I(X;S|Y). 4) LS(X →Y)=H(S|Y) if S is a function of X.
P (cid:16) (cid:17) (2) Proof: The first property results immediately from the
wherethe supremumisoverallpartitionsofthe samplespace definition, while the second property is due to the fact that Y
, and where Sˆ is obtained by quantizing S accordingly. is a function of X. The third property results from making
S
X the relevant RV, thus making (2) equal to (1). Note that,
The information loss relevant w.r.t. an RV S is thus the with (1),
difference of mutual informations between S and the input
L (X Y)=I(S;X Y)=H(X Y) H(X Y,S) (3)
and output of the system (see Fig. 1(b)). Equivalently, the S → | | − |
relevant information loss is exactly the information that X L(X Y) (4)
≤ →
containsaboutS whichisnotcontainedinY.Duetothedata
with equality if X is a function of S. The last property, for
processing inequality (e.g., [10]) it is a non-negativequantity,
S = f(X), follows by expanding I(X;S Y) as H(S Y)
i.e., a deterministic system cannot increase the amount of | | −
H(S Y,X)=H(S Y).
available relevant information. | |
The third property is of particular interest. Essentially, it
As we have already pointed out in the introduction, this statesthattherelevantinformationlosscannotexceedthetotal
definition of relevant information loss is not altogether new: informationloss,soupperbounds(e.g.,thosepresentedin [1])
Plumbleyalreadyintroducedthisquantity(named∆IS(X;Y), for the latter can be used for the former as well. In addition
and omitting the supremum assuming finite mutual infor- to that, it suggests an alternative way to define L(X Y):
→
mation between S and X) in the context of unsupervised Namely as
learning in neural networks [4]. The rationale for this new
L(X Y)=sup(I(S;X) I(S;Y)) (5)
measure was to circumvent some shortcomings of Linsker’s
→ −
S
principle of information maximization [11]: While infomax
where the supremum is over all RVs on the sample space .
works well for Gaussian RVs and linear systems, applying
X
Since by Definition 1 relevant information loss is repre-
the same algorithms to non-Gaussian data just maximizes an
sented by a conditional mutual information, it inherits all of
upperboundon the information.Plumbley’sinformationloss,
itsproperties.Inparticular,asweshownext,adataprocessing
conversely,alsoyieldsclosedformsolutionsforGaussiandata,
inequality (DPI) holds:
but in addition to that one can derive upper bounds on the
relevant information loss whenever the data is non-Gaussian. Proposition2(DataProcessingInequality). LetV W X
Thus,minimizingan upperboundonthe informationlosscan Y be a Markov chain. Then, − − −
be assumed to be more promising than maximizing an upper
bound on the transferred information [4]. LW(X Y) LV(X Y). (6)
→ ≥ →
S
S LS(X Z)
→
LS(X Y) LS(Y Z)
→ → Signal
X Y
System 1 Y System 2 Enhancer
X Z
g h
Fig.3. Theproblemofsignalenhancement: Givena(relevant) information
signal S and its observation X, how shall we design a system such that its
Fig.2. Cascadeoftwosystems:Therelevantinformationlossofthecascade
output, Y,represents S asgoodaspossible?
isthesumofrelevant informationlossesoftheconstituent systems.
Y ofthefirstsystem g isusedastheinputtoa secondsystem
Proof: See Appendix.
h, which responds with the RV Z. In [12] it was shown that
The particular usefulness of the DPI (especially in the
the information loss of this cascade is additive, i.e.,
contextofourwork)reliesonthefactthatboth S X g(X)
− −
and f(S) S X are Markov chains. Comparing this to
− − L(X Z)=L(X Y)+L(Y Z). (10)
Proposition 2 one is tempted to believe that the direction of → → →
theinequalitydependsonthefactwhetherf(S) S X Y or Thesameholdsfortherelevantinformationloss,asPlumbley
− − −
S f(S) X Y isaMarkovchain.ThefollowingCorollary showed in [4]. We thus state
− − −
resolves this complication.
Proposition4(InformationLossinaCascadeisAdditive,[4]).
Corollary 1. Let f be a measurable function defined on the Let Y be the RV obtained by transforming X with a static
sample space of S. Then, function g, and let Z be obtained by transforming Y with h.
Then, the information loss relevant w.r.t. S is given as
L (X Y) L (X Y) (7)
S f(S)
→ ≥ →
L (X Z)=L (X Y)+L (Y Z). (11)
with equality if S f(S) X Y is a Markov chain. S → S → S →
− − −
Proof: See Appendix. Proof: See Appendix.
Before proceeding, we note that the third property of
Proposition 1 is an immediate consequence of this corollary. IV. SIGNAL ENHANCEMENT,RELEVANTINFORMATION
We next show that, inherited from the properties of mutual LOSS, ANDIB WITHSIDEINFORMATION
information, also the relevant information loss obeys a chain
We now turn our attention to the signal enhancement
rule:
problem. Often, the information one wants to retrieve is not
Proposition3(ChainRuleofInformationLoss). Theinforma- directly available, but only through some corrupted observa-
toifonrelloevssanLtSR1nV(Xs in→duYce)dwb.yr.ta. afucnocltlieocntiognsaSt1nisfi=es{S1,...,Sn} tdiiosnto:rItnedfortmhraotuiognh-cnaorrnyliinngeasrigsnyasltsemarse,saunpderiamffpeoctseedd bbyyntoimisee-,
varying,dynamiceffects.Itisessentiallythegoalofthesignal
n
processing engineer to mitigate all these adverse effects; to
LS1n(X →Y)= LSi|S1i−1(X →Y). (8) improve the quality of the observation such that as much
i=1
X information as possible can be retrieved from it with little
The proof follows immediately from the chain rule of
effort.
(conditional) information and is thus omitted. However, this
Looking at this task from the perspective of information
chainrulejustifiesourintuitiveunderstandingofthenatureof
theory, we know that, as the data processing inequality dic-
information loss. We thus emphasize
tates, signal enhancement does not mean that one increases
Corollary 2. The information loss L(X Y) induced by a the amount of information in the observation. At best, one
→
function g can be split into relevant (w.r.t. S) and irrelevant can build a system which preserves as much information as
information loss: possible (see Fig. 3). Conversely, noise, distortion, and other
irrelevant components of the observation should be removed
L(X Y)=L (X Y)+L (X Y) (9)
S XS such that information retrieval can be done easily.
→ → | →
Thisiswhereournotionofrelevantinformationlosscomes
Proof:TheprooffollowsfromProposition3andfromthe
in: If we let S be the information carrying signal and X
fact that since S and Y are conditionally independent given
its corrupted observation, our goal is to find a function g
X we have L (X Y)=L(X Y).
XS
→ → such that the relevant information loss is minimized while
Notethateveninasimplescenariowithadditivenoise,i.e.,
simultaneously maximizing the irrelevant information loss.
X =S+N,itis notstraightforwardtojust identifythe noise
One may cast the resulting optimization problem as follows:
N with the irrelevant information. We will show this in one
of the examples in Section VI.
max L (X Y)
Afinalpropertywewanttopresentconcernsthecascadeof g X|S →
systems (see Fig. 2). Assume, forthe moment,thatthe output s.t. L (X Y) C (12)
S
→ ≤
where C is some constant1. In other words, signal enhance- irrelevant signal components. They accompany the relevant
ment is the maximization of irrelevant information loss. information S by an irrelevance variable S¯, assuming condi-
Although in signal processing these information-theoretic tionalindependencebetweenS andS¯ givenX,and minimize
considerations are only slowly gaining momentum, they are the following functional:
quite widespread in machine learning and neural networks,
I(Y;X) β I(Y;S) γI(Y;S¯) (21)
e.g., [5], [9]. In particular, a variational formulation of max- − −
imizing relevant information is the basis of the information At the same time, the entro(cid:2)pyof the system out(cid:3)putY and the
bottleneck method (IB) [6]: irrelevantinformationI(Y;S¯)areminimized,andtherelevant
information I(Y;S) is maximized. As in IB, β and γ are the
min I(Y;X) βI(S;Y) (13)
p(yx) − weights for these three conflicting goals. We can again use
|
our notionsof relevantand, employingCorollary 2, irrelevant
wheretheminimizationisperformedoverallrelationsbetween
information loss to rewrite the optimization problem. If we
the(discrete)RVsY andX andwhereβ isadesignparameter
restrict ourselves to clustering, one obtains
trading compression for preservation of relevant information,
I(S;Y).Whileinprincipletherelationp(y x)canbestochas- min (β 1)L (X Y) (βγ+1)L (X Y) (22)
tic, in many cases the algorithm is used fo|r (hard) clustering g − S → − X|S →
using a deterministic function. We now try to express (13) where again all constant terms have been dropped. Note that
in terms of relevant and irrelevant information loss: Using here X S takes the place of S¯, automatically fulfilling the
|
L (X Y)=I(S;X) I(S;Y) we obtain requirement of conditional independence in [14].
S
→ −
Unlike for IB, by employingside informationit does make
I(S;Y)=I(S;X) L (X Y) (14)
− S → sense to let β : Compression emerges naturally from
→ ∞
where the first term is obviously independent of p(y x). maximizing the irrelevant information loss (due to γ > 0).
|
Restricting ourselves to the clustering problem, i.e., to deter- This is exactly the approach that has been taken up by the
ministic functions Y = g(X), and letting g be the optimal authors of [7], [8], who introduced the following information
◦
solution, we get processing measure for discrete RVs X, Y, and S:
g◦ = argmin I(Y;X) βI(S;Y) (15) ∆P(X Y S)=H(X S) H(Y S)
g − → | | − |
= argmin H(X)+I(Y;X) βI(S;X) α(H(S Y) H(S X)) (23)
g − − − | − |
+βL (X Y) (16) The authorsargue that the first differencein (23) corresponds
S
→ to complexity reduction (i.e., reduction of irrelevant infor-
= argmin L(X Y)+βL (X Y) (17)
g − → S → mation), while the second term accounts for loss of relevant
= argmin L (X Y) L (X Y) information. Indeed, with
S XS
g − → − | →
+βLS(X →Y) (18) LX|S(X →Y)=I(X;X|Y,S)=H(X|Y,S) (24)
= argmin (β 1)L (X Y) L (X Y) =H(X S) H(Y S) (25)
g − S → − X|S → | − |
(19) we can rewrite ∆P(X Y S) with our notions of relevant
→ |
information loss and state a variational utility function (to be
and thus, the optimization problem can be cast as
maximized) for our signal enhancement problem (12):
min (β 1)L (X Y) L (X Y). (20)
g − S → − X|S → ∆P(X →Y|S)=LX|S(X →Y)−αLS(X →Y) (26)
Note that for large β stronger emphasis is placed on Here, α > 0 is a design parameter trading between the loss
minimizing relevant information loss, and a trivial solution of relevant and irrelevant information.
to this problem is obtained by any bijective g. Aside from the IB method and its variants widely used
In the case of discrete RVs this problem can be circum- in machine learning, many other functions and algorithms
vented by another variant of the IB method: The agglomer- inherently solve our problem of signal enhancement: Quan-
ative IB method proposed in [13] starts with Y = X and tizers used in digital communication systems and regener-
iteratively merges elements of the state space of Y such ative repeaters have the goal to preserve the information-
Y
that the relevant information loss is minimized in each step. carrying (discrete) RV as much as possible while removing
By stopping this algorithm as soon as during some merging continuous-valuednoise.Bandpassfilters(althoughnotyetdi-
stepLS(X Y)>C,atleastalocaloptimumofouroriginal rectlytractableusingournotionsofrelevantinformation loss)
→
signal enhancement problem (12) can be found. removeout-of-bandnoisewhileleavingtheinformationsignal
A step further has been made by the authors of [14], unaltered. And finally, methods for dimensionality reduction
who extended the IB method to incorporate knowledge about (suchastheprinciplecomponentanalysis)removeredundancy
andirrelevancewhile tryingto preservethe interestingpartof
1In somecases it may be beneficial to minimize the relevant information
the observation. In the next section we focus on exactly this
losssubjecttoaminimumreduction ofirrelevant information, orevenusea
variational formulation. class of methods.
V. OPTIMALITYOF THEPCA IN TERMS OFRELEVANT Proof: See Appendix.
INFORMATIONLOSS
We now analyze the relevant information loss of the prin- Before proceeding, we need the following
cipal component analysis (PCA). In this regard, we initially
Definition 2. Let X and Y be continuousRVs with arbitrary
follow the reasoning of Plumbley [4] who showed that the
continuous(joint)distribution,andletX andY beGaussian
G G
PCA in some cases minimizes the relevant information loss.
RVs with same (joint) first and second moments. We define
We thengeneralizePlumbley’sresultsandevenshowthatthis
the conditional divergence
loss can vanish. To this end, we define the PCA as
J(X Y)=D(X Y X Y )=h(X Y ) h(X Y). (32)
Y =g(X)=I Y =I WTX (27) | | || G| G G| G − |
M M M
Clearly, this quantity inherits all its properties from the
where X is an N-dimensional continuous-valued input RV,
Kullback-Leibler divergence, e.g., non-negativity, and can be
Y an M-dimensional output RV (composed of the first M
M consideredas a measureof Gaussianity. We want to pointout
elements of Y), and W the matrix of eigenvectors of the
that despite the similarity, this quantity is not to be confused
covariance matrix of X, C =E XXT . Thus, detW =1
X
with negentropy. For negentropy, instead of using a jointly
wanidthWon−e1s=inWtheT.mFauirnthedrimagoornea,ll.et(cid:8)TIhMe PbeC(cid:9)Aani(sMhe×reNu)s-medatfroixr Gaussian distribution for (XG,YG) for computing h(XG|YG)
one uses a Gaussian distribution for X Y = y to compute
dimensionality reduction and, assuming that one has perfect G|
h(X Y = y) before taking the expectation w.r.t. the true
knowledge of the rotation matrix W, the relative information G|
marginal distribution of Y.
loss equals N M, while the absolute information loss is
−N While most results about the optimality of the PCA are
infinite [2].
restricted to the Gaussian case, we now employ conditional
ItisknownthatthePCA minimizesthemeansquarederror
divergence to generalize some of these results. In particular,
for a reconstruction X = WIT Y of X. In addition to
M M M we maintain
that,asLinskerpointedoutin[11],giventhatXisanobserva-
tionofaGaussianRVScorruptedbyGaussiannoise,thePCA Theorem1. LetX=S+N,with N andS independent,and
maximizesthemutualinformationI(S;Y). Fornon-Gaussian let Y be obtained by performing dimensionality-reducing
M
S(andGaussiannoisewithsphericalsymmetry),thePCAnot PCA. If N is iid (i.e., C is a scaled identity matrix) and
N
only provides an upper bound on the mutual information, but more Gaussian than Y in the sense
also an upper bound on the relevant information loss [4], [5].
J(N˜ N˜ ) J(Y Y ) (33)
We nowwanttogeneralizetheresultssuchthatnon-iidand c M c M
| ≤ |
non-Gaussiannoiseistakenintoaccountaswell.Inparticular,
then the PCA minimizes the Gaussian upper bound on the
we show that the relevant information loss can indeed vanish
relevant information loss LS(X YM).
under some circumstances. Let →
Proof: See Appendix.
X=S+N (28)
This theorem essentially generalizes the result by Plumb-
where S and N are the relevant information and the noise, ley[4],[5],whoclaimedthatthePCAminimizestheGaussian
respectively,withcovariancematricesC andC .Wefurther upper bound on the information loss for spherical Gaussian
S N
assume that S and N are independentand, as a consequence, noise (i.e., for Gaussian N with CN being a scaled identity
C = C + C . Let λ , µ , and ν be the sets matrix). In addition to that, it helps justifying our use of in-
X S N i i i
of eigenvalues of C , C{ ,}an{d C} , resp{ect}ively. Let the formationlossinsteadofinformationtransfer;anupperbound
X N S
eigenvalues be ordered descendingly, i.e., on the latter would not be useful in our signal enhancement
problem.
λ λ λ . (29)
1 2 N Asa nextstep,we restrictourselvesto relevantinformation
≥ ≥···≥
We now write which is concentrated on an L M-dimensional subspace,
≤
but drop the requirement that N is iid. We still assume,
YM =IMWTX=IMWTS+IMWTN=S˜M +N˜M. however, that CN (and, thus, CX) is full rank. Note that due
(30) totheseassumptionsλ >0andµ >0foralli,whileν =0
i i i
Asmentionedbefore,YM iscomposedofthefirstM elements for i>L. We are now ready to state
of the vector Y. Conversely, we let Y , S˜ , and N˜ denote
c c c
the last N M elements of the corresponding vectors. If, as Theorem 2 (Bounds for the PCA). Assume that S has
in our case−, the orthogonal matrix W performs the PCA, the covariance matrix CS with at most rank M, and assume that
covariance matrix of Y is a diagonal matrix with the M N is independent of S and has (full-rank) covariance matrix
M
largest eigenvalues of C . We now present CN. Let further N be more Gaussian than Y in the sense
X
Lemma 1. For above signal model, the relevant information J(N˜c N˜M) J(Yc YM) (34)
| ≤ |
loss in the PCA is given by
where Y is obtained by employing the PCA for dimension-
M
LS(X YM)=h(Yc YM) h(N˜c N˜M). (31) alityreduction.Then,therelevantinformationlossisbounded
→ | − |
Channel main point here is that most of the information at the input
of the quantizer is information about the noise signal N. We
X
S 1,1 Q() Y will now make this statement precise.
∈{− } · With the differential entropy of X given by
N ( a,a) γ 1 a 1
∼U − { i} h(X)= log4a+ − log2a (36)
a a
and with h(X S) = log2a we get the amount of relevant
Fig. 4. Digital Communication System: The input signal S is uniformly |
distributed on {−1,1}, the channel adds uniform noise to the signal. The information available at the input of the quantizer:
channel output, X,isquantized usingasetofthresholds {γi}toobtain the
1
decision variable Y. I(X;S)= (37)
a
Choosing now a single quantizer threshold γ = 0, i.e.,
from above by 1
Y = sgn(X) we obtain a binary symmetric channel with
N cross-overprobability P = a 1.WithH ()beingthebinary
LS(X YM) 1ln µ1 (35) entropy function the meutual2−ainformation2 ·thus computes to
→ ≤ 2 µi!
i=YM+1 I(Y;S)=1 H2(Pe)(see,e.g.,[10])andweobtainarelevant
−
where µ is the set of decreasing eigenvalues of C and information loss of
i N
{ }
where ln is the natural logarithm. a 1 a 1
L (X Y)=H ( − ) − . (38)
S → 2 2a − a
Proof: See Appendix.
Note that this upper bound is non-negative, since – by Conversely, if we use two quantizer thresholds γ1 =1 a
−
assumption on the ordering of the eigenvalues – any term in and γ2 =a 1 we obtaina ternaryoutputRV Y. Interpreting
−
the product cannot be smaller than one. any value γ1 X γ2 as an erasure, we obtain a binary
We finally show that there are cases where, despite the fact erasurechanne≤lwith≤erasureprobability a−a1.Thecorrespond-
thatwereducethedimensionalityofthedata,wearestillable ing mutual information is computed as I(Y;S) = a1 (see,
to preserve all of the relevant information: e.g., [10]), and the relevant information loss vanishes.
As this example shows, while our system actually destroys
Corollary3. AssumethatShascovariancematrixC withat
S aninfiniteamountofinformation(tobeprecise,exactly100%
mostrankM,andassumethatNiszero-meanGaussiannoise
of theavailable information[2]),the relevantinformationloss
independent of S with covariance matrix σ2 I. Let Y be
N M canstillbezero.Signalenhancement,inthesenseofremoving
obtained by employing the PCA for dimensionality reduction.
irrelevant information, was thus successful.
Then, the relevant information loss LS(X YM) vanishes. In this example it is tempting to identify the noise variable
→
Proof: The proof follows from the fact that as C = N with the irrelevant information X S. However, this not
N |
σ2 I, all eigenvalues µ = =µ =σ2 . necessarily leads to a correct result, as we show next.
N 1 N N
As mentioned before, du·e··to dimensionality reduction the To this end, we substitute the quantizerin Fig. 4 by a mag-
caobnsoseluqtueenincfeoirsmtahtaitotnheloisrrselLev(Xant→infoYrmMa)tioisnlionsfisn,iLteX; aS(dXirect ndietnusdietydfeuvnicceti,oin.e.o,fYX′ =ha|Xs e|.veBnystyhmemfaecttryth,aitttchaenpbroebsahboilwitny
YM),isinfiniteaswell.GiventheassumptionsofCor|ollary→3 thatL(X →Y′)=H(X|Y′)=1(e.g.,[12]).Onecanfurther
holds, the PCA is a good solution to our signal enhancement showthat,sincethemarginaldistributionofY′ coincideswith
problem. the conditional distributions Y S = 1 and Y S = 1, the
| − |
mutual information I(Y ;S) = 0. Thus, L (X Y ) = 1.
VI. EXAMPLES As an immediate conseq′uence, L (X SY )=→a 1′. a
We now try to illustrate a few of our results with the help Let us now determine the infoXrm|Sation→loss′ relev−aant w.r.t.
of some examples.In particular,we analyzea communication N. By showing that LN(X Y′) = LXS(X Y′) we
channelwith binaryinputandadimensionality-reducingPCA arguethatnoise and irrelevant→informa6tionar|e not→necessarily
for non-Gaussian data. Unless otherwise noted, log denotes identical. Observe that
the binary logarithm.
L (X Y )=I(X;N Y ) (39)
N ′ ′
→ |
A. Communication channel with uniform noise =H(X Y′) H(X N,Y′) (40)
| − |
Consider the digital communication system depicted in =1 H(S+N N,Y ) (41)
′
− |
Fig. 4. Let the data symbols S be uniformly distributed on
=1 H(S N, S+N ). (42)
1,1 , thus H(S) = 1. Let further N be a noise signal − | | |
{− }
which, for the sake of simplicity, is uniformly distributed on Given we know N, S is uncertain only if S+N yields the
| |
[ a,a], a > 1. Clearly, X = S +N is a continuous-valued same value for both S = 1 and S = 1. In other words, we
− −
signal with infinite entropy. During quantization an infinite have to require that N 1 = N +1. Squaring both sides
| − | | |
amount of information is lost, L(X Y) = [1]. The translates this to requiring N = N, which is fulfilled only
→ ∞ −
TABLEI
With λ =1 and Lemma 1 we thus get
INFORMATIONLOSS(RELEVANTANDIRRELEVANT)FORTHE 3
COMMUNICATIONCHANNELWITHUNIFORMNOISE.
LS(X YM) 0. (51)
Loss Y =sgn(X) Y =|X| → ≤
L(X →Y) ∞ 1 The relevant information loss vanishes. Indeed, the eigen-
LS(X →Y) H2(a2−a1)− a−a1 a1 vector corresponding to the smallest eigenvalue is given as
LX|S(X →Y) ∞ a−a1 p = 1 [1, 1,1]T; thus, for Y one would obtain
LN(X →Y) ∞ 1 3 √3 − 3
LX|N(X →Y) a2−a1 0 X +X X N +N N
Y = 1 3− 2 = 1 3− 2. (52)
3
√3 √3
for N = 0. Since Pr(N = 0) = 0 (N is a continuous RV), Since this component does not contain any signal portion,
we automatically obtain
dropping it does not lead to a loss of relevant information.
a 1 Note that in case the noise sources N do not have the
i
LN(X →Y′)=16= −a =LX|S(X →Y′). (43) same variances, the application of PCA may lead to a loss
of information, even though the relevant information is still
Indeed, the reason why we cannot identify the noise with
concentratedona subspaceoflowerdimensionality.We make
the irrelevant information can be related to N and S not
this precise the next example.
beingconditionallyindependentgivenX.Wesummarizethese
findings in Table I.
C. PCA with large noise variances
B. PCA with non-Gaussian data
We now again assume that we use three sensors to observe
Assume we observe two independent data sources, S and two independent, non-Gaussian data sources which are cor-
1
S , with three sensors which are corrupted by independent, rupted by independent Gaussian noise:
2
unit-variance Gaussian noise N , i = 1,2,3. Our sensor
i
signals shall be defined as X1 =S1+N1, (53)
X =S +N , and (54)
X =S +N , (44) 2 2 2
1 1 1
X =N (55)
X =S +S +N , and (45) 3 3
2 1 2 2
X =S +N . (46) Thistime,however,weassumethatthedatasourceshaveunit
3 2 3
variance, and that the variance of noise source N is i, thus
i
Weassumefurtherthatourdatasourceshavevariancesσ2 and
1 µ ,µ ,µ = 3,2,1 . With this, the covariance matrix of
σ2andarenon-Gaussian,butthattheystillcanbedescribedby { 1 2 3} { }
2 X is given as
a(joint)probabilitydensityfunction.Weobtainthecovariance
matrix of X=[X ,X ,X ]T as 1+1 0 0 2 0 0
1 2 3
C = 0 1+2 0 = 0 3 0 (56)
σ1+1 σ1 0 X  0 0 3   0 0 3 
C = σ σ +σ +1 σ . (47)
X 1 1 2 2
     
0 σ2 σ2+1 and has eigenvalues λ1,λ2,λ3 = 3,3,2 .
{ } { }
  Since C is already diagonal, the PCA only leads to an
Performing the eigenvalue decomposition yields three eigen- X
ordering w.r.t. the eigenvalues; droppingthe componentof Y
values,
corresponding to the smallest eigenvalue, i.e., dimensionality
{λ1,λ2,λ3}={σ1+σ2+1+C,σ1+σ2+1−C,1} reduction from N = 3 to M = 2, yields Y1 = N3 and
(48) Y = S +N . Since we do not have access to S anymore,
2 2 2 1
where C = σ12+σ22−σ1σ2. information is lost – in fact, PCA suggested to drop the most
We now reduce the dimensionof the outputvector Y from informative componentof X (the one with the highest SNR).
p
N = 3 to M = 2 by dropping the component corresponding With Lemma 1 we can now compute the Gaussian upper
to the smallest eigenvalue. Using Lemma 1 we now give an bound on the relevant information loss: Following the proof
upperboundontherelevantloss, LS(X YM).Tothisend, of Theorem 1,
note that N is iid Gaussian, and thus h→(N˜ N˜ ) = h(N˜ ).
c M c
| 1
By assumption, CN = I, and by the orthogonality of the LS(X YM) h(Y3,G) h(N1)= ln2. (57)
→ ≤ − 2
transform,
1
h(N˜ )= ln(2πe). (49) The boundobtainedfromTheorem2 is loose here,evaluating
c
2 to LS(X→YM)< 12ln3.
Conversely, we know that Since in this case the noise is not iid, the condition of
Theorem 1 is not fulfilled, thus the Gaussian upper bound
1
h(Yc|YM)≤h(Yc,G|YM,G)=h(Yc,G)= 2ln(2πeλ3). is not minimized by the PCA. Indeed, by preserving X1 and
(50) X (anddroppingX )therelevantinformationlossvanishes.
2 3
VII. CONCLUSION D. Proof of Lemma 1
Inthisworkwepresentedthenotionofrelevantinformation We start by noting that LS(X YM) = LS(Y YM)
→ →
loss and analyzed many of its elementary properties. We since Y and X are related by an invertible transform. Thus,
argued that relevant information loss is a central quantity in
theproblemofsignalenhancementandthus,ofsystemtheory LS(Y YM) = I(Y;S) I(YM;S) (70)
→ −
in general. A comparison with the literature about machine = h(Y) h(YS) h(Y )+h(Y S)
M M
− | − |
learning and neural networks revealed that many of the algo- = h(Y Y ) h(WTS+WTNS)
c M
rithms introduced there can be reformulated as solutions for | − |
+h(I WTS+I WTNS) (71)
the signal enhancement problem using the notion of relevant M M |
informationloss. Asanexample,we discussedprinciplecom- (=a)h(Y Y ) h(WTN)+h(I WTN)
c| M − M
ponentanalysisused fordimensionalityreductionandderived = h(Y Y ) h(N˜ N˜ ) (72)
c M c M
conditionsunderwhichtherelevantinformationlossvanishes. | − |
Future work will concentrate on different blocks of the where(a)isduetoindependenceofSandN.Thiscompletes
signal processingchain, such as quantizers,sampling devices, the proof.
and filters as well as on investigating a possible connection
betweenrelevantinformationlossandstatespaceaggregation. E. Proof of Theorem 1
APPENDIX By Lemma 1 and Definition 2,
A. Proof of Proposition 2 LS(X YM) = h(Yc YM) h(N˜c N˜M) (73)
→ | − |
The proof follows along the same lines as the proof of the
= h(Y Y ) J(Y Y )
c,G M,G c M
data processing inequality for mutual information (see [10]). | − |
h(N˜ N˜ )+J(N˜ N˜ ) (74)
We first expand I(X;W,V Y) as follows: − c,G| M,G c| M
| h(Y Y ) h(N˜ N˜ )(75)
c,G M,G c,G M,G
I(X;W,V Y)=I(X;V Y)+I(X;W V,Y) (58) ≤ | − |
| | | (=a)h(Y ) h(N˜ N˜ ) (76)
=I(X;W Y)+I(X;V W,Y) (59) c,G c,G M,G
− |
| |
=I(X;W Y) (60) (=b) h(Y ) h(N˜ ) (77)
c,G c,G
| −
where the last line follows from the fact that V and X are 1 N λ
i
= ln . (78)
conditionally independentgiven W. 2 µ
!
Using now Definition 1 we obtain i=YM+1
Here,(a)isduetothefactthatthePCAdecorrelatestheoutput
L (X Y)=I(X;W Y) (61)
W
→ | dataY andthusleadstoindependenceofY andY (in
=I(X;V Y)+I(X;W V,Y) (62) c,G M,G
| | thesenseofDefinition2).Bysimilarreasons(b)followsfrom
=LV(X →Y)+I(X;W|V,Y) (63) the fact that N is iid (CN is a scaled identity matrix, with all
LV(X Y) (64) eigenvaluesbeingequalµi =µ).SincethePCApreservesthe
≥ →
M coordinatescorrespondingto the M largest eigenvaluesof
which completes the proof.
C , the last line (obtained with [10, Thm. 8.4.1] and [16,
Y
B. Proof of Corollary 1 Fact 5.10.14]) represents the smallest Gaussian upper bound
If f(S) S X Y is a Markov chain, the proof follows and completes the proof.
− − −
immediately from Proposition 2. In case S f(S) X Y,
one gets with the proof of Proposition 2 − − − F. Proof of Theorem 2
We recapitulate (76) from the proof of Theorem 1 and get
L (X Y)=L (X Y)+I(X;f(S)S,Y) (65)
f(S) S
→ → |
=LS(X →Y). (66) LS(X→YM)≤h(Yc,G)−h(N˜c,G|N˜M,G) (79)
Thus,inthiscaseProposition2isshowntoholdwithequality. =h(Y ) h(N˜ )+h(N˜ ). (80)
c,G G M,G
−
Furthermore, using Definition 1, the Corollary follows im-
With [10, Thm. 8.4.1] and [16, Fact 5.10.14] we get
mediately from [15, Thm. 3.7.1].
C. Proof of Proposition 4 1 N
h(N˜ )= ln (2πe)N µ (81)
For the proof, note that with Definition 1 we get G 2 i
!
i=1
Y
(a)
LS(X Z) = I(S;X,Y Z) (67) and
→ |
(=b) I(S;X|Y,Z)+I(S;Y|Z) (68) h(Yc,G)= 1ln (2πe)N−M N λi . (82)
2
(a) !
= I(S;X Y)+I(S;Y Z) (69) i=YM+1
| |
where (a) is due to the fact that Y = g(X) and Z = h(Y), If now C denotes the (M M)-covariance matrix of N˜
N˜ M
respectively, and (b) is the chain rule of information. (and, thus, of N˜ ) and µ˜× the set of eigenvaluesof C ,
M,G i N˜
{ }
we obtain [2] ——,“RelativeinformationlossinthePCA,”inProc.IEEEInformation
LS(X→YM)≤ 12ln QNi=M+1Ni=λ1iµQiMi=1µ˜i!. (83) [3] IDTvEeh.rEesHoEior.ynTJorWaahvnnoassri.oklasnIbhn,lofe“.p:ITna(hfIoreTroXWmriya)v,,ti:voLon1alu2.tsh05ae64no,n.ryne0,o4a.Sn22ed9,pn.pe[p2u.c0r1a6sl25.,3iIn–pfT6po]6.rm6.5,a6Ft2ieo–bn5.6p26r0,o1ec0xe.stesnindge,d”
We now complete the proof by providing upper bounds [4] M. Plumbley, “Information theory and unsupervised neural networks,”
Q
on the eigenvalues in the numerator. It is easy to verify Cambridge University Engineering Department, Tech. Rep. CUED/F-
INFENG/TR.78,1991.
that C is the top left principal submatrix of WTC W
N˜ N [5] G. Deco and D. Obradovic, An Information-Theoretic Approach to
(which,bytheorthogonalityofWhasthesameeigenvaluesas NeuralComputing. NewYork,NY:Springer, 1996.
C ). As a consequence,we can employ Cauchy’sinterlacing [6] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck
N
method,” in Proc. Allerton Conf. on Communication, Control, and
inequality [16, Thm. 8.4.5]:
Computing, Sep.1999,pp.368–377.
[7] M. A. Sa´nchez-Montan˜e´s and F. J. Corbacho, “A new information
µ µ˜ µ (84)
i+N−M ≤ i ≤ i processingmeasureforadaptivecomplexsystems,”IEEETrans.Neural
The second bound, λ µ , is derived from Weyl’s inequal- Netw.,vol.15,no.4,pp.917–927,Jul.2004.
i ≤ 1 [8] M.A.Sa´nchez-Montan˜e´s,“Atheoryofinformationprocessingforadap-
ity [16, Thm. 8.4.11]
tivesystems:Inspirationfrombiology,formalanalysisandapplicationto
artificialsystems,”Ph.D.dissertation,UniversidadAuto´nomadeMadrid,
λ ν +µ (85)
i i 1 Jun.2003.
≤
[9] J. C. Principe, Information Theoretic Learning: Renyi’s Entropy and
and by noticing that ν =0 for all j >M.
j KernelPerspectives,ser.InformationScienceandStatistics. NewYork,
Combining all this, we obtain as an upper bound on the NY:Springer, 2010.
information loss [10] T.M.CoverandJ.A.Thomas,ElementsofInformationTheory,2nded.
Hoboken, NJ:WileyInterscience, 2006.
LS(X→YM)≤ 12ln QNi=M+1Ni=λ1iµQiMi=1µ˜i! (86) [11] vRo.lL.i2n1s,kenro,.“3S,elpfp-o.r1g0a5n–iz1a1ti7o,nMinara.p1e9r8c8e.ptualnetwork,”IEEEComputer,
[12] B. C. Geiger, C. Feldbauer, and G. Kubin, “Information loss in static
1ln Ni=MQ+1µ1 Mi=1µi (87) nonlinearities,”inProc.IEEEInt.Sym.WirelessCommunicationSystems
≤ 2 Q Ni=1µQi ! a(IrSWXiSvC:),1A1a0c2h.en4,7N94ov.[2c0s1.1,IpTp]..799–803,extendedversionavailable:
N [13] N. Slonim and N. Tishby, “Agglomerative information bottleneck,” in
= 1ln Qµ1 . (88) AdvancesinNeuralInformationProcessingSystems(NIPS). MITPress,
2 µ 1999,pp.617–623.
i!
i=YM+1 [14] G. Chechik and N. Tishby, “Extracting relevant structures with
This completes the proof. side information,” in Advances in Neural Information Processing
Systems (NIPS). MIT press, 2002, pp. 857–864. [Online]. Available:
REFERENCES books.nips.cc/papers/files/nips15/AA48.pdf
[15] M. S. Pinsker, Information and Information Stability of Random Vari-
[1] B. C. Geiger and G. Kubin, “On the information loss in memoryless ables andProcesses. SanFrancisco, CA:HoldenDay,1964.
systems: The multivariate case,” in Proc. Int. Zurich Seminar on
[16] D. S. Bernstein, Matrix Mathematics. Princeton, NJ: Princeton Uni-
Communications (IZS),Zurich,Feb.2012,pp.32–35,extended version
versityPress,2005.
available: arXiv:1109.4856 [cs.IT].