Table Of ContentInformation Decomposition on Structured Space
Mahito Sugiyama Hiroyuki Nakahara Koji Tsuda
ISIR, Osaka University RIKEN Brain Science Institute Graduate School of Frontier Sciences
JST PRESTO [email protected] The University of Tokyo
[email protected] [email protected]
Abstract—We build information geometry for a partially or- (a) (b)
1234 1234
dered set of variables and define the orthogonal decomposition
of information theoretic quantities. The natural connection be-
6 tween information geometry and order theory leads to efficient 123 124 134 234 123 124 234
decompositionalgorithms.ThisgeneralizationofAmari’sseminal
1
work on hierarchical decomposition of probability distributions
0
on event combinations enables us to analyze high-order statis- 12 13 14 23 24 34 23 24
2
tical interactions arising in neuroscience, biology, and machine
y learning. 1 2 3 4 1 2 4
a
M I. INTRODUCTION
ø ø
Let e ,e ,...,e denote the set of events. All combina-
1 2 n
5 tions of events are regarded as a partially ordered set and Fig.1. Hierarchyofcombinationsoffoureventse1,e2,e3,ande4.Numbers
form a complete hierarchy (Figure 1a). Amari introduced the denotecorrespondingevents.(a)Thecompletehierarchyofcombinationsof
] events.(b)Incompletehierarchybyremovinggraycombinationsin(a).
T orthogonal decomposition of probability distributions defined
I onthecompletehierarchyofevents[1].Thatmethodprovided
.
s a theoretical foundation with which to analyze the higher- II. DUALLYFLATMANIFOLDONPOSETS
c
order interactions in a wide variety of applications, such
[ Suppose that X is a discrete random variable and p(x) =
as firing patterns of neurons [2], [3], gene interactions [4],
Pr(X = x) with x ∈ S is a probability mass function on a
2 and word associations in documents [5]. However, in many
v finitesetS.Ininformationgeometry[1],[7],eachdistribution
applications the hierarchy is often incomplete, because some
3 istreatedasamappingp:S →Randthesetofallprobability
eventcombinationscanneveroccur(Figure1b).Forexample,
3 distributions is understood to be a (|S|−1)-dimensional man-
5 if e1 indicates a person being male and e2 indicates a person ifold S = {p | p(x) > 0 for all x ∈ S, (cid:80) p(x) = 1},
5 havingovariancancer,thecombinationofe ande cannever x∈S
1 2 where probabilities form a coordinate system of S, called the
0 occur. Incomplete hierarchies can also result from a lack of
. p-coordinatesystem.Informationgeometrygivesustwomore
1 data [6].
coordinate systems of S, the θ-coordinate system and the η-
0 We define information geometric dual coordinates on a
6 partially ordered set, or a poset. They lead to an efficient coordinate system, which are known to be dually orthogonal
1 and key to decomposing KL divergence via the mixed coor-
algorithm for decomposing Kullback–Leibler divergence and
: dinate system of θ and η. We introduce such two coordinates
v entropyinanincompletehierarchy.Ourmethodcanbeusedto
i in Section II-A and show decomposition of KL divergence in
X isolate the contribution of each event combination and assess
Section II-B.
its statistical significance [2]. From a theoretical viewpoint,
r We consider the case where S is a partially ordered set,
a our method offers a previously unexplored link between order
or a poset, which is one of the most fundamental structured
theory and information geometry.
space in computer science and mathematics. A partial order
The remainder of this paper is organized as follows. Sec-
“≤” satisfies the following three properties: for all x,y,z ∈
tion II introduces a dually flat manifold on a poset. In
S, (1) x ≤ x (reflexivity), (2) x ≤ y, y ≤ x ⇒ x = y
Section II-A, we show that, given a poset we introduce, the
(antisymmetry), and (3) x≤y, y ≤z ⇒x≤z (transitivity).
manifoldofprobabilitydistributionswillalwayshavethesame
Throughout the paper, we assume that S is always finite and
dually flat structure as that of the exponential family of the
includes the bottom element ⊥ ∈ S; that is, ⊥ ≤ x for all
original variable set (Equations (3) and (5)). In Section II-B,
x∈S. We write the set S\{⊥} by S+.
wepresentanefficientalgorithmtodecomposeinformationon
For a subset I ⊆ S, we denote a lower set ↓I = {x ∈
a poset (Algorithms 1, 2 and Theorem 1). As a representative
S | x ≤ s for some s ∈ I}, an upper set ↑I = {x ∈ S |
application, in Section III, we show that our algorithm can
x≥s for some s∈I}, and ↓x=↓{x}, ↑x=↑{x} for each
efficientlyisolateinformationofarbitraryorderinteractionsof
x ∈ S. In order theory, ↓x is called the principal ideal for x
events. We summarize and conclude the paper in Section IV.
and ↑x is called the principal filter for x [8], [9], which are
Apreprintisavailableathttp://arxiv.org/abs/1601.05533. known to be fundamental mathematical objects in posets.
the e-flat manifold S, which means that our formulation of
η(x) = ∑ p(s)
θ in Equation (4) is the e-affine coordinate. The m-affine
s ≥ x
coordinate η:S → R, an alternative coordinate system that
introduces the duality to S, is given as the expectation of the
parameter F (x) for each s ∈ S. In our case η is given as
s
x
follows:
(cid:88)
η(s)=E[F (x)]= p(x)=Pr(X ≥s). (5)
s
x≥s
log p(x) = ∑ θ(s) Relationships of p, θ, and η are illustrated in Figure 2.
⊥ s ≤ x The two coordinate systems θ and η are connected with
each other by the Legendre transformation. The remarkable
Fig.2. p(x),θ(x),andη(x)onposet.
property is that θ and η are dually orthogonal:
(cid:20) (cid:21)
∂ ∂
A. θ- and η-coordinate Systems E logp(x;θ) logp(x;η) =δ(s,s(cid:48)) (6)
∂θ(s) ∂η(s(cid:48))
Let us first introduce the θ-coordinate system of the mani-
foldS,whichisrealizedasamappingθ:S →R.Intheexpo- for every s,s(cid:48) ∈ S+ with the Kronecker delta δ such that
nential family, θ is known to be the natural parameter, which δ(s,s(cid:48)) = 1 if s = s(cid:48) and δ(s,s(cid:48)) = 0 otherwise [1]. This
istreatedasann-dimensionalvectorθ =(θ1,θ2,...,θn)and property is essential to construct a mixed coordinate system
of θ and η in the next subsection.
the distribution is in the form of
(cid:32) n (cid:33) Our finding connects two fundamental areas, information
p(x;θ)=exp (cid:88)θiF (x)−ψ(θ) (1) geometry and order theory, that have been independently
i
studied to date. Given the θ-coordinate, our result means that
i=1
the p-coordinate is generated from the set of principal ideals
with n functions F ,...,F and a normalizer ψ(θ) [7]. This
1 n and the η-coordinate is generated from the set of principal
is re-written as (cid:80)
filters. More specifically, let f(I,θ) = exp( θ(s)) and
(cid:32) (cid:33) s∈I
(cid:80)
(cid:88) g(I,p) = p(s) for every I ⊆ S. For each x ∈ S, we
p(x;θ)=exp θ(s)F (x)−ψ(θ) (2) s∈I
s have p(x) = f(↑x,θ) with the principal ideal ↑x for x and
s∈S+ η(x)=g(↓x,p) with the principal filter ↓x.
with n=|S+| in our setting, where there exists a one-to-one
indexing mapping ω:S+ → {1,2,...,n} such that θ(s) and B. Information Decomposition via Mixed Coordinate System
Fs correspondtoθω(s) andFω(s) inEquation(1),respectively. We introduce the mixed coordinate system of θ and η [1]
Given a poset S, we propose to define Fs(x) as on a poset, the key tool to analyze distributions on S. The
(cid:26) mixed coordinate system ξ : S+ → R with respect to a
1 if s≤x, I
Fs(x)= 0 otherwise and ψ(θ)=−logp(⊥). subset I ⊆S+ is a coordinate system of S such that
(cid:26) η(x) if x∈S+\I,
Interestingly, from Equation (2), we obtain the expansion of ξ (x):=
I θ(x) if x∈I.
logp(x) as the sum of θ(s) of lower elements s≤x in S:
(cid:88) Usingthesystem,wecanblendtwodistributionspandq:The
logp(x)= θ(s). (3)
mixeddistributionofapairofdistributions(p,q)withrespect
s≤x to I ⊆S+ is the distribution r ∈S such that
Note that this equation can be viewed as a generalization of
(cid:26) η (x)=η (x) if x∈S+\I,
the well-known log-linear model: r p
θ (x)=θ (x) if x∈I,
(cid:88) (cid:88) (cid:88) r q
logp(x)= θixi+ θijxixj + θijkxixjxk
(cid:80)
and r(⊥) = 1 − r(x), where we write θ- and η-
i i<j i<j<k s∈S+
coordinates corresponding to p by θ and η , respectively, to
+···+θ1...nx1...xn−ψ p p
clarify that p, θ , and η are the same point in S. Due to the
p p
for n-dimensional binary vector x=(x1,...,xn)∈{0,1}n. orthogonality of θ and η in Equation (6), this distribution is
Thus, given a probability distribution p ∈ S, the θ- always unique and well-defined.
coordinate system θ:S →R is recursively computed as HereweshowdecompositionoftheKullback–Leibler(KL)
(cid:88) divergence between two probability distributions p,q:
θ(x)=logp(x)− θ(s) (4)
(cid:88) p(x)
s<x D (p,q)= p(x)log (7)
KL q(x)
starting from the bottom θ(⊥)=logp(⊥). x∈S
In information geometry [1], the natural parameter θ of the
using their mixed distribution r.
exponential family is known to be the e-affine coordinate of
MS+ \ I (r) Algorithm 1: Compute the mixed distribution r of (p,q)
with respect to a singleton I ={x∗}
DKL(r, q) 1 COMPUTEMIXEDSINGLE(x∗)
r q EI (r) 2 Let x1,...,xm =x∗ be the topological ordering of ↓x∗
3 Search r(x∗) that gives θr(x∗)=θq(x∗) using
COMPUTETHETASINGLE(x∗,r(x∗)) (e.g.bisection method)
D (p, r) D (p, q)
KL KL 4 return r
= D (p, r) + D (r, q)
KL KL 5 COMPUTETHETASINGLE(x∗,r(x∗))
p 6 foreach s∈↓x∗ do // Initializing flags indicating
7 f(s)←0 // whether objects are already visited
Fig.3. Pythagorastheorem(Theorem1). 8 for i←m−1 to 1 do
9 r(xi)←COMPUTEPSINGLE(xi)
10 for i←1 to m do
Theorem 1 (Pythagoras theorem). Given two distributions 11 Compute θr(xi)
p,q ∈ S and I ⊆ S+. For a mixed distribution r of (p,q) 12 return θr(x∗)
and r(cid:48) of (q,p) with respect to I, 13 COMPUTEPSINGLE(x)
14 r(x)←p(x)
DKL(p,q)=DKL(p,r)+DKL(r,q), (8) 15 foreach s(cid:109)x do // x(cid:108)s iff x<s, x≤y<s⇒x=y
D (q,p)=D (q,r(cid:48))+D (r(cid:48),p). (9) 16 r(x)←r(x)+AGGREGATEDIFF(s,ω(x))
KL KL KL
// ω(x): index of x
Proof. We can directly use Theorem 3 in [1], which shows 17 return r(x)
that D (p,q)=D (p,r)+D (r,q) holds if m-geodesic
KL KL KL 18 AGGREGATEDIFF(x,i)
connecting p and r is orthogonal at r to the e-geodesic 19 if f(x)=i or x(cid:54)≤x∗ then
connecting r and q. Let two submanifolds EI(p) and MI(p) 20 pdiff ←0
of S be 21 else
22 f(x)←i
EI(r):={ν ∈S |θν(x)=θr(x) for all x∈I}, 23 pdiff ←p(x)−r(x)
M (r):=(cid:8)ν ∈S |η (x)=η (x) for all x∈S+\I(cid:9). 24 foreach s(cid:109)x do
S+\I ν r 25 pdiff ←pdiff+AGGREGATEDIFF(s,i)
Since EI(r) and MS+\I(r) are complementary and orthog- 26 return pdiff
onally intersect at r from Equation (6), connection of p and
r (resp. r to q) is clearly m-geodesic (resp. e-geodesic; Fig-
ure3).ThereforeDKL(p,q)=DKL(p,r)+DKL(r,q)follows. Algorithm 2: Compute the mixed distribution r of (p,q)
The second equation DKL(q,p) = DKL(q,r(cid:48))+DKL(r(cid:48),p) with respect to I ⊆S+
can be proved in the same way.
1 COMPUTEMIXEDMULTI(I)
Moreover, for a hierarchical collection {I ,I ,...,I } of 2 repeat
subsets of S such that ∅=I0 ⊆I1 ⊆···⊆I0k =1 S+, k 34 for xC∗O∈MPIUdToEMIXEDSINGLE(x∗) // Algorithm 1
(cid:88)k 5 until convergence of r;
D (p,q)= D (r ,r ), (10)
KL KL i−1 i
i=1
where r is the mixed distribution of (p,q) with respect to I
i i therefore obtain the information gain for every subset I ∈
for each i∈{0,1,...,k}. Note that r =p and r =q.
0 k S as the KL divergence D (p,p ), where p is the mixed
KL I I
Let p be the uniform distribution such that p (x)=1/|S|
0 0 distribution of (p,p ) with respect to I.
0
for all x∈S, which is the origin of the θ-coordinate because
θ (x)=0 for all x∈S+. Since for the entropy H(X) with C. Computation of Mixed Distributions
p0 Here, we show how to compute the mixed distribution r
a probability distribution p
from p and q with a subset I ⊆ S+1. First we present an
(cid:88)
H(X)=− p(x)logp(x)=log|S|−D (p,p ) algorithmtocomputerinasimplecase,whereI isasingleton
KL 0
x∈S and we let I ={x∗}. Since ηr(x)=ηp(x) for all x(cid:54)=x∗, it
is clear that r(x)=p(x) for any x(cid:54)≤x∗. Therefore, we have
holds and log|S| is a constant, entropy decomposition is
focused on computing only r(x) with x≤x∗.
achieved by our KL divergence decomposition in Theorem 1:
Assume r(x∗) is fixed and let I = {s ∈ ↓x∗ | s ≥ x}
≥x
H(X)=log|S|−(D (p,r)+D (r,p )), andI =I \{x}.Foreachx∈↓x∗ withx(cid:54)=x∗,wehave
KL KL 0 >x ≥x
(cid:80) (cid:80)
p(s) = r(s) from η (x) = η (x). Hence
where r is the mixed distribution of (p,p ) with respect to s∈I≥x s∈I≥x p r
0
I ⊆ S+. The entropy is decomposed into the contribution
1An implementation is available at: https://github.com/mahito-sugiyama/
DKL(p,r) of elements in I and the other DKL(r,p0). We can information-decomposition
(cid:80)
r(x) is obtained as r(x) = p(x) + (p(s)−r(s)).
Thus if ↓x∗ is topologically sorted assx∈0I,>xx1,...,xm with SAMPLETSAINBLEEXAIMPLE1.
x =⊥andx =x∗,wecancomputer(x ),r(x ),...,
0 m m m−1 x3 = {1,2,4,5}
r(x0) one after another. The function COMPUTEPSINGLE(x) Events
in Algorithm 1 performs for this computation. Since θ (x ),
r 0 t e
θ (x ), ..., θ (x )=θ (x∗) can be computed after comput- 1 2
r 1 r m r t e
ingallr(x)underfixedr(x∗),θ (x∗)isnumericallycomputed 2 2
r t e , e x1 = {2} x2 = {4,5}
asafunctionofr(x∗).Thisprocessissummarizedinthefunc- 3 4 5
t e , e , e , e
tion COMPUTETHETASINGLE(x∗,r(x∗)) in Algorithm 1. As t4 e1, e2, e4, e5
5 1 2 4 5
the function is continuous, we can use a numerical optimiza-
t6 e3 ⊥ = ø
tionmethod,suchasthebisectionmethod,toefficientlysearch
t e , e , e , e
r(x∗)givingthesolutionθ (x∗)=θ (x∗).Thetimecomplex- 7 1 2 4 5
r q t e , e Fig. 4. Poset generated from samples in
ityofcomputingr isO(h(x∗)|↓x∗|2)≤O(h(x∗)|S|2),where 8 4 5 TableI.
t e , e , e , e
h(x∗)isthenumberofiterationsforsolvingθ (x∗)=θ (x∗). 9 1 2 4 5
r q t e
We next consider the general case. Let I = {x∗,...,x∗}. 10 2
1 l
Althoughitisagaindifficulttoanalyticallycomputethemixed
distribution r, we can numerically compute the distribution r
can be computed by Algorithms 1 and 2. Therefore, the p-
by iterating computation of θ (x∗) for each x∗ while fixing
r i i value can be obtained from data samples since λ is known to
θ(x∗)withj (cid:54)=i,whichisinspiredbyalternatingoptimization
j followtheχ2-distributionwiththedegreesoffreedom|S|−1.
over I mainly used in the field of convex optimization. The
overall process is shown in Algorithm 2. III. ORTHOGONALDECOMPOSITIONOFINTERACTIONS
Lemma 1. Algorithm 2 always converges to the mixed distri- As a representative application, let us consider the problem
bution r ∈S of (p,q) with respect to I ⊆S+. of orthogonal decomposition of event combinations. Suppose
there are n events e ,...,e as discussed in the Introduction.
Proof. Let r ,r ,... be a sequence of distributions in which 1 n
1 2 Foreachsubsetx⊆[n]={1,...,n},letp(x)betheprobabil-
each ri is obtained by the ith run of the function COMPUTE- ityofthecombination(cid:84) e .Theobjectiveistodecompose
MIXEDSINGLE in Algorithm 2. From Theorem 1, we have i∈x i
logp(x) to the sum of coefficients of its subsets s⊆x, which
DKL(ri,r) = DKL(ri,ri+1)+DKL(ri+1,r) for all i, hence correspond to the θ-coordinates: logp(x) = (cid:80) θ(s). The
D (r ,r) ≥ D (r ,r) with the equality holding only if s≤x
KL i KL i+1 order≤isgivenaccordingtotheinclusionrelationship:x≤s
D (r ,r ) = 0. Since there always exists r with j > i
KL i i+1 j if x⊆s. The coefficients θ(s) show the “pure” contributions
such that DKL(ri,rj) > 0 if DKL(ri,r) > (cid:15) for any (cid:15) > 0, of respective interactions (cid:84) e as they are independent of
Algorithm 2 converges to the mixed distribution r. j∈s j (cid:80)
theirfrequencies;thatis,theη-coordinates:η(s)= p(x)
x≥s
Sincethetimecomplexityofcomputingr(x∗)foreachx∗ ∈I AssumethatN samplest1,t2,...,tN aregiven,whereeach
isO(h(x∗)|↓x∗|2),theoveralltimecomplexityofcomputingr sampleti isasetofevents,whichmeansthattheeventsoccur
is O(h(cid:80) h(x∗)|↓x∗|2)≤O(h|S|3(cid:80) h(x∗)), where simultaneously. We estimate each probability p(x) through
x∗∈I x∗∈I
h is the number of iterations until convergence of r. its natural estimator pˆ(x) = |{i ∈ [N] | ti = x}|/N. To
effectively estimate pˆand efficiently compute θ and η from
pˆ pˆ
D. Measuring Statistical Significance of θ
samples, we prune the whole event combinations P([n]) by
Given a distribution p on S, we can assess the statistical excluding combinations that do not frequently appear in the
significance of θp through a likelihood-ratio test, in particular dataset. Given a threshold σ ∈R such that 0≤σ ≤1, we set
a G-test, using decomposition of the KL divergence. Each S+ = {x ⊆ [n] | pˆ(x) ≥ σ} and pˆ(⊥) = 1−(cid:80) pˆ(x).
x∈S+
θp(x) shows a contribution of x on p as it is the coefficient Thus the dimensionality of the manifold S reduces from 2n
of the log expansion of p and is orthogonal to the marginals to at most N. Since any subset of P([n]) is a poset, we can
ηp. applyourdecompositiontechniquepresentedinSectionIIvia
The null and the alternative hypotheses are [2], [4]: computation of θ , η , and mixed distributions. Interestingly,
pˆ pˆ
a sample t can be viewed as a transaction of a database and
H : θ (x)=0,∀x∈I, H : θ (x)(cid:54)=0,∀x∈I, i
0 p 1 p η (I) corresponds to the support of I used in the context of
pˆ
whichmeansthatweknockdownallelementsx∈I byletting frequent pattern (itemset) mining [10].
θ (x) = 0 in the generalized log-linear model logp(x) =
p Example 1. Given samples in Table I, assume that our
(cid:80)
θ (s) in Equation (3). The statistics λ is then given as
s≤x p thresholdσ =0.2.WethenobtainaposetS ={⊥,x ,x ,x }
1 2 3
(cid:88)(cid:18) (cid:18)p(s)(cid:19)(cid:19) with ⊥ = ∅, x1 = {2}, x2 = {4,5}, and x3 = {1,2,4,5},
λ=2N p(s)log =2ND (p,r),
r(s) KL as shown in Figure 4, where pˆ(⊥) = 0.1, pˆ(x1) = 0.3,
s∈S pˆ(x ) = 0.2, and pˆ(x ) = 0.4. Thus, θ are obtained as fol-
2 3 pˆ
where N is the sample size and r is the null distribution, the lows: θ (⊥)=−2.303, θ (x )=1.099, θ (x )=0.693, and
pˆ pˆ 1 pˆ 2
mixed distribution of (p,p ) with respect to I, and hence λ θ (x )=−0.405. Let rˆ be the mixed distribution of (pˆ,p )
0 pˆ 3 x 0
such as single nucleotide polymorphisms (SNPs) in genome-
wide association studies (GWAS) [15] and neural data in
3
neuroscience [16]. To our knowledge, this is the first method
that avoids the curse of dimensionality in orthogonal decom-
2 positionofinteractionsandachievesefficientcomputationand
effective probability estimation from data.
1 Our work promises many interesting future studies, both
in theoretical and practical directions. There will be a more
⊥ interesting theoretical connection between information geom-
0
etry and order theory. Furthermore, it is exciting to apply our
0 1 2 3
decomposition method to real-world scientific datasets such
Fig.5. PosetofnonnegativeintegervectorsinExample2. as firing patterns of neurons and SNPs in GWAS to reveal
unknown associations.
with {x} ∈ S. From these parameters, the KL divergence ACKNOWLEDGMENT
for each interaction is obtained as DKL(pˆ,rˆx1) = 0.0523, This work was supported by JSPS KAKENHI Grant Num-
DKL(pˆ,rˆx2)=0.0170, and DKL(pˆ,rˆx3)=0.0040. Although ber 26880013 (MS) and 26120732 (HN). The research of
p-values of those interactions are larger than 0.99, they are K.T. was supported by JST CREST, JST ERATO, RIKEN
due to small sample size N = 10. If N = 300, for example, PostK,NIMSMI2I,KAKENHINanostructureandKAKENHI
the p-value of x1 becomes 0.001 and it is significant under 15H05711.
the significance level α=0.05.
REFERENCES
The same strategy can be applied to a poset S composed
of vectors of n-dimensional nonnegative integers Zn . We [1] S. Amari, “Information geometry on hierarchy of probability distribu-
≥0 tions,” IEEE Transactions on Information Theory, vol. 47, no. 5, pp.
assume S to be a subset of Zn , where for each pair of 1701–1711,2001.
≥0
vectors x,y ∈ Zn with x = (x1,x2,...,xn) and y = [2] H.NakaharaandS.Amari,“Information-geometricmeasureforneural
≥0 spikes,”NeuralComputation,vol.14,no.10,pp.2269–2316,2002.
(y1,y2,...,yn),wedefinethepartialorder≤asx≤y ifand
[3] H.Nakahara,S.Amari,andB.J.Richmond,“Acomparisonofdescrip-
only if xi ≤ yi for all i ∈ [n], and 0 = (0,0,...,0) ∈ Zn tivemodelsofasinglespiketrainbyinformation-geometricmeasure,”
≥0
corresponds to ⊥. Any subset S ⊂Zn becomes a poset. Neuralcomputation,vol.18,no.3,pp.545–568,2006.
≥0 [4] H. Nakahara, S. Nishimura, M. Inoue, G. Hori, and S. Amari, “Gene
Given N data points x ,x ,...,x of n-dimensional
1 2 N interaction in DNA microarray data is decomposed by information
nonnegative integers. Similar to the previous case, a poset S+ geometric measure,” Bioinformatics, vol. 19, no. 9, pp. 1124–1131,
isobtainedfromdataasS+ ={x∈Zn |pˆ(x)≥σ}usinga 2003.
≥0 [5] Y. Hou, X. Zhao, D. Song, and W. Li, “Mining pure high-order word
thresholdσ ∈R.Wecanapplyourinformationdecomposition
associations via information geometry for information retrieval,” ACM
to S with an empirical probability distribution pˆ. Transactions on Information Systems, vol. 31, no. 3, pp. 12:1–12:32,
2013.
Example 2. Given data points x1,x2,...,x25 ∈ Z2≥0 as [6] E.Ganmor,R.Segev,andE.Schneidman,“Sparselow-orderinteraction
x ,...,x = (0,1), x = (1,0), x ,...,x = (1,1), network underlies a highly correlated and learnable neural population
1 3 4 5 8 code,” Proceedings of the National Academy of Sciences, vol. 108,
x ,...,x = (1,2), x ,...,x = (2,1), x ,...,x =
9 11 12 21 22 25 no.23,pp.9679–9684,2011.
(3,3). We have S ={(0,0),(0,1),(1,1),(1,2),(2,1),(3,3)} [7] S.AmariandH.Nagaoka,Methodsofinformationgeometry. American
if σ =2/25, which is shown in Figure 5. MathematicalSociety,2007.
[8] B. A. Davey and H. A. Priestley, Introduction to lattices and order,
2nded. CambridgeUniversityPress,2002.
IV. CONCLUSION [9] G. Gierz, K. H. Hofmann, K. Keimel, J. D. Lawson, M. Mislove, and
D.S.Scott,ContinuousLatticesandDomains. CambridgeUniversity
In this paper, we have theoretically shown the intriguing
Press,2003.
relationship between two key structures in information geom- [10] C.C.AggarwalandJ.Han,Eds.,FrequentPatternMining. Springer,
etry and order theory: the dually flat structure of a manifold 2014.
[11] N.Bertschinger,J.Rauh,E.Olbrich,andJ.Jost,“Sharedinformation—
of the exponential family and the partial order structure of
new insights and problems in decomposing information in complex
events.Wehaveproposedanefficientalgorithmofinformation systems,” in Proceedings of the European Conference on Complex
decomposition that is applicable to any kind of posets; this is Systems2012. Springer,2013,pp.251–269.
[12] N. Bertschinger, J. Rauh, E. Olbrich, J. Jost, and N. Ay, “Quantifying
in contrast to a number of other studies [11]–[14]. uniqueinformation,”Entropy,vol.16,no.4,pp.2161–2183,2014.
As a representative application, we have demonstrated or- [13] E. Olbrich, N. Bertschinger, and J. Rauh, “Information decomposition
thogonal decomposition of interactions of events. We have andsynergy,”Entropy,vol.17,no.5,pp.3501–3517,2015.
[14] P. L. Williams and R. D. Beer, “Nonnegative decomposition of multi-
shown that the partial order structure can be directly obtained variateinformation,”arXiv:1004.2515,2010.
from data in an efficient manner, and the dimensionality of [15] TheWellcomeTrustCaseControlConsortium,“Genome-wideassocia-
the manifold is reduced from 2n for n variables in previous tionstudyof14,000casesofsevencommondiseasesand3,000shared
controls,”Nature,vol.447,no.7145,pp.661–678,2007.
approaches to, at most, the sample size N. Thus, we can
[16] A. Alivisatos, M. Chun, G. Church, R. Greenspan, M. Roukes, and
performorthogonaldecompositionforrecentlyemerginghigh- R.Yuste,“Thebrainactivitymapprojectandthechallengeoffunctional
dimensionaldatawiththousandsorevenmillionsofvariables, connectomics,”Neuron,vol.74,pp.970–974,2012.
