Table Of ContentCharacteristic-Dependent Linear Rank Inequalities with
Applications to Network Coding ∗
4
1 Randall Dougherty, Eric Freiling, and Kenneth Zeger
0
2
n
a IEEE Transactions on Information Theory
J
Submitted: November 19, 2013
1
1
]
T
I
. Abstract
s
c
[ Twocharacteristic-dependent linearrankinequalitiesaregivenforeightvariables. Specifi-
cally,thefirstinequalityholdsforallfinitefieldswhosecharacteristic isnotthreeanddoesnot
1
v ingeneralholdovercharacteristic three. Thesecondinequalityholdsforallfinitefieldswhose
7 characteristic is three and does not in general hold over characteristics other than three. Ap-
0
plicationsoftheseinequalitiestothecomputationofcapacityupperboundsinnetworkcoding
5
2 aredemonstrated.
.
1
0
4
1
:
v
i
X
r
a
∗ThisworkwassupportedbytheInstituteforDefenseAnalysesandtheNationalScienceFoundation.
R.DoughertyiswiththeCenterforCommunicationsResearch,4320WesterraCourt,SanDiego,CA92121-1969
([email protected]).
E.FreilingandK.ZegerarewiththeDepartmentofElectricalandComputerEngineering,UniversityofCalifor-
nia,SanDiego,LaJolla,CA92093-0407([email protected],[email protected]).
Dougherty-Freiling-Zeger November19,2013
1 Introduction
The studyof informationinequalitiesis asubfield of informationtheory that describes linearcon-
straintsontheentropiesoffinitecollectionsofjointlydistributeddiscreterandomvariables. Histor-
ically,theknowninformationinequalitieswereorignallyallspecialcasesofShannon’sconditional
mutualinformationinequalityI(X;Y|Z) ≥ 0,butlaterweregeneralizedtoothertypesofinequal-
ities, called non-Shannon inequalities. Information inequalities have been shown to be useful for
computingupperboundsonthenetworkcodingcapacities ofcertain networks.
Analagously, the study of linear rank inequalities is a topic of linear algebra, which describes
linearconstraintsonthedimensionsofcollectionsofsubspacesoffinitedimensionalvectorspaces.
Infact, theset ofallinformationinequalitiescanbeviewedassubclassofthesetofalllinearrank
inequalities.
Information inequalities hold over all collections of a certain number of random variables. In
constrast, linear rank inequalities may hold over only certain vector spaces, such as those whose
scalars haveparticularfield characteristics.
Inthispaper,wepresenttwonewlinearrankinequalitiesoverfinitefields,whicharenotinfor-
mationinequalities,andwiththepeculiarpropertythattheyonlyholdforcertainfields,depending
on the associated vector space. The first inequality is shown to hold over all vector spaces when
thefieldcharacteristicisanythingbutthree(Theorem3.1),butdoesnotalwaysholdwhenthefield
characteristic is three (Theorem 3.2). In contrast, the second inequality is shown to hold over all
vector spaces when the field characteristic is three (Theorem 4.1), but does not always hold when
thefieldcharacteristicisnotthree(Theorem4.2). Wealsoshowhowtheseinequalitiescanbeused
toobtainboundson thecapacities ofcertain networks(Corollaries 3.4and 4.3).
It will be assumed that the reader has familiarity with linear algebra, finite fields, information
theory, and network coding. Nevertheless, we will give some brief tutorial descriptions of these
topicsforcompleteness.
1.1 Background
In 2000, Ahlswede, Cai, Li, and Yeung introduced the field of Network Coding [1] and showed
that coding can outperform routing in directed acyclic networks.1 There are presently no known
algorithms to determine the capacity or the linear capacity of a given network. In fact, it is not
evenknownifsuch algorithmsexist.
Informationinequalitiesarelinearinequalitiesthatholdforalljointlydistributedrandomvari-
ables, and Shannon inequalities are information inequalities of a certain form [18]. Both are de-
fined in Section 1.3. It is known [21] that all information inequalities containing three or fewer
variables are Shannon inequalities. The first “non-Shannon” information inequality was of four
variables and was published in 1998 by Zhang and Yeung [24]. Since 1998, various other non-
Shannon inequalities have been found, for example, by Lneˇnicˇka [13], Makarychev, Makarychev,
Romashchenko, and Vereshchagin [14], Zhang [22], Zhang and Yeung [23], Dougherty, Freiling,
andZeger[5],andMatu´sˇ [15]. Additionally,in2007,Matu´sˇdemonstratedaninfinitecollectionof
1Inwhatfollows,by“network”weshallalwaysmeanadirectedacyclicnetwork.
Page1 of47
Dougherty-Freiling-Zeger November19,2013
independentnon-Shannoninformationinequalities[15]andtherewerenecessarilyaninfinitenum-
ber of such inequalities. In 2008, Xu, Wang, and Sun [19] also gave an infinite list of inequalities
butdidnot establishtheirnecessity.
There is a close connection between information inequalities and network coding [4]. Ca-
pacities of some networks have been computed by finding matching lower and upper bounds [6].
Lower bounds have been found by deriving coding solutions. Upper bounds have been found by
using information inequalities and treating the sources as independent random variables that are
uniformlydistributedoverthealphabet. One“holygrail”problemofnetworkcodingistodevelop
an algorithm to compute the coding capacity of an arbitrary network. If such an algorithm exists,
informationinequalitiesmaypotentiallyplay aroleinthesolution.
Ithasbeenshownthatlinearcodesareinsufficientfornetworkcodingingeneral[7]. However,
linear codes may be desirable to use in practice due to ease of analysis and implementation. It
has been shown that the coding capacity is independent of the alphabet size [3]. However, the
linear coding capacity is dependent on alphabet size, or more specifically the field characteristic.
Inotherwords,onecanpotentiallyachieveahigherrateoflinearcommunicationbychoosingone
characteristicoveranother. Toprovideupperboundsforthelinearcodingcapacityforaparticular
field one can look at linear rank inequalities [10]. Linear rank inequalities are linear inequalities
that are always satisfied by ranks2 ofsubspaces of a vectorspace. All informationinequalitiesare
linear rank inequalities but not all linear rank inequalities are information inequalities. The first
exampleofalinearrankinequalitythatisnotaninformationinequalitywasfoundbyIngleton[12].
Information inequalities can provide an upper bound for the capacity of a network, but this upper
bound would hold for all alphabets. Therefore, to determine the linear coding capacity over a
certain characteristiconewouldhaveto considerlinearrank inequalities.
All linear rank inequalities up to and including five variables are known and none of these
depend on thevectorspaces’ field characteristics [8]. Theset ofall linearrank inequalitiesfor six
variableshasnotyetbeen determined. Characteristic-dependentlinearrank inequalitiesare given,
forexample,in [2]and [10].
An inequality is given in [10] which is valid for characteristic two and another inequality is
given which is valid for every characteristic except for two. These inequalities are then used to
provideupperboundsforthelinearcodingcapacity oftwonetworks.
In the present paper, we give two characteristic-dependent linear rank inequalities on eight
variables. Oneis validforcharacteristic threeand theotheris validfor everycharacteristicexcept
for three. These inequalities are then used to provide upper bounds for the linear coding capacity
oftwonetworks.
It is our intention that the techniques presented here may prove useful or otherwise motivate
furtherprogressin determiningnetwork capacities.
2Throughoutthispaper,wewillusetheterminology“rank”ofasubspacetomeanthedimensionofthesubspace
(i.e. therankofamatrixwhosecolumnsareabasisforthesubspace),inordertoparalleltheterminologyofmatroid
theory.
Page2 of47
Dougherty-Freiling-Zeger November19,2013
1.2 Matroids
In this section a very brief review of matroids is given which will enable discussion in subse-
quent sections of a matroid-based method for constructing a particular network that helps in the
derivationofthelinearrank inequalitiespresented in thispaper.
A matroid is an abstract structure that captures a notion of “independence” that is found in
finite dimensional vector spaces, graphs, and various other mathematical topics. We will follow
thenotationand resultsof[17].
Definition1.1. Amatroid,M, isapair(E,I),whereE isafinitesetandI isasetofsubsetsofE
thatsatisfies thefollowingproperties:
(I1) ∅ ∈ I.
(I2) ∀A,B ⊆ E, ifA ⊆ B ∈ I,then A ∈ I.
(I3) ∀A,B ⊆ E, ifA,B ∈ I and |A| > |B|,then∃u ∈ A\B such thatB ∪{u} ∈ I.
ThesetsinI arecalledindependentsets. IfasubsetofE isnotinI,thenitiscalleddependent.
Anexampleofamatroidisobtainedfromlinearalgebra. LetF beafinitefieldandletV(m,F)
be the vector space of all m-dimensional vectors whose components are elements of F. Suppose
A is an m × n matrix over F. Let E = {1,...,n} and I be the set of all X ⊆ E such that the
multisetofcolumnsofAindexedbytheelementsofX islinearlyindependentin thevectorspace
V(m,F). Then M = (E,I)isamatroidcalled thevector matroid ofA.
AmatroidissaidtoberepresentableoverthefieldF ifitisisomorphictosomevectormatroid
overV(m,F).
Forexample,ifF is thebinary field and
a b c d e
1 0 0 1 1
A =
(cid:18)0 1 0 0 1(cid:19)
where a,b,c,d,e denote the columns of A from left to right, then M = (E,I) is a vector matroid
ofA, whereE = {a,b,c,d,e}and
I = {∅,{a},{b},{d},{e},{a,b},{a,e},{b,d},{b,e},{d,e}}.
A baseisamaximalindependentset. LetB(M) denotethesetofall basesofamatroidM. In
ourexample,
B(M) = {{a,b},{a,e},{b,d},{b,e},{d,e}}.
It iswell knownthat allthebasesofamatroidare ofthesamecardinality.
If we let X ⊆ E and I|X = {i ⊆ X : i ∈ I}, then it is easy to see that (X,I|X) is a
matroid. The rank of X, denoted by r(X), is defined to be the cardinality of a base in M|X. In
our example, r(M) = 2. A circuit is a minimal dependent set. The circuits in our example are
{{c},{a,d},{a,b,e},{b,d,e}}.
Page3 of47
Dougherty-Freiling-Zeger November19,2013
1.3 Information Theory and Linear rank Inequalities
In this section we will use the information theoretic concepts of entropy and mutual information
to define and use the linear algebraic concept of linear rank inequalities. Connections between
informationinequalitiesandlinearrank inequalitiesis alsodiscussed.
Let A,B,C be collections of discrete random variables over a finite alphabet X, and let p be
theprobabilitymassfunctionofA. TheentropyofAis defined by
H(A) = − p(u)log p(u).
|X|
Xu
TheconditionalentropyofAgivenB is
H(A|B) = H(A,B)−H(B), (1)
themutualinformationbetween AandB is
I(A;B) = H(A)−H(A|B) = H(A)+H(B)−H(A,B), (2)
and theconditionalmutualinformationbetween Aand B givenC is
I(A;B|C) = H(A|C)−H(A|B,C) = H(A,C)+H(B,C)−H(C)−H(A,B,C). (3)
Wewillmakeuseofthefollowingbasicinformation-theoreticfacts [21]:
0 = H(∅) (4)
0 ≤ H(A) = H(A|∅) (5)
0 ≤ H(A|B) (6)
0 ≤ I(A;B) (7)
H(A,B|C) ≤ H(A|C)+H(B|C) (8)
H(A|B,C) ≤ H(A|B) ≤ H(A,C|B) (9)
I(A;B,C) = I(A;B|C)+I(A;C). (10)
Theequations(5)-(9)wereoriginallygivenbyShannonin1948[18],andcanallbeobtainedfrom
thesingleinequalityI(A;B|C) ≥ 0.
Definition 1.2. Let q beapositiveinteger,and letS ,...,S besubsetsof{1,...,q}. Let α ∈ R
1 k i
for1 ≤ i ≤ k. Alinearinequalityoftheform
α H({A : i ∈ S })+···+α H({A : i ∈ S }) ≥ 0 (11)
1 i 1 k i k
iscalledaninformationinequalityifitholdsforalljointlydistributedrandomvariablesA ,...,A .
1 q
As an example, taking q = 2, S = {1}, S = {2}, S = ∅, S = {1,2}, α = α = 1,
1 2 3 4 1 2
α = −1,andusing(8)showsthatH(A )+H(A )−H(A ,A ) ≥ 0isaninformationinequality.
4 1 2 1 2
Page4 of47
Dougherty-Freiling-Zeger November19,2013
AShannoninformationinequalityisanyinformationinequalitythatcanbeexpressedasafinite
sumoftheform
α I(A ;B |C ) ≥ 0
i i i i
Xi
where each α is a nonnegativereal number. Any information inequality that cannot be expressed
i
intheform abovewillbecalled anon-Shannoninformationinequality.
Linear rank inequalities are closely related to information inequalities. In fact, in order to
describe linear rank inequalities we will borrow notation from information theory to use in the
contextoflinearalgebrain thefollowingmanner.
Suppose A and B are subspaces of a given vector space V, and let hA,Bi denote the span of
A∪B. We will let H(A) denote the rank of A, and let H(A,B) denote the rank of hA,Bi. The
meaningsofsomeotherinformationtheoreticnotationinthecontextoflinearalgebrathenfollows
from (1)-(3). Specifically, note that the conditional entropy notation H(A|B) denotes the excess
rank of subspaceA overthat ofsubspaceA∩B, orequivalently,thecodimensionofA∩B in A;
and themutualinformationnotationI(A;B)denotes therank ofA∩B.
A linearrankinequalityovera vectorspace V isa linearinequalityoftheform in (11), that is
satisfiedby everyassignmentofsubspaces ofV tothevariablesA ,...,A .
1 q
All informationinequalitiesare linearrank inequalitiesoverallfinitevectorspaces, butnotall
linear rank inequalities are information inequalities. For background material on these concepts,
thereaderis referred toHammer, Romashchenko,Shen, and Vereshchagin[11].
The first known example of a linear rank inequality over all finite vector spaces that is not an
informationinequalityistheIngletoninequality[12]:
I(A;B) ≤ I(A;B|C)+I(A;B|D)+I(C;D).
ToseethattheIngletoninequalityisnotaninformationinequality,letA,B,C,Dbebinaryrandom
variables,and letX = (A,B,C,D)withprobabilities:
P(X = 0000) = 1/4
P(X = 1111) = 1/4
P(X = 0101) = 1/4
P(X = 0110) = 1/4.
Then theIngletoninequalityfailssince:
I(A;B) −I(A;B|C)−I(A;B|D)−I(C;D) > 0.
(5−log227)/2 0 0 0
| {z } | {z } | {z } | {z }
1.4 Network Coding
In this section, we will briefly review some concepts of network coding. This will enable the dis-
cussionlaterinthispaperofourconstructionoflinearrankinequalitiesusingnetworksconstructed
from twoparticularmatroids(T8 and non-T8). Formoredetailson networkcoding,see[20].
Page5 of47
Dougherty-Freiling-Zeger November19,2013
A network is a finite, directed, acyclic multigraphwithmessages and demands. Network mes-
sages are arbitrary vectors of k symbols over a finite alphabet A. Each network edge carries a
vector of n symbols from A. Each message originates at a particular node called the source node
for that message and is required by one or more demand nodes. When we draw a network, a
message variable appearing above a node indicates the message is generated by such node3, and
a message variable appearing below a node indicates the message is demanded by such node, For
a given network, the values of k and n can be chosen in order to implement certain codes and to
obtaincertain throughputk/n.
The inputsto a network nodearethevectors carried on itsin-edges as well as themessages,if
any, generated at thenode. The outputsof a network nodeare thepackets carried on its out-edges
as well as any demanded messages at the node. Each output of a node must be a function only
of its inputs. A coding solution for the network is an assignment of such functions to the network
edges. When the values of k and n need to be emphasized, the coding solution will be called a
(k,n)-codingsolution. Thecapacityofanetwork isdefined as:
C = sup{k/n : ∃a(k,n)-codingsolution}.
A solution is called a linear solution, if the alphabet A is a finite field and the edge functions
are linear (i.e. linear combinations of their input vectors where the coefficients are matrices over
thefield).
The linear capacity is defined the same as the capacity but restricting solutionsto be linear. It
isalsoeasilyverifiedthatifxisamessage,thenH(x) = k,andifxisavectorcarriedbyanedge,
thenH(x) ≤ n.
Let us illustrate a method for finding capacity bounds by examining the well-known Butterfly
network, depicted in Figure 1. We assume the network messages x and y are independent, k-
dimensional,randomvectorswithuniformlydistributedcomponents. Theninanysolutionitmust
bethecasethat
H(y|x,z) = 0 (12)
sincey isafunctionofxand z, andalso that
2k = H(x)+H(y) (13)
= H(x,y) [from indep. ofx andy]
≤ H(x,y,z) [from (9)]
= H(x,z)+H(y|x,z) [from (1)]
= H(x,z) [from (12)]
≤ H(x)+H(z) [from (8)]
≤ k +n. (14)
3 We note thatin Figures2 and 3, forconvenience,we labelsourcemessagesabovenodeslyingin boththe top
andbottomlayersineachdiagram.Thisismeanttoindicatethatthereis,infact,aseparate(buthidden)distinctnode
foreachsuchsourcemessage,whoseout-edgesgodirectlytothenodeslabeledbythesourcemessageinthetopand
bottemlayers.
Page6 of47
Dougherty-Freiling-Zeger November19,2013
y
x
n n
1 2
n
3
z
n
4
n
n
5
6
y x
Figure 1: TheButterfly network withsource messages x and y, generated by source nodes n and
1
n ,respectively. Demand nodesn andn demandmessagesy and x, respectively.
2 5 6
Page7 of47
Dougherty-Freiling-Zeger November19,2013
Thisimplies2k ≤ k+n, orequivalentlyk/n ≤ 1. Sincethisboundholdsforall choicesofk and
n,thecodingcapacitymustbeatmost1. Ontheotherhand,asolutionwithk = n = 1isobtained
by taking z = x + y over any finite field alphabet, so the coding capacity is at least 1. Thus the
codingcapacityfortheButterflynetworkisthesameasthelinearcodingcapacitywhichisexactly
equal to1.
The inequalities in (14) were based on random variables x,y,z. Later, in the proofs of Corol-
laries 3.4 and 4.3, we will obtain bounds on the capacities of networks by using linear rank in-
equalities,insteadofinformationinequalities. Inthosecases,certainvectorsubspaceswillbeused
insteadofrandomvariables,but theprocedurewillappear similar.
Page8 of47
Dougherty-Freiling-Zeger November19,2013
2 Preliminaries
In thissection, we givensometechnical lemmas which will be useful forproving themain results
ofthepaper.
If A is a subspace of vector space V, and A is a subspace of A, then we will use the notation
codim (A) = dim(A)−dim(A)torepresentthecodimensionofAinA. Wewillomitthesubscript
A
when itisobviousfrom thecontextwhich spacethecodimensioniswithrespect to.
Lemma 2.1. [10] Let V be a finite dimensional vector space with subspaces A and B. Then the
subspaceA∩B hascodimensionatmostcodim(A)+codim(B) in V.
Proof. WeknowH(A)+H(B)−I(A;B) = H(A,B) ≤ H(V). ThenaddingH(V)tobothsides
oftheinequalitygivesH(V)−I(A;B) ≤ H(V)−H(A)+H(V)−H(B). Thus,codim(A∩B) ≤
codim(A)+codim(B). (cid:4)
Lemma2.2. [10]LetAandBbevectorspacesoverthesamefinitescalarfieldandwithsubspaces
A and B, respectively. Let f : A → B be a linear function such that f(A\A) ⊆ B\B. Then the
codimensionofAinAis atmostthecodimensionof B inB.
Proof. Suppose a base for A consists of a base for A together with the vectors a ,...,a . Let
1 n
γ ,...,γ befieldelementswhicharenotallzero. Thenγ a +···+γ a ∈ A\A,soγ f(a )+
1 n 1 1 n n 1 1
···+γ f(a ) = f(γ a +···+γ a ) ∈ B \B. Thus, the vectors f(a ),...,f(a ) are linearly
n n 1 1 n n 1 n
independentoverthesubspaceB, and thereforecodim (A) = n ≤ codim (B). (cid:4)
A B
Lemma 2.3. [10] Let A and B be vector spaces over the same finite scalar field, let B be a
subspace of B, and let f : A → B be a linear function. Then f(t) ∈ B on a subspace of A of
codimensionatmostthecodimensionof B.
Proof. Let A = {t ∈ A : f(t) ∈ B}. Then f(A\A) ⊆ B\B and the result follows from
Lemma2.2. (cid:4)
Lemma 2.4. [10] Let V be a finite dimensional vector space and let A ,...,A ,B be subspaces
1 k
of V. Then for i = 1,...,k, there exist linear functions f : B → A such that f +···+f = I
i i 1 k
on asubspaceof B ofcodimensionH(B|A ,...,A ).
1 k
Proof. Let W be a subspace of B defined by W = hA ,...,A i ∩ B. The subspace on which
1 k
this lemma holds is W. If H(W) = 0 , then the lemma would be trivially true. So, assume that
H(W) > 0, and let {w ,...,w } be a basis for W. For each j = 1,...,n, choose x ∈ A for
1 n i,j i
i = 1,...,k such that w = x + ··· + x . For each i = 1,...,k, define a linear mapping
j 1,j k,j
g : W → A so that g (w ) = x for all i and j. Then extend g arbitrarily to f : B → A .
i i i j i,j i i i
Now we have linear functions f ,...,f such that f + ··· + f = I on W. The dimension
1 k 1 k
of W is H(W) = I(A ,...,A ;B), so the codimension of W is H(B) − I(A ,...,A ;B) =
1 k 1 k
H(B|A ,...,A ). (cid:4)
1 k
Lemma 2.5. [10]Let V beafinite-dimensionalvector spaceandletA,B, andC besubspacesof
V. Let f : A → B and g : A → C belinear functionssuch thatf +g = 0 onA. Then f = g = 0
on asubspaceof AofcodimensionatmostI(B;C).
Page9 of47
