Spatially Coupled Ensembles Universally Achieve Capacity under Belief Propagation

0.93 MB·English

by  Shrinivas Kudekar

1
Spatially Coupled Ensembles Universally Achieve
Capacity under Belief Propagation
Shrinivas Kudekar∗, Tom Richardson∗ and Ru¨diger Urbanke†
∗Qualcomm, USA
Email: skudekar,tjr @qualcomm.com
{ }
†School of Computer and Communication Sciences
EPFL, Lausanne, Switzerland
Email: ruediger.urbanke@epfl.ch
2
1
0
2 Abstract—Weinvestigatespatiallycoupledcodeensembles.For Another important branch started with the introduction of
transmission over the binary erasure channel, it was recently convolutionalcodes [13] by Elias and the introduction of the
n shown that spatial coupling increases the belief propagation
sequential decoding algorithm by Wozencraft [14]. Viterbi
a threshold of the ensemble to essentially the maximum a-priori
J introduced the Viterbi algorithm [15]. It was shown to be
threshold of the underlying component ensemble. This explains

4 why convolutional LDPC ensembles, originally introduced by optimal by Forney [16] and Omura [17] and to be eminently
1 Felstro¨m and Zigangirov, perform so well over this channel. practical by Heller [18], [19].
Weshowthattheequivalentresultholdstruefortransmission Animportantdevelopmentintransmissionoverthecontinu-
] over general binary-input memoryless output-symmetric chan-
T ousinput,band-limited,additivewhiteGaussiannoisechannel
nels.Moreprecisely,given adesirederrorprobabilityandagap
I tocapacity,wecanconstructaspatiallycoupledensemblewhich was the invention of the lattice codes. It was shown in [20]–
s. fulfillstheseconstraintsuniversallyonthisclassofchannelsunder [24] thatlattice codesachievethe Shannoncapacity.A break-
c beliefpropagationdecoding.Infact,mostcodesinthatensemble through in bandwidth-limited communications came about
[
have that property. The quantifier universal refers to the single when Ungerboeck [25]–[27] invented a technique to combine
ensemble/codewhichisgoodforallchannelsbutweassumethat
1 coding and modulation. Ungerboeck’s technique ushered in a
the channel is known at the receiver.
v new era of fast modems. The technique, called trellis-coded
The key technical result is a proof that underbelief propaga-
9
tion decodingspatiallycoupled ensemblesachieveessentially the modulation (TCM), offered significant coding gains without
9
area threshold of the underlying uncoupled ensemble. compromising bandwidth efficiency by mapping binary code
9
2 We conclude by discussing some interesting open problems. symbols, generated by a convolutional encoder, to a larger
. (non-binary)signalconstellation.In [28],[29] Forneyshowed
1
that lattice codes as well as TCM schemes may be generated
0 I. INTRODUCTION
2 bythesamebasicelementsandthegeneralizedtechniquewas
1 A. Historical Perspective termed coset-coding.
v: Ever since the publication of Shannon’s seminal paper [1] Coming back to binary linear codes, in 1993, Berrou,
i and the introduction of the first coding schemes by Ham- GlavieuxandThitimajshima[30]proposedturbocodes.These
X
ming [2] and Golay [3], coding theory has been concerned codes attain near-Shannon limit performance under low-
r
withfindinglow-delayandlow-complexitycapacity-achieving complexity iterative decoding. Their remarkable performance
a
schemes.Theinterestedreadercanfindan excellenthistorical lead to a flurry of research on the “turbo” principle. Around
reviewin[4].Letusjustbrieflymentionsomeofthehighlights the same time, Spielman in his thesis [31], [32] and MacKay
before focusing on those parts that are the most relevant for andNealin[33]–[36],independentlyrediscoveredlow-density
our purpose. parity-check(LDPC)codesanditerativedecoding,bothintro-
In the first 50 years, coding theory focused on the con- duced in Gallager’s remarkable thesis [37]. Wiberg showed
struction of algebraic coding schemes and algorithms that [38] that both turbo codes and LDPC codes fall under the
were capable of exploiting the algebraic structure. Two early umbrella of codes based on sparse graphs and that their
highlights of this line of research were the introduction of iterative decoding algorithms are special cases of the sum-
Bose-Chaudhuri-Hocquenghem (BCH) codes [5], [6] as well product algorithm. This line of research was formalized by
as Reed-Solomon (RS) codes [7]. Berlekamp devised an Kschischang,Frey,andLoeligerwhointroducedthe notionof
efficient decoding algorithm [8] and this algorithm was then factor graphs [39].
interpretedby Massey as an algorithmfor findingthe shortest The next breakthrough in the design of codes (based on
feedback-shift register that generates a given sequence [9]. sparse graphs) came with the idea of using irregular LDPC
More recently, Sudan introduced a list decoding algorithm codes by Luby, Mitzenmacher, Shokrollahi and Spielman
for RS codes that decodes beyond the guaranteed error- [40], [41]. With this added ingredient it became possible to
correcting radius [10]. Guruswami and Sudan improved upon construct irregular LDPC codes that achieved performance
this algorithm [11] and Koetter and Vardy showed how to within 0.0045dB of the Shannon limit when transmitting
handle soft information [12]. over the binary-input additive white Gaussian noise chan-
2
nel, see Chung, Forney, Richardson and Urbanke [42]. The B. Prior Work on Spatially Coupled Codes
development of these codes went hand in hand with the
The potentialof spatially coupled codes has long been rec-
development of a systematic framework for their analysis by
ognized.Ourcontributionliesthereforenotintheintroduction
Luby,Mitzenmacher,ShokrollahiandSpielman[43],[44]and
of a new codingscheme, butin clarifyingthe mechanismthat
Richardson and Urbanke [45].
make these codes perform so well.
A central research topic for codes on graphs is the interac-
The term spatially coupled codes was coined in [53].
tion of the graphical structure of a code and its performance.
ConvolutionalLDPCcodes(moreprecisely,terminatedconvo-
Turbo codes themselves are a prime example how the “right”
lutionalLDPCcodes),whichwereintroducedbyFelstro¨mand
structure is important to achieve good performance [30].
Zigangirovin[54],andtheirmanyvariantsbelongtothisclass.
Further important parameters and structures are, the degree
Why do we introduce a new term? The three perhaps most
distribution (dd) and in particular the fraction of degree-two
important reasons are: (i) the term “convolutional” conjures
variable nodes, multi-edge ensembles [46], degree-two nodes
up a fairly specific node interconnection structure whereas
in a chain [47], and protographs[48], [49].
experiments have shown that the particular nature of the
Currently sparse graph codes and their associated iterative
connection is not important and that the threshold saturation
decodingalgorithmsare the best“practical”codesin termsof
effect occurs as soon as the connection is sufficiently strong;
their trade-off between performance and complexity and they
(ii) a well known result for convolutional codes says that the
are part of essentially all new communication standards.
boundaryconditionsare“forgotten”exponentiallyfast;butfor
Polar codes represent the most recent development in cod-
spatially coupled codes it is exactly the boundary condition
ing theory [50]. They are provably capacity achieving on
which causes the effect and there is no decay of this effect
binary-input memoryless output-symmetric (BMS) channels
in the spatial dimension of the code; (iii) the same effect
(and many others) and they have low decoding complexity.
has (empirically)been shown to hold in many other graphical
They also have no error floor due to a minimum distance
models, most of them outside the realm of coding; the term
which increases like the square root of the blocklength. The
“spatial coupling” is perhaps then somewhat more generally
simplicity,elegance,andwideapplicabilityofpolarcodeshave
applicable.
made them a popular choice in the recent literature. There
There is a considerable literature on convolutional-like
are perhaps only two areas in which polar codes could be
LDPC ensembles. Variations on the constructions as well
further improved. First, for polar codes the convergence of
as some analysis can be found in Engdahl and Zigangirov
theirperformancetotheasymptoticlimitisslow.Currentlyno
[55], Engdahl, Lentmaier, and Zigangirov [56], Lentmaier,
rigorousstatementsregardingthisconvergenceforthegeneral
Truhachev, and Zigangirov [57], as well as Tanner, D. Srid-
case are known. But “calculations” suggest that, for a fixed
hara, A. Sridharan, Fuja, and Costello [58].
desired error probability, the required blocklength scales like
1/δµ, where δ is the additive gap to capacity and where µ In[59],[60],Sridharan,Lentmaier,CostelloandZigangirov
depends on the channel and has a value around 4, [51], [52]. consider density evolution (DE) analysis for convolutional
Note that random block codes under MAP decoding have LDPC ensembles and determine thresholds for the BEC.
a similar scaling behavior but with µ = 2. This implies a The equivalent results for general channels were reported by
considerably faster convergence to the asymptotic behavior. Lentmaier, Sridharan, Zigangirov and Costello in [60], [61].
The value 2 is a lower bound for µ for any system since the This DE analysis is in many ways the starting point for our
variations of the channel itself imply that µ 2. The second investigation.Bycomparingthethresholdstothethresholdsof
aspect is universality: the code design of pola≥r codes depends theunderlyingensemblesunderMAPdecoding(seee.g.[62]),
onthespecificchannelbeingusedandoneandthesamedesign it quicklybecomes apparentthat an interesting effectmust be
cannotsimultaneouslyachievecapacityoveranon-trivialclass atwork.Indeed,inarecentpaper[63],LentmaierandFettweis
of channels (under successive cancellation decoding). followedthis route and independentlyformulatedthe equality
Letusnowconnectthecontentofthispapertotheprevious of the belief propagation (BP) threshold of convolutional
discussion. Our main aim is to explain the role of a further LDPC ensembles and the MAP threshold of the underlying
structuralelementin the realm of sparse graphcodes(besides ensemble as a conjecture.
the previously discussed such examples), namely that of ArepresentationofconvolutionalLDPCensemblesinterms
“spatial coupling.” We will show that this coupling of graphs of a protograph was introduced by Mitchell, Pusane, Zigan-
leadstoaremarkablechangeintheirperformance.Ensembles girov and Costello [64]. The correspondingrepresentation for
designed in this way combine some of the nice elements of terminated convolutionalLDPC ensembles was introducedby
polar codes (namely the fact that they are provably capacity Lentmaier, Fettweis, Zigangirov and Costello [65]. A variety
achieving under low complexity decoding) with the practical ofconstructionsofLDPCconvolutionalcodesfromthegraph-
advantages of sparse graph codes (the codes are competitive coverperspectiveisshownbyPusane,Smarandache,Vontobel,
already for moderate lengths). Perhaps most importantly, it and Costello [66].
is possible to construct universal such codes for the whole A pseudo-codewordanalysis of convolutionalLDPC codes
class of BMS channels. Here, universality refers to the fact was performed by Smarandache, Pusane, Vontobel, and
that one and the same ensemble is good for a whole class of Costello in [66]–[68]. Such an analysis is important if we
channels,assuming that at the receiverwe have knowledgeof wanttounderstandtheerror-floorbehaviorofspatiallycoupled
the channel. ensembles.
3
In [69], Papaleo, Iyengar, Siegel, Wolf, and Corazza study the threshold saturation phenomenon also applies to general
the performance of windowed decoding of convolutional BMS channels.
LDPC codes on the BEC. Such a decoder has a decoding FortypicalsparsegraphensemblestheMAPthresholdisnot
complexity which is independent of the chain length, an equaltothe Shannonthresholdbutthe Shannonthresholdcan
important practical advantage. Luckily, it turns out that the onlybereachedbytakingasequenceofsuchensembles(e.g.,
performance under windowed decoding, when measured in a sequence of increasing degrees). There are some notable
terms of the threshold, approaches the “regular”” (without exceptions, like MN ensembles or HA ensembles. Kasai and
windowed decoding) threshold exponentially fast in the win- Sakaniwa take this as a starting point to investigate in [77]
dowsize,see[70],[71].Thethresholdsaturationphenomenon whether by spatially coupling such ensembles it is possible
therefore does not require an infinite window size. to create ensembles which are universally capacity achieving
The scaling behavior of spatially coupled ensembles, i.e., under BP decoding.
the relationship between the chain length, the number of
variables per section, and the error probability is discussed E. Spatial Coupling for General Communication Scenarios,
by Olmos and Urbanke in [72]. Signal Processing, Computer Science, and Statistical Physics
The principle which underlies the good performance of
spatially coupled ensembles is broad. It has been shown to
C. Prior Results for the Binary Erasure Channel
apply to a variety of problems in communications, computer
Itwas recentlyshownin [53] thatfortransmission overthe science, signal processing, and physics. To mention some
BEC spatially coupled ensembles have a BP threshold which concrete examples, the threshold saturation effect (dynami-
is essentially equal to the MAP threshold of the underlying cal/algorithmic threshold of the system being equal to the
uncoupledensemble.Further,this thresholdis also essentially static or condensation threshold) of coupled graphicalmodels
equal to the MAP threshold of the coupled ensemble. This has been observed for rate-less codes by Aref and Urbanke
phenomena was called threshold saturation in [53] since the [78], for channels with memory and multiple access channels
BP threshold takes on its largest possible value (the MAP with erasure by Kudekar and Kasai [79], [80], for CDMA
threshold). This significant improvement in the performance channels by Takeuchi, Tanaka, and Kawabata [81], for relay
is due to the spatial coupling of the underlying code. Those channels with erasure by Uchikawa, Kasai, and Sakaniwa
“sections”ofthecodethathavealreadysucceededindecoding [82], for the noisy Slepian-Wolf problem by Yedla, Pfister,
canhelptheirneighboringlessfortunatesectionsinthedecod- and Narayanan [83], and for the BEC wiretap channel by
ing process. In this manner, the information propagates from Rathi, Urbanke, Andersson, and Skoglund [84]. Uchikawa,
the “boundaries”, where the bits are known perfectly towards Kurkoski, Kasai, and Sakaniwa recently showed an improve-
the “middle”. In a recent paper [63], Lentmaier and Fettweis ment of the BP threshold has also for transmission over
independently formulated the same statement as a conjecture the unconstrained AWGN channel using low-density lattice
andprovidednumericalevidenceforitsvalidity.Theyattribute codes [85]. Further, Yedla, Nguyen, Pfister and Narayanan,
theobservationoftheequalityofthetwothresholdstoG.Liva. demonstrated the universality of spatially-coupled codes in
It was shown in [64], [67], [68], [73] that if we couple the 2-user binary input Gaussian multiple-access channel and
componentcodes whose Hamming distance grows linearly in finite state ISI channels like the dicode-erasure channel and
theblocklengththenalsotheresultingcoupledensembleshave the dicode channel with AWGN [86], [87]. In [86] they
thisproperty(assumingthatthenumberof“sections”orcopies show in addition that for a fixed rate pair, spatially-coupled
of the underlying code is kept fixed). The equivalent result is ensembles universally saturate the achievable region (i.e., the
true for stopping sets. This implies that for the transmission setofchannelgainparametersthatareachievableforthefixed
over the BEC the block BP threshold is equal to the bit BP rate pair) under BP decoding. Similarly, in [87] they provide
threshold and that such ensembles do not exhibit error floors numerical evidence that spatially coupled ensembles achieve
under BP decoding. the symmetricinformationrate for the dicodeerasure channel
and the dicode channel with AWGN.
In signal processing and computer science spatial coupling
D. Prior Results for General Binary-Input Memoryless
has found success in the field of compressed sensing [88]–
Output-Symmetric Channels
[91]. In [88], Kudekar and Pfister use sparse measurement
As pointed out in a preceding section, BP thresholds for matriceswithsub-optimalverificationdecodingandshowthat
transmission over general BMS channels were computed by spatial coupling boosts thresholds of sparse recovery.In [90],
means of a numerical procedure by Lentmaier, Sridharan, [91], Krzakala, Me´zard, Sausset, Sun, and Zdeborova as well
ZigangirovandCostello in[61].Further,in [74] (conjectured) as Donoho,Javanmard, and Montanarishow that by carefully
MAP thresholds for some LDPC ensembles were computed designing dense measurement matrices using spatial coupling
according to the Maxwell construction. Comparing these two one can achieve the best possible recovery threshold, i.e., the
values, one can check empirically that also for transmission oneachievedbytheoptimalℓ decoder.Thus,thephenomena
0
over general BMS channels the BP threshold of the coupled of threshold saturation is also demonstrated in this case. This
ensembles is essentially equal to the (conjectured) MAP developmentis quite remarkable.
threshold of the underlying ensemble. Indeed, recently both Statistical physics is another very natural area in which
[75] as well as [76] provided further numerical evidence that the threshold saturation phenomenon is of interest. For the
4
so-called random K-SAT problem, random graph coloring, to individual codes rather than ensembles and how this gives
and the Curie-Weiss model,spatially coupledensembleswere riseto codeswhichareuniversallyclosetocapacityunderBP
investigated by Hassani, Macris, and Urbanke, [92]–[94]. In decoding for the whole class of BMS channels.
all these cases, the threshold saturation phenomenon was WeendinSectionIV-Ewithadiscussionofwhatchallenges
observed. This suggests that it might be possible to study still lie ahead. In particular, spatial coupling has been shown
difficult theoretical problems in this area, like the existence empirically to lead to the threshold saturation phenomenonin
of the static threshold, by studying the dynamical threshold a wide class of graphical models. Rather than proving each
of a chain of coupled models, perhaps an easier problem. such scenario in isolation, we want a common framework to
Furtherspatially-coupledmodelswereconsideredbyTakeuchi analyze all such systems.
and Tanaka [95]. Many of the proofs are relegated to the appendices. This
makesitpossibleto readthe materialontwo levels– acasual
F. Main Results and Consequences level,skippingalltheproofsandfollowingonlytheflowofthe
argument, and a more detailed level, consulting the material
In this paper we show that for transmission over general
in the appendices.
BMS channels coupled ensembles exhibit the threshold sat-
uration phenomenon. By choosing e.g. regular component
ensembles of fixed rate and increasing degree, this implies II. UNCOUPLEDSYSTEMS
that coupled ensembles can achieve capacity over this class
A. Regular Ensembles
of channels. More precisely, for each δ > 0 there exists a
Definition 1 ((d,d )-Regular Ensemble): Fix 3 d
coupled ensemble which achieves at least a fraction 1−δ of d ,d,d N,andlnrsothatnd/d N.The(d ,d≤)-reglul≤ar
capacity universally, under belief propagation decoding, over r l r l r l r
∈ ∈
LDPCensembleofblocklengthn isdefinedas follows.There
the whole class of BMS channels.The qualifier ”universal”is
arenvariablenodesandndl checknodes.Eachvariablenode
important here. dr
hasdegreed andeachchecknodehasdegreed .Accordingly,
Coupled ensembles inherit to a large degree the error floor l r
each variable node has d sockets, i.e., d places to connect
behaviorof the underlyingensemble. Further,such an ensem- l l
an edge to, and each check node has d sockets. Therefore,
ble can be chosen so that it has a non-zero error correcting r
there are in total d n variable-node sockets and the same
radius, and hence does not exhibit error floors. To achieve l
number of check-node sockets. Number both kinds from 1
this, it suffices to take the variable-node degree to be at least
to nd. Consider the set of permutations Π on 1,...,nd .
five. This guaranteesthata randomlychosengraphfrom such l l
{ }
Endow this set with a uniform probability distribution. To
an ensemble is an expander with expansion exceeding three-
sample from the (d ,d )-regular ensemble, sample from Π
quarters with high probability. This expansion guarantees an l r
and connect the variable to the check node sockets according
error correcting radius under the so-called flipping decoder
to the chosen permutation. This is the configurationmodel of
[96] as well as under the BP decoder, assuming that we
LDPC ensembles. It is inspired by the configurationmodelof
suitablyclipboththereceivedaswellastheinternalmessages
random graphs [98, Section 2.4]. (cid:4)
[97].
Although one can empirically observe the threshold sat-
uration phenomenon for a wide array of component codes, B. Binary-Input Memoryless Output-Symmetric Channels
we state and prove the main result only for regular LDPC
Throughoutwewillassumethattransmissionistakingplace
ensembles. This keeps the exposition manageable.
over a BMS channel. Let X denote the input and let Y
be the output. Further, let p(Y = y X = x) denote the
G. Outline |
transition probability describing the channel. An alternative
In Section II we briefly review regular LDPC ensembles characterizationofthe channelis bymeansofits so-calledL-
and their asymptotic (in the blocklength) analysis. Much of distribution,denoteitbyc.Moreprecisely,cisthedistribution
this material is standard and we only include it here to set of
thenotationandtomakethepaperlargelyself-contained.The
p(Y X =1)
two most important exceptions are our in-depth discussion of ln |
p(Y X = 1)
the Wasserstein distance and the the so-called area threshold, | −
in particular the (Negativity) Lemma 27. conditioned that X =1.
In Section III we review some basic properties of coupled Givenc,wewritec, c,and C todenotethecorresponding
| | | |
ensembles. Using simple extremes of information combining D distribution, the D distribution and the cdf in the D -
| | | |
techniques, we will see in Section III-G that coupling indeed domain, respectively, see [62, Section 4.1.4].
increases the BP threshold significantly, even though these Typically we do not consider a single channel in isolation
simple arguments are not sufficient to characterize the BP butawholefamilyofchannels.Wewrite BMS(σ) todenote
{ }
threshold under coupling exactly. thefamilyparameterizedbythescalarσ.Oftenitwillbemore
We state our main result, namely that the BP threshold of convenienttodenotethisfamilyby c ,i.e.,tousethefamily
σ
{ }
coupled ensembles is essentially equal to the area threshold of L-densities which characterize the channel family. If it is
of the underlying component ensemble, in Section IV. We important to make the range of the parameter σ explicit, we
also discusshowonecaneasily strengthenthisresultto apply will write c σ.
{ σ}σ
5
Sometimes it is convenient to use the natural parameter The three fundamental channel families BEC(ǫ) 1,
of the family. For example, for the three fundamental chan- BSC(p) 12,and BAWGNC(σ) areallcomple{te,order}ed0,
nels, the BEC, the binary symmetric channel (BSC) and the {smooth, a}n0d symm{etric. }∞0
binary additive white-Gaussian noise channel (BAWGNC),
the corresponding channel families are given by BEC(ǫ) 1,
1 { }0 C. MAP Decoder and MAP Threshold
BSC(p) 2, and BAWGNC(σ) . Other times, it is more
{ }0 { }∞0 The bit maximum a posteriori (bit-MAP) decoder for bit i
convenient to use a common parameterization. E.g., we will
write BMS(h) to denote a channel family where BMS(h) findsthevalueofxi whichmaximizesp(xi|y1n).Itminimizes
denote{s the elem}ent in the family of entropy h. the bit error probability and is optimal in this sense. The
block maximum a posteriori (block-MAP) decoder finds the
Assume that we are given a channel family BMS(σ) σ.
{ }σ codeword xn which maximizes p(xn yn). It minimizes the
We say that the family is complete if H(BMS(σ)) = 0, 1 1 | 1
block error probability and is optimal in this sense.
H(BMS(σ)) = 1, and for each h [0,1] there exists a
parameterσ sothatH(BMS(σ))=h.H∈ereH()istheentropy Definition 2 (MAP Threshold): Consider an ordered and
functional defined in Section II-D. · complete channel family {ch}. The MAP threshold of the
(d ,d )-regular ensemble for this channel family is denoted
Letp (z x) denotethe transition probabilityassociated l r
to a BMZS|cXhan|nelc andletp (y x) denotethetransition by hMAP(dl,dr) and defined by
′ Y X
dperogbraadbeilditywiotfharensopthecetrtBoMciSftchhear|neneexlisc|t.sWaechtahnenneslapyZthYa(tzc′yi)s inf{h∈[0,1]:linm→i∞nfE[H(X1n|Y1n(h))/n]>0},
so that | | where H(Xn Yn(h)) is the conditional entropy of the trans-
1 | 1
mitted codeword Xn, chosen uniformly at random from the
1
p (z x)= p (y x)p (z y). code, given the received message Yn(h) and where the ex-
Z|X | Xy Y |X | Z|Y | DpeicsctautsiosinonE:[·D]eisfinwertPthe=(dPl,rdXr)-r=egXu1ˆla(rYenns)em, wblhee.re Xˆ (Yn(cid:4))
We will use the notation c ≺c′ to denote that c′ is degraded istheMAPestimateoef,ibitib{aseid6onthieo1bse}rvationYni.No1te
wrt c (as a mnemonic think of c as the erasure probability of 1
that by the Fano inequality we have H(X Yn) h (P ).
a BAEuCseafnudl crehpalraaccete≺rizwatiitohn<of).degradation,see [62, Theorem Assume that we are transmitting above ih|MAP1(dl,≤dr) 2so eth,iat
E[H(Xn Yn)/n] δ >0.1 Then
4.74], is that c c is equivalent to 1 | 1 ≥
′
≺ n n n
1 1
1 1 h (E[ P ]) E[ h (P )] E[ H(X Yn)/n]
Z0 f(x)|c|(x)dx≤Z0 f(x)|c′|(x)dx (1) 2 nXi=1 e,i ≥ E[nHX(i=X1n 2Yne),i/n]≥ δXi>=10. i| 1
for all f(x) that are non-increasing and concave on [0,1]. ≥ 1 | 1 ≥
Inwords,ifwearetransmittingabovetheMAPthreshold,then
Here, c(x) is the so called D -density associated to the L-
| | | | theensembleaveragebit-errorprobabilityislowerboundedby
density c, see [62, p. 179]. In particular, this characterization
implies that F(a) F(b) for a b if F() is either the h−21(δ),astrictlypositiveconstant.Thisensembleistherefore
≤ ≺ · not suitable for reliable transmission above this threshold.
Battacharyyaortheentropyfunctional.Thisistruesince both
In generalwe cannot conclude from E[H(Xn Yn)/n] δ
are linear functionals of the distributions and their respective 1 | 1 ≤
that the average error probability is small.2
kernels in the D -domain are decreasing and concave. An
| |
alternative characterization in terms of the cumulative distri-
bution functions C(x) and C (x) is that for all z [0,1], D. Belief Propagation, Density Evolution, and Some Impor-
′
| | | | ∈
tant Functionals
1 1
C(x)dx C′ (x)dx. (2) In principle one can investigate the behavior of coupled
| | ≤ | |
Zz Zz ensembles under any message-passing algorithm. We limit
A BMS channel family BMS(σ) σ is said to be ordered our investigation to the analysis of the BP decoder, the most
{ }σ
by degradation if σ σ implies c c . (The reverse
order, σ1 ≥ σ2, is a1lso≤all2owed but wσe1g≺enerσa2lly stick to the for1Walel hnav>eEn[0H,(Xlet1ns |sYay1n.)F/unrt]h≥er,12folrim1in≤fnn→≤∞nn10E,[HE([HX(1nX|1nY|1nY(1nh)))/]n>] i0s
stated convention.) strictlypositiveunlessthechannelistrivial. Theclaimfollowsbytakingthe
We say that an L-density c is symmetric if a( y) = minimum of all of the bounds for 1 ≤ n ≤ n0 as well as the bound for
a(y)e−y. We recall that all densities which stem from− BMS n>2Thni0s. is possible if we have the slightly stronger con-
channels are symmetric, see [62, Sections 4.1.4, 4.1.8 and dition E[Pni=1H(Xi|Y1n)/n] ≤ δ. In this case δ ≥
4w.i1lA.l9t]hB.eMAreSlflocrdeheannnsonitteiemlsefanwmtihoiilncyhsywcmemceiotsrnyssiaedixedprltiaocrietbleysyismnmmtohoeetthrsieciq.fuWfeole.r nnn111EEE[[[PPPninini===111HEPeY(,X1in]i[2≤|Ym112ninδ)x].pT(h=xe|lYas1nn1t)Es]t][ePpnii=n=1thEeY1pnn1re[hEvi2[oP(umsnii=cn1hxa2pinP(exo,i|f]Y,in1nes)qo)u]a]litthie≥ast
{ σ} followssinceunderMAPdecodingtheerrorprobability conditioned thatwe
allcontinuouslydifferentiablefunctionsf(y)sothatey/2f(y) observed y1n is equal to minxp(x|y1n). An alternative way to prove this
is bounded, the integral f(y)cσ(y)dy exists and is a con- is to realize that H(Xi|Y1n) represents a BMS channel with a particular
entropyandtouseextremesofinformationcombiningtofindtheworsterror
tinuously differentiable function with respect to σ, see [62,
R probabilitysuchachannelcanhave.Theextremalchannelinthiscaseisthe
Definition 4.32]. BEC.
6
powerful local message-passing algorithm. We are interested F(a ⊛(a ⊛(⊛d a ))
≥ BEC BEC i=3 i
in the asymptotic performance of the BP decoder, i.e., the
=
performance when the blocklength n tends to infinity. This ···
F(a ⊛(a⊛d 1))
asymptoticperformanceis characterizedbythe so-called den- ≥ d BEC−
sity evolution (DE) equation [45]. ≥F(aBEC⊛(aB⊛EdC−1))=αd.
Definition 3 (Density Evolution): For ℓ 1, the DE equa-
≥ The same remarkand the same prooftechniqueapplies to the
tion for a (d ,d )-regular ensemble is given by
l r other cases.
xℓ =c⊛(xℓ(cid:24)−d1r−1)⊛dl−1. H(Lae⊛mmb)a+6H(D(aua(cid:24)libty)R=ulHe(–a)[+62H, p(b.)1.96]): For any a and b
Here, c is the L-density of the BMS channel over which Note:We givea simpleproofof thisidentityattheendofthe
transmission takes place and xℓ is the density emitted by proof of Lemma 53.
variable nodes in the ℓ-th round of density evolution. Initially
we have x = ∆ , the delta function at 0. The operators ⊛
0 0 F. Fixed Points, Convergence, and BP Threshold
and (cid:24) correspond to the convolution of densities at variable
We say that the density x is a fixed point (FP) of DE for
and check nodes, respectively, see [62, Section 4.1.4]. (cid:4)
the (d ,d )-regular ensemble and the channel c if
As mentioned, all distributions associated to BMS channels l r
are symmetric and symmetry is preserved under DE, see [62, x=c⊛(x(cid:24)dr−1)⊛dl−1. (3)
Chapter 4] for details. There are a number of functionals of
More succinctly,when the underlyingensemble is understood
densities are of interest to us. The most importantfunctionals
from the context, we say that (c,x) is a FP.
are the Battacharyya, the entropy, and the error probability
One way to generatea FP is to initialize x with ∆ andto
functional. For a density a these are denoted by B(a), H(a), 0 0
run DE, as stated in Definition 3. We call such a FP a FP of
and E(a), respectively. Assuming a is an L-density, they are
forwardDE.TheresultingFPsarethe“natural”FPssincethey
given by
have a natural operational meaning – if we pick sufficiently
B(a)= a(y)e−y/2dy, H(a)= a(y)log2(1+e−y)dy, long ensembles, these are the FPs which we can observe in
simulations when we run the BP decoder.
Z Z
E(a)= 1 a(y)e (y/2+y/2)dy. Definition 7 (Weak Convergence): We say that a sequence
− | |
2 of distributions a converges weakly to a limit distribution
Z { i}
Weendthissectionwiththefollowingusefulfact.Theproof a if for the corresponding cumulative distributions in the
can be found in Appendix A. D -domain, call them Ai , for all bounded and continuous
| | { }
Lemma 4 (Entropy versus Battacharyya): For any L- functions f(x) on [0,1] we have
density a, B2(a) H(a) B(a). 1 1
≤ ≤ lim f(x)dA (x)= f(x)dA(x).
i
i→∞Z0 | | Z0 | |
E. Extremes of Information Combining and the Duality Rule
An equivalentdefinitionis that A (x) convergesto A(x) at
i
| | | |
When we are operating on BMS channels, the quantities points of continuity of x. (cid:4)
appearingintheDEequationsaredistributions.Thesearehard A simple proofof the followinglemma can be foundatthe
totrackanalyticallyingeneral,unlesswearetransmittingover end of Section II-I.
the BEC. Oftenwe onlyneedbounds.In thesecases extremes Lemma 8 (Convergence of Forward DE – [62, Lemma 4.75]):
ofinformationcombiningideasarehandy,see[99]–[103],[62, The sequence x of distributions of forward DE converges
ℓ
{ }
p. 242]. weakly to a symmetric distribution.
Lemma 5 (Extremes of Information Combining): Let F() Lemma 9 (BP Threshold): Consider an ordered and com-
·
denoteeitherH(·)orB(·)andletα∈[0,1].LetaBEC andaBSC plete channel family {cσ}. Let xℓ(σ) denote the distribution
denoteL-densitiesfromthefamilies BEC(ǫ) and BSC(p) , in the ℓ-th round of DE when the channelis c . Then the BP
{ } { } σ
respectively, so that F(aBEC)=F(aBSC)=α. Then for any b, threshold of the (dl,dr)-regular ensemble is defined as
(i) min F(a⊛b)=F(a ⊛b)
(ii) maxaa::FF((aa))==ααF(a⊛b)=F(aBBESCC⊛b) σBP(dl,dr)=sup{σ :xℓ(σ)ℓ→→∞∆+∞}.
(iii) min F(a(cid:24)b)=F(a (cid:24)b) Inotherwords,theBPthresholdischaracterizedbythelargest
a:F(a)=α BSC
(iv) max F(a(cid:24)b)=F(a (cid:24)b) channel parameter so that the forward DE FP is trivial.
a:F(a)=α BEC
Discussion: Although the extremes of information combining WehavejustseenthattheFPsofforwardDEareimportant
boundsareonlystatedforpairsofdistributions,theynaturally since they characterize the BP threshold. But there exist FPs
extend to more than two distributions. E.g., we claim that that cannot be achieved this way. Let us review a general
min F(a⊛d)=F(a )d =αd.Toseethis,let a d method of constructing FPs. Assume that, given a channel
a:F(a)=α BEC { i}i=1
be any set of distributions so that F(ai) = α. Then we can family {cσ}, we need a FP x which has a given error
use Lemma 5 repeatedly to conclude that probability E(x), entropy H(x), or Battacharyya parameter
B(x). Such FPs can often be constructed, or at least their
F(a ⊛(⊛d a )) F(a ⊛(⊛d a ))
1 i=2 i ≥ BEC i=2 i existencecanbeguaranteed,byaprocedureintroducedin[74].
=F(a ⊛(a ⊛(⊛d a )) Let us recall this procedure for the case of fixed entropy.
2 BEC i=3 i
7
Consider a smooth, complete, and ordered family ch and H. The Wasserstein Metric: Definition and Basic Properties
{ }
the(dl,dr)-regularensemble.LetusdenotebyThtheordinary In the sequel we will often need to measure how close var-
density evolution operator at fixed channel ch. Formally, ious distributions are. Sometimes it is convenient to compare
Th(a)=ch⊛(a(cid:24)dr−1)⊛dl−1. (4) their entropy or their Battacharyya constant. But sometimes a
more general measure is required. The Wasserstein metric is
For any e [0,1], we define the density evolution operatorat our measure of choice.
fixed entro∈py e, call it Re, as Definition 12 (Wasserstein Metric – [104, Chapter 6]):
Let a and b denote two D -distributions. The Wasserstein
Re(a)=Th(a,e)(a), (5) metr|ic|, deno|te|d by d(a, b|),|is defined as
| | | |
where h(a,e) is the solution of H(Th(a)) = e. Whenever 1
no such value of h exists, Re(a) is left undefined. Since, d(|a|,|b|)= sup f(x)(|a|(x)−|b|(x))dx , (6)
for a given a, the family Th(a) is ordered by degradation, f(x)∈Lip(1)[0,1](cid:12)Z0 (cid:12)
H(Th(a)) is a non-decreasing function of h. As a conse- where Lip(1)[0,1] denotes(cid:12)(cid:12)the class of Lipschitz cont(cid:12)(cid:12)inuous
quence the equation H(Th(a)) = e cannot have more than functions on [0,1] with Lipschitz constant 1. (cid:4)
a single solution. Furthermore, by the smoothness of the Discussion: In the sequel we will say that a function f(x) is
channel family ch, H(Th(a)) is continuous as a function of Lip(c) as a shorthand to mean that it is Lipschitz continuous
h. Notice that H(T (a)) = 0: if the channel is noiseless withconstantc.Ifwe wanttoemphasizethedomain,thenwe
0
the output density at a variable nodes is noiseless as well. writee.g.,Lip(c)[0,1].Whyhavewedefinedthemetricinthe
Therefore, a necessary and sufficient condition for a solution D -domain? As the next lemma shows, convergence in this
h(a,e) to exist (when the family ch is complete) is that |me|tric implies weak convergence. Since all the distributions
H(DT1e(fian)i)tio=nH1(0(a(D(cid:24)dEr−a1t)F⊛idxle−d1)En≥treo{p(ysee}e):ThSeeotream0 =6 ince[.74F]o)r. oinfsitneatedreosftathreesDym-dmometariicn,.iTtosueffiasceesotuorlnoootkataitonth,eh|oDw|e-vdeorm, awine
ℓ 0 compute aℓ+1 =Re(aℓ). (cid:4) willformallywriteexpressionsliked(a,b), i.e.,wewillallow
≥
Discussion: It can be shown that if the aboveproceduregives the arguments to be e.g. L-distributions. It is then implied
rise to an infinite sequence, i.e., if Re() is well-defined at thatthe metric is determinedusing the equivalent D -domain
· | |
each step, then this sequence has a converging subsequence. representations as defined above.
In fact, in practice one observes that the sequence itself Lemma 13 (Basic Properties of the Wasserstein Metric):
converges. The computation of the convolutions is typically In the following, a, b, c, and d denote L-distributions.
donenumericallyeitherbysamplingorviaFouriertransforms In the D domain we have the following expressions for
| |
as in ordinary density evolution. Due to the monotonicity B(a) and H(a) (compare this to the expressions in the L
of H(Th(aℓ)) in h, the value of h(aℓ,e) can be efficiently domain given in Section II-D),
found by a bisection method. The procedure is halted when
1
some convergencecriterion is met – e.g., one can require that B(a)= 1 x2 a(x)dx,
| | − | |
s(amaplrloepretrhlayndaefithnreeds)hodlids.tance between aℓ and aℓ+1 becomes Z01p 1 x
H(a)= h − a(x)dx,
2
Any FP of the above transformation Re, i.e., any a such | | Z0 (cid:16) 2 (cid:17)| |
that a=Re(a), is also a FP of ordinary density evolution for where h (x)= xlog x (1 x)log (1 x) is the binary
the channel ch with h = h(a,e). Furthermore, if a sequence entropyf2unction−.See [2104−], [10−5] form2ore−detailson metrics
of densities such that aℓ+1 =Re(aℓ) converges(weakly) to a for probability measures.
density a, then a is a FP of Re, with entropy e. (i) Alternative Definitions:
d(a,b)= inf E[X Y ],
G. BP Threshold for Large Degrees p(x,y):p(x) a;p(y) b | − |
∼| | ∼| |
What happens to the BP threshold when we fix the design 1
d(a,b)= A(x) B(x)dx.
rate r =1−dl/dr and increase the degrees?The proofof the Z0 || | −| | |
following lemma, which uses basic extremes of information
(ii) Boundedness: d(a,b) 1.
combining arguments, can be found in Appendix B. ≤
(iii) MetrizableandWeakConvergence:TheWassersteinmet-
Lemma 11 (Upper Bound on BP Threshold): Consider
ricinducestheweaktopologyonthespaceofprobability
transmission over an ordered and complete family ch of
{ } measures on [0,1]. In other words, the space of proba-
BMS channels using an (d ,d )-regular dd and BP decoding.
l r bility measures under the weak topology is metrizable
Let r = 1− ddrl be the design rate and let hBP(dl,dr) denote and convergence in the Wasserstein metric is equivalent
the BP threshold. Then,
to weak convergence(see [104, Theorem 6.9]).
h ( 1 ) (iv) Polish Space: The space of probability distributions on
hBP(dl,dr)≤ 1−((1−2 r2)√ddrr)−e1−2√dr−1. s[0e,p1a]rambleetrmizeetdribcyspthaceeW,ia.es.s,earsPteoilnisdhisstpaanccee,aisndaacnoymmpleeate-
In particular, by increasing dr while keeping the rate r fixed, sure can be approximated by a sequence of probability
the BP threshold converges to 0. measureswithfinitesupport,i.e.,distributionsoftheform
8
n c δ(x x ), where n c = 1, c 0, and I. Wasserstein Metric and Degradation
i=1 i − i i=1 i i ≥
xi [0,1]. Further, the space is compact. (See [104, When densities ordered by degradation, some the Wasser-
P ∈ P
Theorem 6.18].)
stein metric inherits some additional properties.
(v) Convexity: Let α [0,1]. Then Lemma 14 (Wasserstein Metric and Degradation): In the
∈
following a and b denote L-distributions.
d(αa+α¯b,αc+α¯d) αd(a,c)+α¯d(b,d).
≤ (i) Wasserstein versus Degradation: Let a b. Let A and
≺ | |
In general, if α =1, then B denote the corresponding D -domain cdfs. Define
i i |D(|a,b) = 1x(B(x) A(x|))d|x. Note that D(a,b)
d( Pαiai, αibi) αid(ai,bi). can be seen0as a| m|easu−re|of| how much b is degraded
≤ R
wrt a since it is the averageof the non-negativeintegrals
(vi) RegularityXwrt ⊛: TXhe WasserstXein metric satisfies the
1(B(x) A(x))dx (cf. (2)). Then
regularity property d(a⊛c,b⊛c) 2d(a,b), so that z | | −| |
≤
R D(a,b) d2(a,b)/4.
d(a⊛c,b⊛d) d(a⊛c,b⊛c)+d(b⊛c,b⊛d) ≥
≤
Furthermore, D(a,b) 1 and for any symmetric densi-
2d(a,b)+2d(c,d), ≤
≤ ties such that a b c, D(a,c)=D(a,b)+D(b,c).
andfori 2andanydistributionc,d(a⊛i⊛c,b⊛i⊛c) (ii) Entropy and Ba≺ttach≺aryya Bound Wasserstein Distance:
≥ ≤
2id(a,b). Let a b. Then
≺
(vii) Regularity wrt (cid:24): The Wasserstein metric satis-
d(a,b) 2 (ln2)(H(b) H(a)) 2 B(b) B(a)
fies the regularity property d(a (cid:24) c,b (cid:24) c) ≤ − ≤ −
≤
d(a,b) 1 B2(c) d(a,b), so that and B(b)−pB(a)≤ 2(H(b)−H(a))p.
− ≤ (iii) Continuity for Ordered Families: Consider a smooth
q p
d(a(cid:24)c,b(cid:24)d)≤d(a(cid:24)c,b(cid:24)c)+d(b(cid:24)c,b(cid:24)d) sfaomtihlyatoBf L(-)diisstricbounttiionnusou{scσw}σσrtoσrdered[σb,yσ]d.eTgrhaednatitohne
d(a,b)+d(c,d). · ∈
≤ Wasserstein metric is also continuous in σ.
Further, Discussion: Property (i) is particularly useful. Imagine a
sequence of distributions a n ordered by degradation,
i { i}i=0
d(a(cid:24)i,b(cid:24)i)≤d(a,b) (1−B2(a))i−2j(1−B2(b))j−21. fi.reo.m, a0[62≺] tah1at≺D·(a··,≺a )an=. The1nz(aA0 ≺ anAan)ddzweisknnoonw-
Xj=1 0 n 0 | |n − | |0
negativesince itis the “average”ofthe non-negativeintegrals
(viii) Regularity wrt DE: Let Tc() denote the DE operator for 1(A A )dz. Now noteRthat D(, ) is additive and
thedd(d ,d )andthechan·nelc.Thend(T (a),T (b)) y | |n − | |0 · ·
l r c c that D(a ,a ) 1. From these two facts we can conclude
αd(a,b), with ≤ R 0 n ≤
that there must exist an index i, 0 i n 1, so that
≤ ≤ −
D(a ,a ) 1. More generally, we can conclude for any
α=2(dl−1)dr−1(1−B2(a))dr−21−j(1−B2(b))j−21. 1so≤thikat≤i+D1n(at≤h,aatnthe)re mmusitnexistkan,i1ndex i,2k0.≤Thiis≤fnoll−owks,
Xj=1 i i+k ≤ {n k+1 } ≤ n
by upper bounding the average−of all these n k +1 such
(ix) Wasserstein Bounds Battacharyya and Entropy: −
distances. By property (i) this implies “closeness” also in the
B(a) B(b) d(a,b) 2 d(a,b), Wasserstein sense.Inwords,in asequenceofdistributionsor-
| − |≤ − deredbydegradationwearealwaysabletofindasubsequence
d(a,b)
H a) H(b hp p of distributions which are “close” in the Wasserstein sense.
2
| − |≤ 2
As an exercise in using the basic properties of the Wasser-
(cid:0) (cid:1) 1(cid:16) (cid:17)
d(a,b) 2 d(a,b). stein distance, let us give a proof of Lemma 8.
≤ ln2 −
Proof: Since we are considering a sequence of dis-
p p
(x) Battacharyya Sometimes Bounds Wasserstein: tributions obtained by forward DE, we have x x
ℓ ℓ+1
≻
for ℓ 0. Therefore, the quantities D(x ,x ) are non-
d(∆0,a)≤ 1−B(a)2 ≤ 2(1−B(a)), negativ≥e and they are additive in the senseℓthaℓt+D1(x ,x ) =
0 n
d(∆+∞,a)≤pB(a). p ℓn=−01D(xℓ,xℓ+1). Further, D(·,·) is upper bounded by 1.
It follows that x forms a Cauchy sequence wrt to D(, )
Discussion: Perhaps the most useful property of the Wasser- P { ℓ} · ·
and hence also wrt d(, ). This in turn implies that x
stein metric is that it interacts nicely with the operations of · · { ℓ}
converges wrt d(, ) and this convergence is equivalent to
variable- and check-node convolution. This is the essence · ·
weak convergence. Finally, symmetry can be tested in terms
of properties (vi), (vii), and (viii). For example, it is easy
of bounded continuous functionals and weak convergence
to see why property (viii) might be useful: Given that two
preserves such functionals.
distributionsaandbareclose,itassertsthatafteroneiteration
of DE these two distributions are again close. Indeed, as we
J. GEXIT Curve
will see shortly, depending on the Battacharyya parameter of
the starting distributions the distance might in fact become As we have discussed in the preceding section, FPs of DE
smaller, i.e., we might have a contraction. play a crucial role in the asymptotic analysis. E.g., the BP
9
threshold is characterized by the existence/non-existence of a
non-trivial FP of forward DE for a particular channel.
An even more powerful picture arises if instead of looking
0.6 0.6
atasingleFPatatimewevisualizeawholecollectionofFPs. R = 1
2
InordertovisualizemanyFPsatthesametimeitisconvenient
0.4 0.4
toprojectthem.E.g.,giventheFPpair(c,x)wemightdecide ))
1
toplotthe point(H(c),H(x)) inthe two-dimensionalunitbox 0.2 (u 0.2
x
[0,1] [0,1]. ,1 BP MAP
× ( ǫ ǫ
Example 15 (BP EXIT Curve for BEC): Note that for the 0.0 0.2 0.4 0.6 0.8 ǫ 0.0 0.2 0.4 0.6 0.8 ǫ
BEC, erasure probability is equal to Battacharyya parameter,
and also equal to entropy. Even though all these parameters
are equal in this case, our language will reflect that we are Fig.1. Left:TheBPEXITcurveofthe(dl =3,dr =6)-regularensemble
when transmitting over the BEC. The curve has a characteristic “C” shape.
plotting entropy. Right:TheconstructionoftheMAPthresholdfromtheBPEXITcurve.The
RatherthanplottingxitselfitisconvenienttoplottheEXIT darkgrayareaisequaltothedesignrateofthecode.
value (1 (1 x)dr−1)dl. This is the locally best estimate
− −
of a bit based on the internal messages only, excluding the
Intheremainderofthissectionweaddressthefirstquestion,
directobservation.Forthischoicetheresultingcurveisusually
i.e., we will discuss a particularly effective choice of the
called the BP EXIT curve, see [106], [107] and [62, Sections
projection operator. In the next section we will address the
3.14 and 4.10]. It is the BP EXIT curve since the estimate
question of the existence and nature of this curve for the
is a BP estimate. And it is the BP EXIT (where the E stands
general case, presenting some partial results.
for “extrinsic”)curvesince the estimate excludesthe received
A good choice for the projection operator for general
value associated to this bit.
channels is the GEXIT functional [74]. For the BEC this
The FP equation is x=ǫ(1 (1 xdr−1))dl−1, which we coincideswiththeEXITfunctionalthatwesawinExample15.
− −
can solve for ǫ to get ForthegeneralcasetakeaFP(cσ,xσ)anddefiney=x(cid:24)σdr−1.
x Then
ǫ(x)= . (7)
(1−(1−xdr−1))dl−1 G(c ,y⊛dl)= ddσH(cσ⊛y⊛dl),
Using (7) we can write down the parametric characterization σ d H(c )
dσ σ
of the BP EXIT curve
where we think of y as fixed with respect to σ. In words,
(cid:16)(1−(1−xxdr−1))dl−1,(1−(1−x)dr−1)dl(cid:17). ByG⊛P(dcldσe,(ct·h)oedmienengat)sruovrpeeyrssuothsfetthhreeatcdioheaconifsgitoehneofocfehnaatnnrgyoepvyianroifaebnthltereonpcoyhdaoenfnuecnlσdc⊛er
This curve is shown in the left-hand side in Figure 1 for the σ
as a function of σ.
(3,6)-regular ensemble and has a typical C shape. In fact,
Discussion: Note that if the parameterization in σ is Lips-
one can show that, in this case, for ǫ < ǫBP(d ,d ) (the BP
l r chitz,i.e.,ifforsomepositiveconstantα, H(c ) H(c )
threshold) there is only one FP at x = 0 corresponding to ασ σ , then the derivative d H(c ) |existσs2al−mostσe1ve|ry≤-
pxe=rfe0ctadnedcothdeinogt;hfeorriǫs=theǫBFP(Pdlc,odrrr)esthpeornedianrge 2toFfPosr,woanrde iDsEa;t w|he2re−.Fu1r|ther,inthiscase alsoHdσ(cσ⊛σy⊛dl)isLipschitzand
hencedifferentiablealmosteverywhere.Thisfollowssince by
and for ǫ > ǫBP(d ,d ) there are exactly 3 FPs of DE, one
l r (the Duality Rule in) Lemma 6, for σ σ ,
of the FPs is at x=0 and the remaining two FPs are strictly 2 ≥ 1
positive,oneofwhichisstable,denotedbyxs(ǫ),whereasthe [H(cσ2 ⊛y⊛dl)−H(cσ1 ⊛y⊛dl)]
other is unstable, denoted by xu(ǫ). The stable FP is the FP +[H(c (cid:24)y⊛dl) H(c (cid:24)y⊛dl)]
which is reached by forward DE. For details see Lemma 59. σ2 − σ1
=[H(c ) H(c )] ασ σ ,
A quantity which will appear throughout this paper is the σ2 − σ1 ≤ | 2− 1|
value of the unstable FP when transmitting over BEC(ǫ = where the last step on the right-hand side assumes that the
1). We denote this FP by xu(1). More precisely, xu(1) is the parameterizationis such that H(cσ) increases in σ. The claim
smaller non-zerosolutionof x=(1 (1 x)dr−1)dl−1. Note follows since both terms on the left are non-negative (due
− −
that x (1) depends on the degrees, but we drop it from the to degradation), so that in particular the first term is upper
u
notation for ease of exposition. bounded by ασ σ , i.e., it is Lipschitz. This formulation
2 1
| − |
Discussion: The above example raises the following two alsoshowsthatthenumeratorisnolargerthanthedenominator
questions. (1) We have a large degree of freedom in selecting (so that the ratio exists) and that the GEXIT value is upper
the projection operator. Which one is “best”? (2) From the bounded by 1 (and is non-negative).
above example we see that the set of FPs forms a smooth We get the GEXIT curve by plotting (H(cσ),G(cσ,y⊛dl))
curve. Indeed, for the BEC it is not hard to see that the only for a family of FPs c ,x . This is shown in Figure 2 for
σ σ
{ }
FPs are the ones on the curve together with all the FPs of the (3,6)-regular ensemble assuming that transmission takes
the form(c ,∆ ), wherec is anyelementof the family of place over the BAWGNC. In the last section we have already
ǫ + ǫ
∞
BEC channelsand∆ correspondsto erasurevalue of0. Is explained how we can construct in the general case FPs by
+
this picture still valid f∞or general channel families? a numerical procedure. To plot Figure 2 we have used this
10
procedure to get a complete family of FPs for all entropies For a proof of the following see Lemma 4.77, [62].
from 0 to 1. In each of the two pictures of Figure 2 there is Lemma 16 (GEXIT for Smooth and Ordered Channels):
a small black dot. This dot marks a particular FP and the two For a smooth, ordered, channel family ch h, f(h,w),
{ }
smallinletsshowthecorrespondingdistributionofthechannel as a function of w, exists, is continuous, non-negative,
c as well as the message distribution emitted at the variable non-increasing and concave on its entire domain. Further
σ
nodes,call it x . For a detailed discussion we referthe reader f(h,0)=1 and f(h,1)=0.
σ
to [62], [74]. We remark that the above lemma also holds when ch is
{ }
piece-wise linear.
) x ) x
·, σ ·, σ
σ σ
c c
G( G( K. Existence of GEXIT Curve
c c
0.6 σ 0.6 σ As we briefly discussed above, for the BEC it is trivial to
see that the BP GEXIT curve indeed exists. But for general
0.4 0.4
BMS channels this is not immediate. The aim of this section
0.2 0.2 is to show the existence of the BP GEXIT curve for at least
a subset of parameters.
0.0 0.2 0.4 0.6 H(cσ) 0.0 0.2 0.4 0.6 H(cσ) Letusfirstrecallthefollowinglemmawhichwasstatedand
provedinaslightlyweakerformin[108].Fortheconvenience
of the reader we reproduce the proof in Appendix E.
Fig. 2. The BP GEXIT curve for the (dl = 3,dr = 6)-regular ensemble Lemma 17 (Sufficient Condition for Continuity): Assume
andtransmissionovertheBAWGNC.Eachpointonthecurvecorrespondsto
thatcommunicationtakesplace overan orderedand complete
aFP(cσ,xσ)ofDE.Thetwofigures show theFPdensity xas wellas the
inputdensitycσ fortwopoints onthecurves(seeinlets). family ch h, where h=H(ch), using the dd pair (dl,dr).
Then{,fo}r any h [0,1], there exists at mostone density xh
∈
Why do we use this particular representation? As we will so that (ch,xh) forms a FP which fulfills
discuss in detail in Section II-L, assuming this curve indeed
exists and is “smooth”, the area which is enclosed by it is B(ch)(dl 1)(dr 1)(1 B(xh)2)dr−2 <1. (9)
− − −
equal to r=1 d /d , the design rate of the ensemble.
− l r Furthermore,ifsuchadensityxh exists,thenitcoincideswith
ThisiseasytoseefortheBEC.Tosimplifynotation,denote
the forward DE FP. Finally, B(xh) is Lipschitz continuous
the GEXIT value in this case by G(ǫ,ydl), where ǫ is the with respect to B(ch). More precisely, if two FPs (ch ,xh )
erasure probability, x is the FP for this channel parameter, 1 1
xan)ddry−1=)d1l.−Le(1t −usxi)ndter−gr1a.teWtehethaerneahawvheicGh(ǫis,yednlc)lo=se(d1b−y(t1h−is Ban(dxh(ic)h22),dxrh−2)2s≤ati1sf−yδthefocrosnodmiteioδn>B(0c,hith)(ednl−1)(dr−1)(1−
curve.We call the correspondingintegralthe GEXIT integral. 1
B(xh ) B(xh ) B(ch ) B(ch ) . (10)
For our particular case it is given by | 1 − 2 |≤ δ | 1 − 2 |
1 The following lemma states that, at least for sufficiently
(1 (1 x)dr−1)dldǫ= (1 (1 x)dr−1)dlǫ′(x)dx largeentropies,theBPGEXITcurveindeedexistsandiswell
− − − −
Z Z0 behaved.
=ǫ(x)(1−(1−x)dr−1)dl|10+ Lemma 18 (Continuity For Large Entropies): Assume that
1 communication takes place over an ordered and complete
−dl(dr−1)Z0 ǫ(x)(1−x)dr−2(1−(1−x)dr−1)dl−1dx family {ch}h, where h = H(ch), using the dd pair (dl,dr).
1 Consider the set of FP pairs (ch,xh) obtained by applying
=1 dl(dr 1) x(1 x)dr−2dx forward DE to each channel c{h. Let }
− − −
Z0
=1+dlx(1−x)dr−1|10−dlZ01(1−x)dr−1dx=1− ddrl. ab((xx))==((1dl−(11)−2(dxr)dr−11))2dxl−(11, x)2(dr−2),
− − −
Perhapssurprisingly,theresultstaysvalidforgeneralchannels c(x)= x/a(x).
as we will discuss in Section II-L. This propertyis one of the
main ingredients in our proof. Let x˜ be the uniqpue solution in (0,1] of the equation
Note that given ch and zh, the GEXIT functional G(ch,zh)
can be expressed in the form zh(w)f(h,w)dw, where a(x)−b(x)=0. (11)
kfe(rhn,ewl)isisgcivaellnedbyasthe GEXITkernRel.In the|D|-domainthis hT˜(hdeln,dr, tchhe) =famhilByMS(c({x˜()c)h,,xsha)ti}s1hfi=e˜hs(dl(,9dr),,{cihs}),Lipscwhiitthz
{ }
1dch(z) (1+iz)(1+jw) (1 iz)(1 jw) continuous wrt to the Battacharyya parameter of the
dh 4 log2 1+(1−+iz)(1−+jw) dz. channel, where hBMS() is the function which maps the
Z0 (cid:16)i,jX=±1 (cid:16) (cid:17)(cid:17) Battacharyya constant ·of an element of the family to the
=k(z,w) corresponding entropy. Further, B(xh) xu(1) > 0 for all
| {z }(8) h≥h˜(dl,dr,{ch})3. ≥