Table Of ContentAn LDPCC decoding algorithm
based on Bowman-Levin approximation
—Comparison with BP and CCCP—
Masato Inoue, Miho Komiya and Yoshiyuki Kabashima
5
Department of Computational Intelligence and Systems Science,
0
Interdisciplinary Graduate School of Science and Engineering,
0
2 Tokyo Institute of Technology, Yokohama 226-8502, Japan
Email: [email protected], [email protected], and [email protected]
n
a
J
8
2 Abstract—Belief propagation (BP) and the concave convex whereµ=1,...,M denotestheparityindexandµdenotesthe
procedure (CCCP) are both methods that utilize the Bethe set of node indices involved in the µ-th parity. Similarly, l =
] free energy as a cost function and solve information processing 1,...,N denotes the bit index and l denotes the set of parity
h tasks. We have developed a new algorithm that also uses the
indices linking to the l-th bit. |µ| and |l| denote the degree
c Bethe free energy, but changes the roles of the master variables
e and the slave variables. This is called the Bowman-Levin (BL) of µ-th parity and the l-th bit, respectively. The proportion
m approximation in the domain of statistical physics. When we means the normalization of a probability function – i.e., the
- applied the BL algorithm to decode the Gallager ensemble of summation of the probability for all possible arguments x –
t short-length regular low-density parity check codes (LDPCC)
a should be 1.
over an additive white Gaussian noise (AWGN) channel, its
st average performance was somewhat better than that of either We consider a noisy channel with additive white Gaussian
t. BP or CCCP. This implies that the BL algorithm can also be noise (AWGN);i.e., the probabilityofthe receivedcodesy is
a successfully applied to other problems to which BP or CCCP defined as
m
has already been applied.
- N (y −x)2
nd I. INTRODUCTION P(y|x)∝ exp − l2σ2l , (2)
l (cid:18) (cid:19)
o Recently, various statistical inference algorithms have be- Y
c come of interest in the field of large-scale information pro- where σ2 denotes the variance of the noise. The posterior
[
cessing. Belief propagation (BP) [1] and the concave convex probability of the sent code can then be expressed as
2 procedure (CCCP) [2] are among the most effective of the
v methodswhichminimizetheBethefreeenergy[3],[4].Inthe M N y
29 field of practical application (e.g., the problem of decoding P(x|y)∝ 1+ xl" exp xlσ2l #. (3)
5 low-density parity check code (LDPCC) [5], [6]), BP and Yµ lY∈µ Yl (cid:16) (cid:17)
0 CCCP have both been successfully applied [7].
To infer the sent code x by y, we employ the maximum
1 However, they are not the only methods that minimize the
posterior marginal (MPM) solution,
4 Bethe free energy. In this paper, we focus on the method
0
of Lagrange undetermined multipliers used by both BP and
/ xˆ =argmax P(x|y), (4)
mat CofCmCaPs,tearndvadriearbivleesaanndewslaavlegovraitrhiamblebsy. Texhcishaanpgpirnogacthh,ecarollleeds l xl Xx\l
Bowman-Levin(BL) approximation[8], is sometimesused in which minimizes the bit error rate. On the other hand, the
-
d thefield ofstatisticalphysicsasa waytofindanextremum(a maximum a posteriori (MAP) solution minimizes the block
n saddle, local minimum, or local maximum) of the Bethe free error rate,
o
energy.
c
: xˆ =argmaxP(x|y), (5)
v II. LOWDENSITY PARITY CHECK CODE (LDPCC) x
i
X The LDPCC decoding problem can be handled within butis generallydifficulttodeterminebecauseofthe exponen-
r a Bayesian framework. The prior probability of the codes, tial calculation cost.
a
consisting of N binary bits (x∈{+1,−1}N), is defined as
III. BETHEFREE ENERGY
M
P(x)∝ 1+ x , (1) One purpose of the Bethe free energy approach is to
l
µ l∈µ determineasetofmarginalprobabilitiesofagivenprobability,
Y Y
(a) parity checks wehereattempttodecodetheLDPCCbyminimizingF under
the consistency condition (10) expecting that {q(x )} well
l l
approximate the marginal probabilities even in the case that
bits
the parity connection does not have the tree structure.
(b) parity checks
IV. LAGRANGE MULTIPLIERS
bits
TominimizetheBethefreeenergyundertheconstraint(10),
weintroduceLagrangeundeterminedmultipliers,λ (x ).The
Fig.1. Examplesoftheparityconnection. (a)Treestructure and(b)loopy µl l
structure. objective function to minimize is
G({b },{q},{λ })≡F +L, (13)
µ l µl
whichprovidestheMPMsolutionhere.TheBethefreeenergy,
F, is defined using Kullback-Leibler (KL) divergence as where
M
M
F ≡ KL(b (x )||φ (x )) (6)
µ µ µ µ L≡ λ (x ) b (x )−q(x ) , (14)
µl l µ µ l l
µ
XN Xµ Xl∈µXxl xXµ\l
+ (1−|l|)KL(ql(xl)||ψl(xl)), and we solve the following three equations:
l
X
∂G b (x )
where P(x|y) can be represented as 0 = =ln µ µ +1+ λ (x ), (15)
µl l
∂b (x ) φ (x )
µ µ µ µ
M N l∈µ
X
P(x|y) ∝ φµ(xµ) ψl(xl)1−|l| , (7) ∂G ql(xl)
"Yµ #"Yl # 0 = ∂ql(xl) =(1−|l|)(cid:18)lnψl(xl) +1(cid:19)−µ∈lλµl(xl),
N X
(16)
φ (x ) ∝ 1+ x ψ (x ) , (8)
µ µ lY∈µ l"Yl l l # 0 = ∂λ∂G(x ) = bµ(xµ)−ql(xl). (17)
ψl(xl) ∝ exp xlσy2l, (9) µl l xXµ\l
(cid:16) (cid:17) Using x ∈ {+1,−1} and the normalization conditions of
using (normalized) probability functions, φ (x ) and ψ (x ) l
µ µ l l
the probability functions, we can reduce q and λ to linear
in the current case. The introduced test probability functions l µl
functions as
ofcreeks,{b (x )},andbits,{q(x )},arerequiredtosatisfy
µ µ l l
the consistency of the marginal probabilities: 1+x tanhh
q (x ) = l l, (18)
l l
2
b (x )=q(x ) (l ∈µ). (10)
µ µ l l y
λ (x ) = −x h − l +η . (19)
xXµ\l µl l l µl σ2 µl
{b (x )} and {q (x )} that minimize F are expected to (cid:16) (cid:17)
µ µ l l We also sometimes use m ≡ hx i = tanhh . Using
approximate the marginal probabilities of P(x|y). l l ql(xl) l
these expressions, we can reduce Eqs. (15) - (17) to
The Bethe free energy approach gives the exact marginal
probabilitieswhen the parity connectionhas the tree structure
N
(Fig. 1(a)). In such cases, any probability of x can be ex- b (x ) ∝ 1+ x exp(x h ) , (20)
µ µ l l µl
pressed as a product of its marginal probabilities, {bµ(xµ)} l∈µ " l #
and {q(x )}, as Y Y
l l
1 y
M N h = h − l , (21)
Q(x) = b (x ) q(x )1−|l| . (11) l |l|−1 µ′l σ2
µ µ l l µ′∈l
" #" # X
Then,theBethefreeeYµnergycoincidYelswiththeKLdivergence m = xlxl xµ\lbµ(xµ), (22)
l
b (x )
between the test and the true probabilities, P xl Pxµ\l µ µ
F =KL(Q(x)||P(x|y)), (12) respectively. From EqPs. (20P) and (22), we obtain
which implies that minimizing the Bethe free energy leads h = h +atanh tanhh (23)
l µl µl′
to the correct probability Q(x) = P(x|y), and, therefore,
l′∈µ\l
Y
the exact MPM solution can be assessed from the obtained
{q(x )}.Unfortunately,theBethe free energydoesnotrepre- Now,wehavetwotypesofvariable:{h} and{h },andtwo
l l l µl
senttheKLdivergenceforloopygraphs(Fig.1(b)).However, types of simultaneous equation: (21) and (23).
V. BELIEFPROPAGATION (BP) the master variables. After the convergenceof the inner loop,
BP considersthedouble-indexedh,{hµl},to bethe master the outer loop is performed to determine htl+1.
variables. Specifically, from Eqs. (21) and (23), we obtain
1 y
hµ′l+atanh tanhhµ′l′ inner loop: hµl ← 2σ2l + (htl −hµ′l)
l′∈Yµ′\l µ′X∈l\µ
= |l|1−1 hµ′′l− σy2l (24) +htl −atanh tanhhµl′,(30)
µX′′∈l l′Y∈µ\l
for any {l,µ′ ∈l}. BP ingeniously rearranges the leftside of outer loop: htl+1 ← σy2l + (htl −hµ′l). (31)
this equation with the average without µ: µ′∈l
X
VII. BOWMAN-LEVIN (BL)
1
h +atanh tanhh
|l|−1 µ′l µ′l′ BL considers the single-indexed h, {hl}, to be the master
µ′X∈l\µ l′∈Yµ′\l variables. Specifically, BL determines {hµl} by first using
{h } in Eq. (23). It requires some iteration to be solved,
1 y l
= |l|−1 hµ′′l− σ2l (25) resulting in an inner loop:
µ′′∈l
We then obtain the iterative substitutionXto converge {h }. inner loop: hµl ←htl −atanh tanhhµl′. (32)
µl
l′Y∈µ\l
y
loop: h ← l + atanh tanhh . (26)
µl σ2 µ′l′ Because the determination of the slave variables, {hµl}, de-
µ′X∈l\µ l′∈Yµ′\l pends on the provisionalvalues of the master variables, {hl},
Oncethemastervariablesaredetermined,wecaneasilyobtain BL also needs a double-loop algorithm. Eq. (21) implies that
the slave variables, {h }, by update
l
y
l
result: h ← + atanh tanhh . (27)
l σ2 µ′l′ outer loop: ht+1 ← 1 h − yl , (33)
µX′∈l l′∈Yµ′\l l |l|−1 µ′l σ2
µ′∈l
To lower the calculation cost, we check whether the esti- X
mated sent code, xˆ ≡ signh , satisfies all parities for every may be employed for the outer-loop using the converged
l l
loop of Eq. (26). We stop the iteration loop if we reach any variables {h }.
µl
codeword, or the number of loops reaches an upper limit. Eq. (33), however, does not provides satisfactory results
as this empirically increases the Bethe free energy. This is
VI. CONCAVE CONVEX PROCEDURE (CCCP)
because the outer loop (33) is interpreted as
CCCP is a double loop algorithm utilizing convex opti-
mization. The convexityof the Bethe free energy is generally ht+1 ←ht+κ∂Gt. (34)
not guaranteed because of the negative coefficient, 1−|l|, of l l ∂hl
the second term in Eq. (6). So, CCCP employs the following
additional term at every outer loop step t. where κ ≡ cos|hl|−−21htl is positive. Gt denotes G, which is
regarded as a function of only {h} at outer-loop step t.
l
N
F˜t ≡ F + |l|KL(q (x )||qt(x )) (28) Inordertoresolvethisdifficulty,weusethenaturalgradient
l l l l descent method [9] instead of Eq. (33) as
l
X
M N ∂Gt
= KL(bµ(xµ)||φµ(xµ))+ KL(ql(xl)||ψl(xl)) outer loop: ht+1 ←htl −kH−1 ∂h , (35)
µ l
X X where k denotes a small positive step width, and H denotes
N
ψ (x )
+ |l|q (x )ln l l . (29) the Fisher information matrix defined as
l l qt(x )
l l l ∂logQ(x)∂logQ(x)
X H ≡ (36)
Equation (29) guarantees the convexity of F˜t({bµ},{ql}), i,j (cid:28) ∂hi ∂hj (cid:29)Q(x)
because KL divergencefunctionis convex,and the third term cosh−2h (i=j)
is a linear function. Besides, F necessarily decreases if F˜t = i (37)
0 (i6=j)
decreasesbecausetheadditionalterm isnon-negative,andthe (cid:26)
additional term itself disappears if {q} converges. assuming the following approximation:
l
Intheinnerloop,similartoBP,CCCPconsidersthedouble-
N
indexed h, {hµl}, to be the master variables. On the other Q(x)≃ q (x ). (38)
l l
hand, in the outer loop, single-indexedh, {h }, are treated as
l l
Y
0
10 BP 0 BP
10
CCCP CCCP
BL BL
−1
10
e correctly decoded
at −1
or r10−2 ate10
err r r
block/bit 10−3 block erro10−2 still in progress
−4
10
wrongly decoded
−3
−5 10
10
1 2 3 4 100 101 102 103 104
E /N [dB]
b 0 # of outer loops
Fig. 2. Block (upper three lines) / bit (lower three lines) error rates of
Fig. 3. Time evolution (outer loop steps) of the not-correctly-decoded
BP, CCCP, and BL algorithms. Configurations were as follows: code length
rates(upperthreelines)andwrongly-decodedrates(lowerthreelines)forBP,
N = 486, number of parities M = 243, degree of parity |µ| = 6,
CCCP,andBLalgorithms. Thedatacorresponds tothecaseofEb/N0=2
degree of bit |l| = 3 ((3,6)-regular LDPCC), limit of outer loops: 10000
inFig.2.
(the limit of loops in the case of BP), number of inner loops fixed as
6 for CCCP and BL, and step width k = 0.3 in BL. Eb/N0[dB] is
definedas10log10(1/(2σ2(N−M)/N)). Thenumberofcommunications
were 103,3×103,104,3×104,105,3×105, and 106 for Eb/N0 = cases. In the early outer-loop steps, BP tended to reach the
1.0,1.5,...,4.0, respectively. Each error bar denotes a 99% confidence
interval basedonabinomialdistribution. correct codeword first, and then CCCP and BL followed. In
the later steps, BL continuously improved the block error
rate, while CCCP had little effect after about the 500-th
We then obtain
step. The effect of BP was intermediate. The rates for the
outer loop: ht+1 ←ht−k∂Gt, (39) wrongly-decodedcase were almost the same among the three
l l ∂ml algorithms. These results suggest that the BL algorithm will
where outperformBP and CCCP if we can afford a high calculation
∂Gt y cost – for example, 1000 outer-loop steps.
∂m = hµl− σ2l −(|l|−1)htl. (40) The calculation cost is roughly proportional to the number
l
Xµ∈l ofinnerloops(weregardthatofBPtobe1).So,ifwesetthe
VIII. VALIDATION numberofinnerloopsas6forCCCPandBL,thecostratioof
We validated the performance of the BL algorithm by BP, CCCP, and BL will be about 1:6:6. If we consider the
comparingit with that of BP and CCCP througha simulation average number of outer loops, the difference could become
of the Gallager ensemble of the short-length regular LDPCC. larger (e.g., 1 : 10 : 12), but this depends on the upper limit
As the decoding performance greatly depends on the parity on the number of outer loops.
checkmatrix,thesimulationwasdoneoveranLDPCCensem- Parallelizationisalsoanimportantfactorregardingcalcula-
ble; that is, we remade the matrix for every communication tioncost.Briefly,parallelizationoftheBPloopispossible.Itis
accordingto the Gallager’sconstruction[5]. We assumed that also possibleforthe outerloopsofCCCP and BL, butnotfor
the decoder knows the true noise variance of the AWGN theinnerloopofCCCP. Ontheotherhand,itisindispensable
channel,σ2.Inthesimulation,BLperformedsomewhatbetter for the inner loop of BL to achieve fast convergence.
than both BP and CCCP. Parameter optimization of the three algorithms is a real
Figure 2 shows the block and bit error rates of each problem. In the case of BP, we have to determine only the
algorithm over various signal-to-noise ratios (E /N ). BL upper limit of the outer loops. For CCCP, we also have to
b 0
performed better than BP and CCCP, especially in the area determine the number of inner loops. For BL, in addition to
where E /N was around 2 dB. The error floor appeared in the CCCP parameters,we haveto determinethe step widthof
b 0
the area where E /N was greater than about 2.5 dB. This the natural gradient descent. Empirically, the configuration of
b 0
errorfloorprobablyoccurredduetotheshortloopoftheparity the step width appears rather robust since the simulated BL
check matrix. performance generally exceeded that of the other algorithms
Figure 3 shows the time (outer-loop step) evolution of the (Fig. 2) even though they shared a common step width
rate for both the not-correctly-decoded and wrongly-decoded configuration (i.e., k =0.3).
The optimization of the parity check matrix is also a
problem, especially for short-length LDPCC. We will further
investigate the dependence of these algorithms on the matrix
in our future work.
IX. CONCLUSION
The method we have proposed minimizes the Bethe free
energybased on the Bowman-Levin(BL) approximation.The
BL algorithm combined with the natural gradient descent
method successfully converges. We have compared our BL
algorithm to the belief propagation (BP) and concave convex
procedure (CCCP) algorithms with respect to the decoding
problemoftheGallagerensembleofshort-lengthregularlow-
density parity check codes (LDPCC) over an additive white
Gaussian noise (AWGN) channel. Simulation showed that the
BL algorithm outperformed the BP and CCCP algorithms,
although the BL calculation cost was greater. This suggests
that the BL algorithm can be successfully applied to other
problems to which BP or CCCP have already been applied.
ACKNOWLEDGEMENTS
This work was partially supported by a Grant-in-Aid for
Scientific Research on Priority Areas No. 14084206.
REFERENCES
[1] J. Pearl, Probabilistic Reasoning in Intelligent Systems, Morgan Kauf-
mann,1988.
[2] A.L.Yuille, NeuralComput.,14,1691-1722, 2002.
[3] Y.Kabashima andD.Saad,Europhys.Lett.,44,668-674,1998.
[4] J.S. Yedidia, W.T.Freeman and Y. Weiss, Understanding Belief Propa-
gation and its Generalization, http://www.merl.com, MITSUBISHI RE-
SEACHLABORATORIES.
[5] R.G.Gallager, Lowdensityparitycheckcodes,inResearchMonograph
series,Cambridge, MITPress,1963.
[6] D.J.C.MacKay,IEEETrans.Inform.Theory,IT-45(2),399-431,1999.
[7] T.ShibuyaandK.Sakaniwa,IEICETrans.onFundamentals,E86-A(10),
2601-2606, 2003.
[8] D.R.BowmanandK.Levin,Phys.Rev.B,25,3438,1982.
[9] S.Amari:NeuralComput.10,251-276,1998.
