Table Of ContentChapter 1
6.897 Algorithmi
Introdu
tion to Coding Theory September 5, 2001
Le
ture 1
Le
turer: Madhu Sudan S
ribe: Ryan O'Donnell
1.1 Error
orre
ting
odes and this
lass
This
ourse tea
hes the mathemati
s underlying obje
ts
alled error
orre
ting
odes (ECCs). We
won't de(cid:12)ne what they are today - for reasons to be
lari(cid:12)ed later. For now it will suÆ
e to know
that they are
ombinatorial obje
ts that possess
ertain extremal properties whi
h are very useful
for the
ommuni
ation and storage of information.
Modulothede(cid:12)nitionofECCs,we
anstilldis
ussthebigoutlineofthis
ourse. The
ontentsofthis
ourse
anbe divided into four roughlyequal parts. Our (cid:12)rst part will be explain someof the basi
onstru
tions of error-
orre
ting
odes. Next we will show \negative results" showing limitations,
or bounds, on the performan
e of
odes. The two parts above are essentially
ombinatorial in
nature, with few
omputational themes. We will get into the
omputational aspe
ts in the third
and fourth parts of this
ourse. The third part of the
ourse is where we fo
us on algorithmi
tasks
asso
iated with ECCs and some solutions to these tasks. Finally, in the fourth part we will dis
uss
some
onsequen
es one
an derive in
omputational
omplexity theory as a result of the existen
e
of ECCs and from the algorithmi
apabilities surrounding them.
1.1.1 Standard referen
es
Sin
e the subje
t is quite old, there are plenty of texts. Yet we won't follow any one of them. Here
are
omments on some of them.
1. The text by Ma
Williams and Sloane [74℄.
1-1
(a) This is possibly the most referen
ed text in CS?
(b) Unfortunately, it is getting outdated pretty fast.
(
) Has detailed
overageof (too) many families of
odes.
(d) Coverageof algorithms is not so good.
2. The text by van Lint [117℄.
(a) The book is mu
h more
on
ise and handy than Ma
Williams and Sloane. Easier to get
to some of the results here, if they are available.
(b) Still, the emphasis is more
ombinatorial than algorithmi
.
3. A text by Blahut [19℄.
(a) Book was targetted at engineers rather than mathemati
s, or so the author
laims.
(b) Yet the
overage is ex
ellent, espe
ially for insight, motivations, de(cid:12)nitions, even algo-
rithms
(
) The main drawba
kis that it is out of print and not easily available(not in MIT library,
e.g.).
4. The Handbook of Coding Theory [88℄.
(a) O(cid:11)ers extensive
overageof many re
ent themes.
(b) Sometimes ex
essive. (E.g. 130 pages of table of best known
odes.)
(
) But
ontains many interesting
hapters. E.g.,
hapter on algebrai
-geometry
odes, and
the
hapter on deep-spa
e appli
ations.
1.2 Information Theory
Even though this
ourse is on
oding theory, we start with a brief
overage of information theory.
Part of the reason is histori
. Coding theory was initiated by two seminal papers:
1. In 1948, Shannon wrote a detailed treatise on the mathemati
s behind
ommuni
ation [100℄.
2. In 1950, Hamming, motivated by the task of
orre
ting small number of errors on magneti
storagemedia, wrote the (cid:12)rst paper introdu
ing error-
orre
ting
odes [48℄.
In this le
ture we will dis
uss the main results from the Shannon paper, whi
h founded the theory
of information, while also
o-founding (with Hamming) the theory of error-
orre
ting
odes. The
theory of information provides the motivation for many questions in
oding theory and so we will
study this (cid:12)rst.
Shannon
onsidered the problem of two parties, say Ali
e and Bob, who wish to
ommuni
ate
digitally. In parti
ular, Ali
e wishes to send a message to Bob over a
ertain digital
hannel.
Shannon
onsidered both \noiseless"and \noisy"
hannels. We start by
onsidering the \noiseless"
ase.
1-2
1.2.1 Noiseless
oding
In this
ase, the
hannels perfe
tly transmits the bits Ali
e wants to send to Bob. However, we
might imagine that the
hannel is slow, so our goal is to
ompress the information we want to send
down to as few bits as possible. Then we send these bits a
ross the
hannel for Bob to re
eive and
de
ompress. We start with an example whi
h shows how to e(cid:11)e
t su
h a
ompression s
heme.
1.2.2 An example
Consider transmitting the
ontents of a pie
e of paper, whi
h
ontains some handwritten material
on it, by fax. If the paper has just a small amount of bla
k text on a white ba
kground, when it is
digitized it might
onsistof 99% 0's and 1% 1's. Let's say for the sakeof argumentthat we wantto
transmit amessagein whi
h ea
hbit is independently 0with probability .99and 1with probability
.01.
Considerthe followingen
odings
heme: Wesplitthemessageupintoblo
ksof10bits. Iftheblo
k
is 0000000000, we send the bit 0. If the blo
k is anything else, say x, we send the string 1x. Now
the expe
ted length of the en
oding of a blo
k is:
1(cid:1)Pr[blo
k is 0000000000℄+11(cid:1)Pr[blo
k is not 0000000000℄ = 11(cid:0)10q
10
where q :=Pr[blo
k is 0000000000℄=:99 (cid:21):9. Hen
e the expe
ted length of the en
oding of one
10 bit blo
k is at most 2 bits. Thus the original message has been
ompressed to within 20% of its
length! (Food for thought: Do we really need the probabilisti
model to a
heive this
ompression?)
Can we do better? This is exa
tly the question Shannon raised and answered with his theorem on
noiseless
oding.
Entropy
To analyze how mu
h a distribution
ould be
ompressed, Shannon introdu
ed some mathemati
al
de(cid:12)nitions asso
iated with information. The sour
e of information is modeled as a probability dis-
tribution,andamessageisarandomvariabledrawnfromthisdistribution. Thegoalof
ompression
is to en
ode the support of the probability spa
e (using, say binary strings) so as to minimize the
expe
ted length of the message. To any sour
e (or more
orre
tly, to any distribution), Shannon
assignedanon-negativerealnumber, thathe termed \entropy",thatmeasurestheinformation
on-
tent of the distribution. He then showed that this quantity (to within an additive term of one bit)
measures the
ompressibility of a bit.
The simplest possible distribution is a
oin (cid:13)ip, in whi
h heads has probability p and tails has
probability 1(cid:0)p. The entropy assigned to this is:
def 1 1
H(p)=plog2 +(1(cid:0)p)log2 :
p 1(cid:0)p
We
an generalize this notion to arbitrary distributions.
De(cid:12)nition 1.1 Let U be any (cid:12)nite set, and let D be a probability distribution on U, i.e., D :U !
P
[0;1℄with x2UD(x)=1. LetX bearandom variable distributeda
ording toD. Thentheentropy
1-3
1
of D is :
defX 1
H(D)= D(x)log2 :
D(x)
x2U
The noiseless
oding theorem of Shannon essentially says the following:
Theorem 1.2 For every (cid:12)nite set U and for every distribution D : U ! [0;1℄, There exists an
(cid:3) (cid:3)
en
oding fun
tion En
: U ! f0;1g and a de
oding fun
tion De
: f0;1g ! U su
h that for every
x2U, De
(En
(x)) =x, and
Expx D[jEn
(x)j℄2[H(D);H(D)+1℄;
where jsj denotes the length of a string s. Conversely, no error-free en
oding
an do better.
Shannon's a
tual theorem is often stated di(cid:11)erently, but we will state the theorem in the above
sin
eitis
learer. Wedon't provethe theoremhere,but themainidea fortheexisten
eresultisthe
0
following: (1)Roundall probabilities downsothat they be
omepowersof2(i.e.,
onstru
t D su
h
0 (cid:0)i 0
that for every x, D (x)=2 for some integer i, and D(x)=2<D (x)(cid:20)D(x). (2) Show that there
0 (cid:0)i
exists an en
oding En
that en
odes an element x, whose new probability D (x) equals 2 , with i
bits (so that no strings have the same en
oding). (3) Con
lude that this en
oding has an expe
ted
length in the desired range! The
onverse is harder to prove- we won't get into it.
An elegant algebrai
al identity
Theentropyfun
tionexhibitssomeveryni
emathemati
alproperties. Forexampleifadistribution
D de
omposes into two independent distributions D1 and D2 (i.e., U = U1(cid:2)U2, D1 : U1 ! [0;1℄
and D2 : U2 ! [0;1℄ and D(x;y) =D1(x)D2(y)), then H(D) =H(D1)+H(D2). This fa
t
an be
use to prove an interesting algebrai
identity, whi
h is otherwise quite tri
ky to prove.
Fromtheabovepropertyandtheentropyofabit,we(cid:12)ndthattheentropyinnindependentp-biased
oin (cid:13)ips is nH(p). On the other hand, this should be equal to the entropy of the distribution D
#10sinx #00sinx
given by D(x):=p (1(cid:0)p) . Using the se
ond formula, we
al
ulate this as:
Xn (cid:18)n(cid:19) 1
Dtlog2
t Dt
t=0
t n(cid:0)t
where Dt :=p (1(cid:0)p) . Thus we get
Xn (cid:18)n(cid:19) 1
Dtlog2 =nH(p):
t Dt
t=0
Entropy and its variants play an important rolein
ombinatori
sand probability. Some very useful
notionstokeeptra
kofinthisareaarethoseofmutualinformation,
onditionalentropy,andrelative
entropy!
To turn ba
k to our example, what is the optimal
ompressionfa
torfor the
ase of ourmessage to
be faxed, whose bits were 1 with probability 1%? The answer is H(:01) (cid:25) 8% | message
an be
ompressed to within H(:01) of their original length, if we know ones o
ur with probability about
1%.
1In le
ture, we used a randomvariable X drawna
ording to D and de(cid:12)ned this to be the entropy of X, rather
than D. In these notes we will swit
h to the more appropriate notation, whi
h de(cid:12)nes it to be a fun
tion of the
distributionandnotthevariable.
1-4
1.2.3 Noisy
hannels
For the purposes of this
ourse, the more interesting notion of a
hannel is the
lass of \noisy"
hannels
onsidered by Shannon | i.e.,
hannels that (cid:13)ip some of the bits sent a
ross them. The
problem here is for Ali
e to en
ode the message she wishes to send in su
h a way that even if the
hannel
orruptssomeof the bits in the en
oding, Bob will be able to de
ode the result into Ali
e's
original message, with high probability. Shannon
onsidered a large
lass of probabilisti
models
for noisy
hannels, and for ea
h proved the surprising theorem that you
ould always over
ome a
onstant error rate by sending an en
oded message that was longer by a
onstant fa
tor.
The general model Shannon gave for
hannels is as follows. There is an input alphabet (cid:6) and an
outputalphabet(cid:0)(bothusually(cid:12)nite). Thenwehaveabipartitegraphwith(cid:6)ontheleftand(cid:0)on
the right; ea
h edge ((cid:27);(cid:13)) is labeled with a probability of the
hannel
onverting a (cid:27) to a (cid:13). (The
hannel operatesindependently on ea
h
hara
ter.) Of
ourse,forea
h(cid:27) on theleft, the sum ofthe
labels on the edges tou
hing it must be 1.
We will illustrate his results for su
h
hannels only in
ases where (cid:6)(cid:18)(cid:0). (In su
h
ases, it is
lear
what an error is.)
Two
ommonly
onsidered
hannels:
1. Binary Symmetri
Channel | the
hannel
onsidered in the theorem. (cid:6) = (cid:0) = f0;1g, (0;0)
and (1;1) get probability 1(cid:0)p, and (0;1) and (1;0) get probability p.
2. Binary Erasure Channel | in this
hannel, bits don't get (cid:13)ipped | rather they get erased.
Spe
(cid:12)
ally, (cid:6)=f0;1g,(cid:0)=f0;1;?g,(0;0)and(1;1)getprobability1(cid:0)p,and(0;?)and(1;?)
get probability p.
Noisy
oding theorem
The noisy
oding theorem of Shannon is a powerful and general one. When spe
ialized to the
ase
of a binary symmetri
hannel with error probability p, we get the following result.
Theorem 1.3 For every p < 1=2, there exists a
onstant
< 1 and a pair of fun
tions E :
k
k
k k
f0;1g ! f0;1g , and D : f0;1g ! f0;1g with the following property: If we pi
k a message
k
uniformly at random from f0;1g , en
ode with E and then send the result a
ross the noisy
hannel,
2
and de
ode the result, then we re
over the original message with probability 1(cid:0)o(1).
Theorem 1.3 applies | with di(cid:11)erent
onstants | to the binary erasure
hannel as well.
Hamming notations
We need just a few de(cid:12)nitions before pro
eeding with the proof of the theorem.
n
De(cid:12)nition 1.4 If x and y are in (cid:6) , then the Hamming distan
e between them is (cid:1)(x;y):= # of
oordinates on whi
h x and y di(cid:11)er.
2Theo(1)termdependsonlyonk.
1-5
n
De(cid:12)nition 1.5 The Hamming ball of radius r
entered at y is B(y;r):=fx2(cid:6) :(cid:1)(x;y)(cid:20)rg.
n
De(cid:12)nition 1.6 Vol(r;n)denotesthevolumeof(any)radius-rballinf0;1g ;i.e., jB(y;r)j. Exa
tly,
Pr (cid:0)n(cid:1) (H(p)+o(1))n
thisquantityis i=0 i . Ifwe(cid:12)xsomep>0andletn!1thenwegetVol(pn;n)=2 .
Proof of Theorem 1.3
Proof The proof is highly non-
onstru
tive; it uses the probabilisti
method.
k n
Let n>k be de
ided upon later. Pi
k the en
oding fun
tion E : f0;1g ! f0;1g at random, i.e.,
k n
for every m2f0;1g , E(m) is
hosen uniformly at random from f0;1g , independently of all other
hoi
es.
n k n
The de
oding fun
tion D : f0;1g ! f0;1g works as follows. Given a string y 2 f0;1g , we (cid:12)nd
k
(non-
onstru
tively) the m 2 f0;1g su
h that (cid:1)(y;E(m)) is minimized. This m is the value of
D(y).
We now prove that D and E have the required property, with high probability, over the random
hoi
e of E.
k 0 0
Fix m2f0;1g asalsoE(m). (The rest ofthe proofwill usethe fa
tthat E(m) form 6=m is still
random.) the message being sent. Denote by y the
orrupted version of E(m) that is re
eived by
Bob. Let (cid:17) denote the error ve
tory(cid:0)E(m). Note that the (cid:17) is a random variablewith ea
hof its
oordinates being 1 w.p. p and 0 w.p. 1(cid:0)p, independent of other
oordinates.
Fix (cid:15) > 0. Let r = (p+(cid:15))n. In order for D(y) 6= m at least one of the following two events must
o
ur:
1. y62B(E(m);r) (i.e., too many errorso
urred in transmission.)
0 0
2. There exists some m 6=m su
h that E(m)2B(y:r). (The errors take the re
eived word too
lose to the en
oding of some other message.)
In parti
ular, if neither event o
urs, then m is the unique message su
h that E(m) is within a
distan
e of r from y and so D(y)=m.
We showthat foran appropriate
hoi
eofn, theevents abovehappen with low probability. Forthe
(cid:12)rst event to happen, it must be that (cid:17) has more than p+(cid:15) fra
tion of 1s. We
an apply Cherno(cid:11)
bounds (see appendix at the end of this le
ture) to see that:
(cid:0)((cid:15)2=2)n
Pr[y 62B(E(m);r))℄ (cid:20)2 :
(cid:17)
For any (cid:15)>0, we
an pi
k n to be large enough that the above quantity is as small as we want.
0
We now move onto the se
ond event. First (cid:12)x y and an m 6= m and
onsider the event that
0 0
E(m) 62 B(y;r). The probability of this event, taken over the random variable E(m), is exa
tly
n H(p)n
Vol(B(y;r))=2 . UsingtheapproximationVol(B(y;pn)(cid:25)2 andignoring(cid:15)(whi
hisarbitrarily
0
small), we (cid:12)nd that for every m 6=m.
0 H(p)n(cid:0)n
Pr[E(m)2B(y;r)℄ (cid:25)2 :
E
1-6
Using the union bound, we get that
0 0 k(cid:0)n+H(p)n
Pr[9m 6=m s.t. E(m)2B(y;r)℄(cid:25)2 :
E
1
Thus we (cid:12)nd that if
> , then k=n<1(cid:0)H(p) and thus the quantity aboveis less than one.
1(cid:0)H(p)
This gives the
hoi
e of
that we need.
Our proof is not yet
omplete! Why? Well, we have argued that a (cid:12)xed m is likely to be de
oded
orre
tly, but what about other messages? To
omplete the argument, we will a
tually need to lose
a little bit.
(cid:0)((cid:15)2=2)n k(cid:0)n+H(p)n
LetÆ =2 +2 ,denotethetotalprobabilityoferrorineitherofthetwostepsabove.
We have shown that for any (cid:12)xed m, for a random
hoi
e of E and the asso
iated D, the expe
ted
de
oding error probability is at most Æ. Thus we
an now
on
lude that this expe
tation
ontinues
k
to hold if we make m a random variable
hosen uniformly from f0;1g . We get
Expm;E;(cid:17)[D(E(m)+(cid:17))6=m℄(cid:20)Æ:
In parti
ular this implies that there exists an E su
h that for the
orresponding D, we have
Expm;(cid:17)[D(E(m)+(cid:17))6=m℄(cid:20)Æ:
This
on
ludes the proof of Shannon's theorem.
Capa
ity of the
hannel
Shannon's theorem above shows that if we are willing to slow down the e(cid:11)e
tive transmission rate
of the
hannel to 1=
< 1(cid:0)H(p), then we
an a
heive error-free
ommuni
ation with vanishingly
small probability. Furthermore this quantity 1(cid:0)H(p) is only afun
tion of the \noisy
hannel"and
not of the number of bits we wish to transmit over it. I.e., the e(cid:11)e
tive rate of transmission
an be
an absolute
onstant independent of n. Shannon
alled this quantity (the limiting rate of k=n, as
n!1) the
apa
ity of the noisy
hannel.
Forthe BinarySymmetri
Channel, how large
an its
apa
ity be? The proof aboveshows that the
apa
ity, denoted CBSC(p), is at least 1(cid:0)H(p). It is natural to ask if this quantity is an artifa
t of
the proof, orisit the
orre
t
apa
ityforthe
hannel. Shannon provedthat the latter wasthe
ase.
We will prove this in the next le
ture, but (cid:12)rst let us see why this is the
ase intuitively.
Suppose that we are a
tually is a setup there is a noiseless
hannel between Ali
e's lo
ation and
Bob's, but this
hannel has been \hija
ked" by Eve and Fred. Say Ali
e and Eve are in the same
physi
al lo
ation while Fred and Bob are at the other. To send a message over, Ali
e must hand
it over to Eve who then sends it through the
hannel (after some potential
orruption). Similarly
at the re
eiving end, Fred re
eives the message and hands it over to Bob. Suppose Eve and Fred
want to use this
hannel to ex
hangesome messagesof their own (at Ali
e & Bob's expense). They
do so by informing Ali
e and Bob that the
hannel is noisy with bits being (cid:13)ipped with probability
p. They advise Ali
e and Bob to use some en
oding/de
oding s
hemes. Ali
e and Bob agree on an
en
oding s
heme E and a de
oding s
heme D and in their naivette share these fun
tions with Eve
and Fred as well.
IntruthitmaybethatEvewishestousethe\noise"tosendsomemessagesofherowntoFred. Say
she has a message (cid:17) whi
h is a plain paper image, where ea
h pixel of the page is 1 independently
1-7
with probability p. The way she sends (cid:17) to Fred (at Ali
e's expense) is that when Ali
e gives her
a en
oded message E(m) to transmit, Eve sends over E(m)+(cid:17). Fred re
eives y = E(m)+(cid:17) (the
hannel does not introdu
e any noise) at the re
eiving end and passes it on, untampered, to Bob,
but also retains a
opy. As far as Ali
e and Bob are
on
erned, nothing maili
ious is o
uring -
with high probability D(y) = m and so they are ex
hanging messages at
apa
ity of the \noisy
hannel". But note the situation w.r.t. Eve and Fred. Fred also knows E and D and
an
ompute
(cid:17) =y(cid:0)E(D(y)) =E(m)+(cid:17)(cid:0)E(m), with high probability. So Eve and Fred are also ex
hanging
messages among themselves (with some small probability of ex
hanging in
orre
t messages). But
now if we
onsider Ali
e & Eve together at one end of the noiseless
hannel, and Bob & Fred
together at the other end, the parties are ex
hanging bits at a rate of at least C(p)+H(p) a
ross
the noiseless
hannel (assuming we believe the tightness of the noiseless
oding theorem). Sin
e we
are normalizingso that the rate the noiseless
hannel is 1, we get C(p)+H(p)<1! This is the link
between the noisy
ase and the noiseless
ase of the Shannon theorems.
Appendix
Notation used in the le
ture. We mention some notation that we used earlier or may use in
(cid:21)0 +
later le
tures. Zdenotes the set of integers, Z denotes the set of non-negative integers, and Z
the set of positive integers. R denote the set of reals, and Q the rationals. For real numbers a and
b, the notation [a;b℄ stands for the
losed interval from a to b, i.e., the set fx 2 Rja (cid:20) x (cid:20) bg,
while (a;b) is the open interval between a and b. For an integer k, we will use [k℄ to denote the
set f1;::: ;kg. If D is a distribution on the universe U, then X D denotes a random variable X
drawn from U a
ording to the distribution D. For an event E (cid:18) U, the quantity PrX D[X 2 E℄
denotes the probability of the event E when X is
hosen a
ording to D. For a real-valued fun
tion
f : U ! R, the quantity ExpX D[f(X)℄ denotes the expe
ted value of f(X) when X is
hosen
a
ording to D. When the distribution D is
learfrom
ontext, we may abbreviatethese quantities
toPr[E℄andExpX[f(X)℄. SimilarlyVarX D[f(X)℄denotesthevarian
eoff(X)(i.e.,Var(f(X))=
2 2
Exp[(f(X)) ℄(cid:0)Exp[f(X)℄ ).
Somebasi
probabilityfa
ts Hereisaqui
kre
apofbasi
fa
tsonprobabilityandexpe
tations.
Probability One of the most used fa
ts on probability is the union bound: Pr[E1[E2℄(cid:20)Pr[E1℄+
Pr[E2℄: Note that the bound makes no \independen
e" assumptions.
Expe
tations Analogous to the above we have: Exp[X1+X2℄ = Exp[X1℄+Exp[X2℄: Note that
this is an equality! If random variables are independent, then we get a produ
t relationship
Exp[X1X2℄=Exp[X1℄Exp[X2℄.
Converting probabilities to expe
tations: Sin
e expe
tations are more amenable to algebrai
manipulations, it is often useful to
onvert statements on probability to statements of events.
The standard way to do this is to use \indi
ator variables". For an event E, let IE be the
0=1-valued variable given by IE(X) = 1 if X 2 E and IE(X) = 0 otherwise. Then we have
ExpX[IE(X)℄=Pr[E℄.
Converting expe
tations to probabilities: Sin
eprobabilitiesarethequantitiesthathavemore
intuitivemeaning,thesearethemorestandardtargetsofourinvestigation. Sin
eexpe
tations
(cid:12)gure in proofs, we would like to (cid:12)nd ways to
onvert statements on expe
tations ba
k into
probability statements. This
onversion is not standard. Several \tail inequalities" are used
to a
heivethis
onversion,e.g., Markov's,Cheby
hev's,and Cherno(cid:11)'s: We statethem below:
1-8
Markov's inequality For a non-negative random variable X and positive real (cid:11), then
E[X℄
Pr[X (cid:21)(cid:11)℄(cid:20) .
(cid:11)
Cheby
hev's inequality Initsgeneralform,thisinequalityisjustanappli
ationofMarkov's
2
inequality to the random variable Y , for arbitrary Y. In the general form, it is quite
hard to see its strength, so we give a spe
ial form.
Pn
LetY = i=1Yi,wheretheYi'sidenti
allydistributedrandomvariablesthatarepairwise
2
(but not fully) independent. Let Exp[Yi℄=(cid:22) and Var[Yi℄=(cid:27) . Then for (cid:21)>0, we have
(cid:27)2
Pr[Y (cid:21)((cid:22)+(cid:21))n℄(cid:20) n(cid:21)2.
Cherno(cid:11) bounds If Y1;::: ;Yn are
ompletely independent it is possible to get stronger
bounds on the probability that their sum deviates mu
h from their expe
tation. We
onsider the spe
ial
ase of variables taking values in the interval [0;1℄
LetY1;::: ;Yn beindependentandidenti
allydistributedrandomvariablestakingvalues
P (cid:0)((cid:21)2=2)n
in the interval [0;1℄ with mean (cid:22). Let Y = iYi. Then Pr[Y (cid:21) ((cid:22)+(cid:21))n℄ (cid:20) e ,
where e is the base of the natural logarithm.
For further elaboration on probabilities and expe
tations one may
onsult the text on Randomized
Algorithms [82℄.
1-9
Chapter 2
6.897 Algorithmi
Introdu
tion to Coding Theory September 10, 2001
Le
ture 2
Le
turer: Madhu Sudan S
ribe: Omprakash Gnawali
Today we will
over the following topi
s:
(cid:15) Converse to Shannon's
oding theorem.
(cid:15) Some remarks on Shannon's
oding theorem.
(cid:15) Error
orre
ting
odes.
(cid:15) Linear
odes.
2.1 Converse to Shannon's
oding theorem
Re
all that the Binary Symmetri
Channel (BSC) with parameter p is the
hannel that transmits
bits, (cid:13)ipping ea
h transmitted bit with probability p independent of all other events. Lets start by
re
alling Shannon's
oding theorem informally.
Over the BSC with parameter p, it is possible to transmit information at any rate less
than 1(cid:0)H(p).
(To give a sense of how to make the above formal, here is the formal version in all its quanti(cid:12)ed
1
glory: \For every p < 2 and (cid:15);Æ > 0, there exists an n0 < 1 su
h that for every n (cid:21) n0 and
k n n k
k (cid:20)(1(cid:0)H(p)(cid:0)(cid:15))n, there exist fun
tions E :f0;1g !f0;1g and D :f0;1g !f0;1g su
h that
Pr [D(E(m)+(cid:17))=m℄(cid:21)1(cid:0)Æ;
(cid:17) Dp;n;m Uk
k
where Uk is the uniform distribution on f0;1g and Dp;n is the distribution on n bits
hosen inde-
pendently with ea
h bit being 1 with probability p.")
We will now proof a
onverse to this theorem.
2-10
