Table Of ContentERGEBNISSE DER MATHEMATIK
UND IHRER GRENZGEBIETE
UNTER MITWIRKUNG DER SCHRIFTLEITUNG DES
"ZENTRALBLATT FüR MATHEMATIK"
HERAUSGEGEBEN VON
P. R. HALMOS· R. REMMERT . B. SZÖKEFALVI-NAGY
UNTER MITWIRKUNG VON
L. V. AHLFORS . R.BAER . F.L.BAUER . R.COURANT . A. DOLD
j.L.DOOB· S.EILENBERG· M.KNESER· H.RADEMACHER
B. SEGRE . E. SPERNER
REDAKTION P. R. HALMOS
======
NEUE FOLGE· BAND 31
REIHE:
WAHRSCHEINLICHKEITSTHEORIE
UND MATHEMATISCHE STATISTIK
BESORGT
VON
J.
L. DOOB
SPRINGER-VE RLAG
BERLIN . GÖTTINGEN . HEIDELBERG . NEW YORK
1964
CODING THEOREMS
OF INFORMATION THEORY
BY
J. WOLFOWITZ
PROFESSOR OF MATHEMATICS, CORNELL UNIVERSn'y
SECOND EDITION
SPRINGER-VERLAG
BERLIN . GÖTTINGEN . HEIDELBERG . NEW YORK
1964
Alle Rechte,
insbesondere das der Übersetzung in fremde Sprachen,
vorbehalten
Ohne ausdrückliche Genehmigung des Verlages
ist es auch nicht gestattet, dieses Buch oder Teile- daraus
auf photomechanischem Wege (Photokopie, Mikrokopie) oder auf andere Art
zu vervielfältigen
ISBN 978-3-662-00239-1 ISBN 978-3-662-00237-7 (eBook)
DOI 10.1007/978-3-662-00237-7
©
by Springer-Verlag, Berlin . Göttingen - Heidelberg 1961 und 1964
Library 01 Congress Catalog Card Number 64-23107
Softcover reprint of the hardcover 2nd edition 1964
Titel Nr.4575
DEDICATED TO THE MEMORY OF
ABRAHAM WALD
Preface to the Second Edition
The imminent exhaustion of the first printing of this monograph
and the kind willingness of the publishers have presented me with the
opportunity to correct a few minor misprints and to make a number of
additions to the first edition. Some of these additions are in the form of
remarks scattered throughout the monograph. The principal additions are
Chapter 11, most of Section 6.6 (inc1uding Theorem 6.6.2), Sections 6.7,
7.7, and 4.9. It has been impossible to inc1ude all the novel and inter
esting results which have appeared in the last three years. I hope to
inc1ude these in a new edition or a new monograph, to be written in
a few years when the main new currents of research are more clearly
visible.
There are now several instances where, in the first edition, only a
weak converse was proved, and, in the present edition, the proof of a
strong converse is given. Where the proof of the weaker theorem em
ploys a method of general application and interest it has been retained
and is given along with the proof of the stronger result. This is wholly
in accord with the purpose of the present monograph, which is not only
to prove the principal coding theorems but also, while doing so, to
acquaint the reader with the most fruitful and interesting ideas and
methods used in the theory.
I am indebted to Dr. SAMUEL KOTZ for devoted and valuable help
in checking my revisions and for constructing the index, and to Pro
fessor H. P. BEARD for intelligent and conscientious reading of proofs.
As earlier, I am grateful to the Air Force Office of Scientific Research
and the Office of Naval Research for continned support; it is a pleasure
to add the name of R. J. LUNDEGARD to the earlier list of their staff
members to whom my thanks are due.
Comell University, July, 1964 J. WOLFOWITZ
Preface to the First Edition
This monograph originated with a course of lectures on information
theory which I gave at comen University du ring the academic year
1958-1959. It has no pretensions to exhaustiveness, and, indeed, no
pretensions at an. Its purpose is to provide, for mathematicians of some
maturity, an easy introducton to the ideas and principal known theorems
of a certain body of co ding theory. This purpose will be amply achieved if
the reader is enabled, through his reading, to read the (sometimes
obscurely written) literature and to obtain results of his own. The theory
is obviously in a rapid stage of development; even while this monograph
was in manuscript several of its readers obtained important new results.
The first chapter is introductory and the subject matter of the
monograph is described at the end of the chapter. There does not seem
to be a uniquely determined logicalorder in which the material should
be arranged. In determining the final arrangement I tried to obtain an
order which makes reading easy and yet is not illogical. I can only hope
that the resultant compromises do not eam me the criticism that I
failed on both counts. There are a very few instances in the monograph
where astated theorem is proved by a method which is based on a result
proved only later. This happens where the theorem fits in one place for
completeness and the method of proof is based on a theorem which fits
in elsewhere. In such cases the proof is given with the theorem, and the
reader, duly wamed, can come back to the proof later. This procedure,
which occurs very rarely, will surely cause the reader no trouble and can
be blamed on the compromises described above.
This monograph certainly contains many errors. However, I hope
that the errors still left are not so great as to hinder easy reading and
comprehension.
My gratitude is due to several friends and 'colleagues: HELEN P.
BEARD, L. J. COTE, J. H. B. KEMPERMAN, and J. KIEFER. They read
the manuscript or parts of it and made suggestions which were always
valued though not always accepted. The Air Force Office of Scientific
Research and the Office of Naval Research subsidized my work at various
times and I am grateful to them; M. M. ANDREw, DOROTHY M. GILFORD,
R. G. POHRER, and O. A. SHAW of their staffs have always been coopera
tive and helpful. The invitation to publish in the Ergebnisse series came
from J. L. DOOB, to whom my thanks are due.
comen University, September. 1960 J. W OLFOWITZ
Contents
1. Heuristic Introduction to the Discrete Memoryless Channel. 1
2. Combinatorial Preliminaries . . . . . 6
2.1. Generated sequences . . . . . . 6
2.2. Properties of the entropy function 10
Remarks . . . . . . . . 14
3. The Discrete Memoryless Channel 14
3.1. Description of the channel 14
3.2. A coding theorem . . . . . 17
3.3. The strong converse . . . . 19
3.4. Strong converse for the binary symmetric channel 21
3.5. The finite-state channel with state calculable by both sender
and receiver . . . . . . . . . . . . . . . . . . . . . 22
3.6. The finite-state channel with state calculable only by the sen-
der. . . . . . 27
Remarks . . 30
4. Compound Channels 33
4.1. Introduction 33
4.2. The canonical channel 35
4.3. A coding theorem 36
4.4. Strong converse. . . 40
4.5. Compound d.m.c. with c.p.f. known only to the receiver or
only to the sender . . . . . . . . . . . . . . . . . . . 40
4.6. Channels where the c.p.f. for each letter is stochastically deter-
mined . . . . . . . . . 43
4.7. Proof of Theorem 4.6.4. . . . . . . . . . 47
4.8. The d.m.c. with feedback. . . . . . . . . 49
4.9. Strong converse for the d.m.c. with feedback 51
Remarks . . . . . . . . . . . . . . . 54
5. The Discrete Finite-Memory Channel . . 55
5.1. The discrete channel . . . . . . . 55
5.2. The discrete finite-memory channel 58
5.3. The coding theorem for the d.f.m.c. 59
5.4. Strong converse of the coding theorem for the d.f.m.c. 60
5.5. Rapidity of approach to C in the d.f.m.c. 61
5.6. Discussion of the d.f.m.c. . . . . 62
Remarks . . . . . . . . . . 64
6. Discrete Channels with a Past History . 64
6.1. Preliminary discussion . . . . . 64
6.2. Channels with a past history . . 66
6.3. Applicability of the coding theorems of Section 7.2 to channels
with a past history . . . . . . . . . . . . . . . . . . 69
x
Contents
6.4 .. A channel with infinite duration of memory of previously
transmitted letters. . . 69
6.5. A channel with infinite duration of memory of previously
received letters . ... . 75
6.6. Indecomposable channels . 77
6.7. The power of the memory. 88
Remarks ...... . 89
7. General Discrete Channels . . . 90
7.1. Alternative description of the general discrete channel 90
7.2. The method of maximal codes 91
7.3. The method of random codes 96
7.4. Weak converses . . . . . 99
7.5. Digression on the d.m.c. 102
7.6. Discussion of the foregoing . 105
7.7. Channels without a capacity 107
Remarks . . . . . . . . 109
8. The Semi-Continuous Memoryless Channel 109
8.1. Introduction . . . . . . . . . . 109
8.2. Strong converse of the coding theorem for the s.c.m.c. 114
8.3. Proof of Lemma 8.2.1 . . . . . . . . . . . . 118
8.4. The strong converse with 0 (vn) in the exponent 121
Remarks . . . . . . . . . . . . . . . . . 124
9. Continuous Channels with Additive Gaussian Noise . . 124
9.1. A continuous memoryless channel with additive Gaussian
noise ......................... 124
9.2. Message sequences within a suitable sphere . . . . . . . . 127
9.3. Message sequences on the periphery of the sphere or within a
shell adjacent to the boundary . . . . . 129
9.4. Another proof of Theorems 9.2.1 and 9.2.2 130
Remarks . . . . 133
10. Mathematical Miscellanea . . . . . . . . 133
10.1. Introduction . . . . . . . . . . . 133
10.2. The asymptotic equipartition property 133
10.3. Admissibility of an ergodic input for a discrete finite-memory
channel . . . . . . 137
11. Group Codes. Sequential Decoding. . 140
11.1. Group Codes. . . . . . . . . 140
11.2. Canonical form of the matrix M 144
11.3. Sliding parity check codes. 145
11.4. SequentiaI decoding. 146
Remarks. . 151
References 152
Index ... 155
List of ChanneIs Studied or Mentioned 156
1. Heuristic Introduction to the Discrete
Memoryless Channel
The spirit of the problems discussed in the present monograph can
already be gleaned from a consideration of the discrete memoryless
channel, to a heuristic discussion of which the present chapter is devoted.
In this discussion there will occur terms not yet precisely defined, to
which the reader should give their colloquial meaning. This procedure
is compatible with the purpose of the present chapter, which is to
motivate the problems to be discussed later in the book, and not to
carry forward the theory logically. The reader scomful of such un
mathematical behavior or in no need of motivation may proceed at
on ce to Chapter 2 without any lass of logical continuity. Such definitions
as are given in the present chapter will be repeated later.
We suppose that a stream of symbols, each of which is a letter of an
alphabet, is being sent over a discrete, noisy, memoryless channel.
"Discrete" (a term of engineering origin) means here that the alphabet
of the letters being sent, and the alphabet of the letters being received
(which need not be the same) each have a finite number (> 1) of symbols.
We take these two numbers to be the same, say a. Since the actual
symbols will not matter we may assurne, and do, that both alphabets
consist of the numbers (letters) 1, ... , a. (It is easy to admit the possi
bility that one alphabet may have fewer symbols than the other. In that
case ais the larger number of symbols, and the modifications needed in
the theory below will be trivially obvious.) "Noisy" means that the
letter sent may be garbled in transmission by noise (= chance error).
Let the probability that the letter (number) j will be received when the
letter i is sent be w (j / i). Ofcourse
w(l/i)+ ... +w(a/i)=l, i=l, ... ,a.
"Memoryless" means that the noise affects each letter independently
of all other letters.
We shall assurne that all the "words" we send have n letters. There
is no reason for this assumption except that the theory has been developed
under it, and that its elimination would make things more difficult.
In ordinary writing, where the words of a language have different length,
the word is determined by the spaces which separate it from its neighbors.
1 Ergebn. d. Mathem., N. F., Bd. 31, Wolfowitz, 2. Auf!.
2 1. Heuristic Introduction to the Discrete Memoryless Channel
Such a "space" is really a letter of the alphabet. In a theory such as
the one to be studied below, one seeks the most efficient use, in a certain
sense, of the letters of the alphabet to transmit the words of a language.
It is not at all certain that setting aside one letter of the alphabet to
serve as a "space" is really the most efficient use of the letter (this even
ignoring the fact that, because of noise, a "space" may not be received
as a "space"). If one letter is not to serve to separate one word from the
next then this will have to be done 'bythe code (definition later), a
requirement which would enormously complicate the problem. Thus
we assurne that . all the words sent are sequences of n integers, each
integer one of 1, ... , a. (Any such sequence will be called an n-sequence.)
Asymptotically (with n) this requirement willmake no difference.
Suppose we have a dictionary of N words in a certain language.
Here language could, for example, mean what is colloquially meant by
a language, or it could mean the totality of words in a certain military
vocabulary, or in a given diplomatie code book, or in a prescribed
technical handbook. The words in the dictionary are of course what are
colloquially known as words; they are written in the alphabet of the
language (if the latter is English, for example, its alphabet is not the
alphabet 1, ... , a of the letters sent over the channel) and they are
not in general n-sequences or even of the same length (certainly not if
the' language is English). We wish to transmit (synonym for "send")
any of these words in any arbitrary order and with any arbitrary fre
quency over the channel.
If there were no noise we would proceed as follows: . Let n be the
smallest integer such that an > N. Construct in'any manner a one-to-one
correspondence between the words of the dictionary and N of the an
n-sequences, this correspondence of course known to both sender and
receiver. When one wishes to send any word in the dictionary one sends
the n-sequence which corresponds to it. The receiver always receives
the n-sequence correctly, and hence knows without error the word being
sent.
When there is noise we must expect that in general there will be a
positive probability that an 1!-sequence sent will be incorrectly received,
i.e., received as another sequence (error of transmission). For example,
when w (j I i) > 0 for every i, j = 1, ... , a, this will surely be the case.
We would like then to be able to send over the channel any of the N
words of the dictionary (more properly, any of the N n-sequences which
correspond to the N words of the dictionary), with a prob ability < A,
say, that any word of the dictionary (or rat her n-sequence corresponding
to it) sent will be incorrectly understood by the receiver. (Of course it is
in general impossible that A = o. We shall be satisfied if A > 0 is suitably
small, and shall henceforth always assurne, unless the contrary is expli-
