Information Theory: Coding Theorems for Discrete Memoryless Systems

523 Pages·2011·3.699 MB·English

by  Imre Csiszar

InformationTheory
CodingTheoremsforDiscreteMemorylessSystems
This book is widely regarded as a classic in the field of information theory, providing
deep insights and expert treatment of the key theoretical issues. It includes in-depth
coverageofthemathematicsofreliableinformationtransmission,bothintwo-terminal
and multi-terminal network scenarios. Updated and considerably expanded, this new
edition presents unique discussions of information-theoretic secrecy and of zero-error
information theory, including substantial connections of the latter with extremal com-
binatorics.Thepresentationsofallcoresubjectsareself-contained,eventheadvanced
topics,whichhelpsreaderstounderstandtheimportantconnectionsbetweenseemingly
differentproblems.Finally,320end-of-chapterproblems,togetherwithhelpfulsolving
hints, allow readers to develop a full command of the mathematical techniques. This
is an ideal resource for graduate students and researchers in electrical and electronic
engineering,computerscience,andappliedmathematics.
Imre Csiszár is a Research Professor at the Alfréd Rényi Institute of Mathematics of
theHungarianAcademyofSciences,wherehehasworkedsince1961.HeisalsoPro-
fessor Emeritus of the University of Technology and Economics, Budapest, a Fellow
of the IEEE, and former President of the Hungarian Mathematical Society. He has
received numerous awards, including the Shannon Award of the IEEE Information
TheorySociety(1996).
JánosKörnerisaProfessorofComputerScienceattheSapienzaUniversityofRome,
Italy,wherehehasworkedsince1992.Priortothis,hewasamemberoftheInstitute
ofMathematicsoftheHungarianAcademyofSciencesforover20years,andhealso
workedatAT&TBellLaboratories,MurrayHill,NewJersey,fortwoyears.
The field of applied mathematics known as Information Theory owes its origins and
earlydevelopmenttothreepioneers:Shannon(USA),Kolmogorov(Russia)andRényi
(Hungary). This book, authored by two of Rényi’s leading disciples, represents the
elegant and precise development of the subject by the Hungarian School. This sec-
ondeditioncontainsnewresearchoftheauthorsonapplicationstosecrecytheoryand
zero-errorcapacitywithconnectionstocombinatorialmathematics.
AndrewViterbi,USC
Information Theory: Coding Theorems for Discrete Memoryless Systems, by Imre
Csiszár and János Körner, is a classic of modern information theory. “Classic” since
its first edition appeared in 1979. “Modern” since the mathematical techniques and
the results treated are still fundamentally up to date today. This new edition was long
overdue.Beyondtheoriginalmaterial,itcontainstwonewchaptersonzero-errorinfor-
mationtheoryandconnectionstoextremalcombinatorics,andoninformationtheoretic
security, a topic that has garnered very significant attention in the last few years. This
bookisanindispensablereferenceforresearchersandgraduatestudentsworkinginthe
excitingandever-growingareaofinformationtheory.
GiuseppeCaire,USC
The first edition of the Csiszár and Körner book on information theory is a classic, in
constantusebymostmathematically-orientedinformationtheorists.Thesecondedition
expandsthefirstwithtwonewchapters,oneonzero-errorinformationtheoryandone
oninformationtheoreticsecurity.Theseusethesameconsistentsetoftoolsasedition
1 to organize and prove the central results of these currently important areas. In addi-
tion,therearemanynewproblemsaddedtotheoriginalchapters,placingmanynewer
researchresultsintoaconsistentformulation.
RobertGallager,MIT
The classic treatise on the fundamental limits of discrete memoryless sources and
channels–anindispensabletoolforeveryinformationtheorist.
SergioVerdu,Princeton
Information Theory
Coding Theorems for Discrete Memoryless Systems
IMRE CSISZÁR
AlfrédRényiInstituteofMathematics,HungarianAcademyofSciences,Hungary
JÁNOS KÖRNER
SapienzaUniversityofRome,Italy
CAMBRIDGE UNIVERSITY PRESS
Cambridge,NewYork,Melbourne,Madrid,CapeTown,
Singapore,SãoPaulo,Delhi,Tokyo,MexicoCity
CambridgeUniversityPress
TheEdinburghBuilding,CambridgeCB28RU,UK
PublishedintheUnitedStatesofAmericabyCambridgeUniversityPress,NewYork
www.cambridge.org
Informationonthistitle:www.cambridge.org/9780521196819
Firstedition(cid:2)c AkadémiaiKiadó,Budapest1981
Secondedition(cid:2)c CambridgeUniversityPress2011
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished1981
Secondedition2011
PrintedintheUnitedKingdomattheUniversityPress,Cambridge
AcatalogrecordforthispublicationisavailablefromtheBritishLibrary
ISBN978-0-521-19681-9Hardback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceor
accuracyofURLsforexternalorthird-partyinternetwebsitesreferredto
inthispublication,anddoesnotguaranteethatanycontentonsuch
websitesis,orwillremain,accurateorappropriate.
TothememoryofAlfrédRényi,
theoutstandingmathematician
whoestablishedinformationtheoryinHungary
1
5 2 3 4
10 6 7 9
11 12 8
13 15
14 16 17
Dependence graph of the text; numbers refer to chapters
Contents
Prefacetothefirstedition pageix
Prefacetothesecondedition xi
Basicnotationandconventions xii
Introduction xv
PartI Informationmeasuresinsimplecodingproblems 1
1 Sourcecodingandhypothesis
testing;informationmeasures 3
2 Typesandtypicalsequences 16
3 FormalpropertiesofShannon’s
informationmeasures 34
4 Non-blocksourcecoding 48
5 Blowinguplemma:acombinatorialdigression 71
PartII Two-terminalsystems 81
6 Thenoisychannelcodingproblem 83
7 Rate-distortiontrade-offinsourcecodingandthesource–channel
transmissionproblem 107
8 Computationofchannelcapacityand(cid:2)-distortionrates 120
9 Acoveringlemmaandtheerrorexponentinsourcecoding 132
10 Apackinglemmaandtheerrorexponentinchannelcoding 144
11 Thecompoundchannelrevisited:zero-errorinformationtheoryandextremal
combinatorics 184
viii Contents
12 Arbitrarilyvaryingchannels 209
PartIII Multi-terminalsystems 241
13 Separatecodingofcorrelatedsources 243
14 Multiple-accesschannels 272
15 Entropyandimagesizecharacterization 304
16 Sourceandchannelnetworks 354
17 Information-theoreticsecurity 400
References 461
Nameindex 478
Indexofsymbolsandabbreviations 482
Subjectindex 485