Table Of ContentINFORMATION AND
COMMUNICATION
THEORY
IEEEPress
445HoesLane
Piscataway,NJ08854
IEEEPressEditorialBoard
EkramHossain,EditorinChief
GiancarloFortino AndreasMolisch LindaShafer
DavidAlanGrier SaeidNahavandi MohammadShahidehpour
DonaldHeirman RayPerez SarahSpurgeon
XiaoouLi JeffreyReed AhmetMuratTekalp
INFORMATION AND
COMMUNICATION
THEORY
STEFAN HO¨ ST
LundUniversity,Sweden
Copyright©2019byTheInstituteofElectricalandElectronicsEngineers,Inc.Allrightsreserved.
PublishedbyJohnWiley&Sons,Inc.,Hoboken,NewJersey.
PublishedsimultaneouslyinCanada.
Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformor
byanymeans,electronic,mechanical,photocopying,recording,scanning,orotherwise,exceptas
permittedunderSection107or108ofthe1976UnitedStatesCopyrightAct,withouteithertheprior
writtenpermissionofthePublisher,orauthorizationthroughpaymentoftheappropriateper-copyfeeto
theCopyrightClearanceCenter,Inc.,222RosewoodDrive,Danvers,MA01923,(978)750-8400,fax
(978)750-4470,oronthewebatwww.copyright.com.RequeststothePublisherforpermissionshould
beaddressedtothePermissionsDepartment,JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ
07030,(201)748-6011,fax(201)748-6008,oronlineathttp://www.wiley.com/go/permission.
LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbesteffortsin
preparingthisbook,theymakenorepresentationsorwarrantieswithrespecttotheaccuracyor
completenessofthecontentsofthisbookandspecificallydisclaimanyimpliedwarrantiesof
merchantabilityorfitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedbysales
representativesorwrittensalesmaterials.Theadviceandstrategiescontainedhereinmaynotbesuitable
foryoursituation.Youshouldconsultwithaprofessionalwhereappropriate.Neitherthepublishernor
authorshallbeliableforanylossofprofitoranyothercommercialdamages,includingbutnotlimitedto
special,incidental,consequential,orotherdamages.
Forgeneralinformationonourotherproductsandservicesorfortechnicalsupport,pleasecontactour
CustomerCareDepartmentwithintheUnitedStatesat(800)762-2974,outsidetheUnitedStatesat
(317)572-3993orfax(317)572-4002.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmay
notbeavailableinelectronicformats.FormoreinformationaboutWileyproducts,visitourwebsiteat
www.wiley.com.
LibraryofCongressCataloging-in-PublicationDataisavailable.
ISBN978-1-119-43378-1
PrintedintheUnitedStatesofAmerica.
10 9 8 7 6 5 4 3 2 1
CONTENTS
PREFACE ix
CHAPTER1 INTRODUCTION 1
CHAPTER2 PROBABILITYTHEORY 5
2.1 Probabilities 5
2.2 RandomVariable 7
2.3 ExpectationandVariance 9
2.4 TheLawofLargeNumbers 17
2.5 Jensen’sInequality 21
2.6 RandomProcesses 25
2.7 MarkovProcess 28
Problems 33
CHAPTER3 INFORMATIONMEASURES 37
3.1 Information 37
3.2 Entropy 41
3.3 MutualInformation 48
3.4 EntropyofSequences 58
Problems 63
CHAPTER4 OPTIMALSOURCECODING 69
4.1 SourceCoding 69
4.2 KraftInequality 71
4.3 OptimalCodewordLength 80
4.4 HuffmanCoding 84
4.5 ArithmeticCoding 95
Problems 101
CHAPTER5 ADAPTIVESOURCECODING 105
5.1 TheProblemwithUnknownSourceStatistics 105
5.2 AdaptiveHuffmanCoding 106
5.3 TheLempel–ZivAlgorithms 112
5.4 ApplicationsofSourceCoding 125
Problems 129
v
vi CONTENTS
CHAPTER6 ASYMPTOTICEQUIPARTITIONPROPERTYANDCHANNEL
CAPACITY 133
6.1 AsymptoticEquipartitionProperty 133
6.2 SourceCodingTheorem 138
6.3 ChannelCoding 141
6.4 ChannelCodingTheorem 144
6.5 DerivationofChannelCapacityforDMC 155
Problems 164
CHAPTER7 CHANNELCODING 169
7.1 Error-CorrectingBlockCodes 170
7.2 ConvolutionalCode 188
7.3 Error-DetectingCodes 203
Problems 210
CHAPTER8 INFORMATIONMEASURESFORCONTINUOUSVARIABLES 213
8.1 DifferentialEntropyandMutualInformation 213
8.2 GaussianDistribution 224
Problems 232
CHAPTER9 GAUSSIANCHANNEL 237
9.1 GaussianChannel 237
9.2 ParallelGaussianChannels 244
9.3 FundamentalShannonLimit 256
Problems 260
CHAPTER10 DISCRETEINPUTGAUSSIANCHANNEL 265
10.1 M-PAMSignaling 265
10.2 ANoteonDimensionality 271
10.3 ShapingGain 276
10.4 SNRGap 281
Problems 285
CHAPTER11 INFORMATIONTHEORYANDDISTORTION 289
11.1 Rate-DistortionFunction 289
11.2 LimitForFixPb 300
11.3 Quantization 302
11.4 TransformCoding 306
Problems 319
CONTENTS vii
APPENDIXA PROBABILITYDISTRIBUTIONS 323
A.1 DiscreteDistributions 323
A.2 ContinuousDistributions 327
APPENDIXB SAMPLINGTHEOREM 337
B.1 TheSamplingTheorem 337
BIBLIOGRAPHY 343
INDEX 347
PREFACE
Informationtheorystartedasatopicin1948whenClaudeE.Shannonpublishedthe
paper“Amathematicaltheoryofcommunication.”Asthenamereveals,itisathe-
oryaboutwhatcommunicationandinformationareinamathematicalsense.Shan-
non built a theory first to quantify and measure the information of a source. Then,
by viewing communication as reproduction of information, the theory of commu-
nication came into view. So, without information, or choices, there cannot be any
communication. But these two parts go hand in hand. The information measure is
basedontheamountofdataneededforreconstruction,andcommunicationisbased
ontheamountofdataneededtobetransmittedtotransferacertainamountofinfor-
mation.Inamathematicalsense,choicesmeanprobabilities,andthetheoryisbased
onprobabilitytheory.
Humans have always strived to simplify the process of communication and
spreadingknowledge.Inthe1450swhenGutenbergmanagedtogethisprintingpress
inoperation,itsimplifiedthespreadingofthewrittenwords.Sincethen,wehaveseen
many different technologies that have spread the information—from the telegraph,
viathetelephoneandtelevision,totheInternetasweknowittoday.Whatwillcome
nextwecanonlyspeculate,butalltheseexistingandforthcomingtechnologiesmust
complywiththetheoriesthatShannonstated.Therefore,informationtheoryasatopic
isthebaseforeveryoneworkingwithcommunicationsystems,bothpast,present,and
forthcoming.
Thisbookisintendedtobeusedinafirstcourseininformationtheoryforcom-
municationengineeringstudents,typicallyinhigherundergraduateorlowergraduate
level.Sincethetheoryisbasedonmathematicsandprobabilitytheory,acertainlevel
ofmaturityinthesesubjectsisexpected.Thismeansthatthestudentsshouldhavea
coupleofmathematicscoursesintheirtrunk,suchasbasiccalculusandprobability
theory.Itisalsorecommended,butnotrequired,thattheyhavesomeunderstandingof
digitalcommunicationasaconcept.Thistogetherwillgiveasolidgroundforunder-
standingtheideasofinformationtheory.Thefirstrequirementofalevelofmaturity
inmathematicscomesfrominformationtheorythatsetsupamathematicalmodelof
informationinverygeneralterms,inthesensethatitisvalidforallkindsofcommu-
nication.Onsomeoccasions,thetheorycanbeseenasprettyabstractbystudentsthe
firsttimetheyengagewithit.Then,tohavesomeunderstandingofcommunication
onaphysicallayerbeforehandmighthelptheunderstanding.
Theworkwiththisbook,aswellasthecontent,hasgrownwhilelecturingthe
courseoninformationandcommunication.Bythetimeitstarted,Iusedoneofthe
mostrecommendedbooksinthisfield.Butitlackedtheengineeringpartofthecourse,
suchastheintuitiveunderstandingandadescriptionofwhatitmeansinreality.The
ix
