Introduction to Information Theory and Data Compression, Second Edition

362 Pages·2003·4.651 MB·English

by  D.C. Hankerson, Greg A. Harris, Peter D. Johnson Jr.

Introduction to
Information
Theory
and
Data
Compression
Second Edition
© 2003 by CRC Press LLC
DISCRETE
MATHEMATICS
and
ITS APPLICATIONS
Series Editor
Kenneth H. Rosen, Ph.D.
AT&T Laboratories, Middletown, New Jersey
Abstract Algebra Applications with Maple,
Richard E. Klima, Ernest Stitzinger, and Neil P. Sigmon
Algebraic Number Theory, Richard A. Mollin
An Atlas of the Smaller Maps in Orientable and Nonorientable Surfaces,
David M. Jackson and Terry I. Visentin
An Introduction to Crytography, Richard A. Mollin
Combinatorial Algorithms: Generation Enumeration and Search,
Donald L. Kreher and Douglas R. Stinson
The CRC Handbook of Combinatorial Designs,
Charles J. Colbourn and Jeffrey H. Dinitz
Cryptography: Theory and Practice, Second Edition, Douglas R. Stinson
Design Theory, Charles C. Lindner and Christopher A. Rodgers
Frames and Resolvable Designs: Uses, Constructions, and Existence,
Steven Furino, Ying Miao, and Jianxing Yin
Fundamental Number Theory with Applications, Richard A. Mollin
Graph Theory and Its Applications, Jonathan Gross and Jay Yellen
Handbook of Applied Cryptography,
Alfred J. Menezes, Paul C. van Oorschot, and Scott A. Vanstone
Handbook of Constrained Optimization,
Herbert B. Shulman and Venkat Venkateswaran
Handbook of Discrete and Combinatorial Mathematics, Kenneth H. Rosen
Handbook of Discrete and Computational Geometry,
Jacob E. Goodman and Joseph O’Rourke
Introduction to Information Theory and Data Compression,
Darrel R. Hankerson, Greg A. Harris, and Peter D. Johnson
Network Reliability: Experiments with a Symbolic Algebra Environment,
Daryl D. Harms, Miroslav Kraetzl, Charles J. Colbourn, and John S. Devitt
RSA and Public-Key Cryptography
Richard A. Mollin
Quadratics, Richard A. Mollin
Verification of Computer Codes in Computational Science and Engineering,
Patrick Knupp and Kambiz Salari
© 2003 by CRC Press LLC
Darrel Hankerson
Greg A. Harris
Peter D. Johnson, Jr.
Introduction to
Information
Theory
and
Data
Compression
Second Edition
CHAPMAN & HALL/CRC
A CRC Press Company
Boca Raton London New York Washington, D.C.
© 2003 by CRC Press LLC
C3138-discl. Page 1 Friday, January 17, 2003 1:19 PM
Library of Congress Cataloging-in-Publication Data
Hankerson, Darrel R.
Introduction to information theory and data compression / Darrel R. Hankerson, Greg A.
Harris, Peter D. Johnson.--2nd ed.
p. cm. (Discrete mathematics and its applications)
Includes bibliographical references and index.
ISBN 1-58488-313-8 (alk. paper)
1. Information theory. 2. Data compression (Computer science) I. Harris, Greg A. II.
Johnson, Peter D. (Peter Dexter), 1945- III. Title. IV. Series.
Q360.H35 2003
005.74¢6—dc21 2002041506
CIP
This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with
permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish
reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials
or for the consequences of their use.
Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical,
including photocopying, microfilming, and recording, or by any information storage or retrieval system, without prior
permission in writing from the publisher.
The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works,
or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying.
Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation, without intent to infringe.
Visit the CRC Press Web site at www.crcpress.com
© 2003 by CRC Press LLC
No claim to original U.S. Government works
International Standard Book Number 1-58488-313-8
Library of Congress Card Number 2002041506
Printed in the United States of America 1 2 3 4 5 6 7 8 9 0
Printed on acid-free paper
© 2003 by CRC Press LLC
Preface
This textbook is aimed at graduate students and upper level undergraduates
in mathematics, engineering,and computerscience. The materialand the ap-
proachofthetextweredevelopedoverseveralyearsatAuburnUniversityintwo
independentcourses,InformationTheoryandDataCompression.Althoughthe
materialinthetwocoursesisrelated,wethinkitunwiseforinformationtheory
tobeaprerequisitefordatacompression,andhavewrittenthedatacompression
sectionofthetextsothatitcanbereadbyorpresentedtostudentswithnoprior
knowledgeofinformationtheory.Therearereferencesinthedatacompression
partto results and proofsin the informationtheorypart of the text, and those
who are interested may browse over those references, but it is not absolutely
necessaryto doso. Infact, perhapsthe bestpedagogicalorderofapproachto
these subjects is the reverse of the apparentlogical order: students will come
toinformationtheorycuriousandbetterpreparedforhavingseensomeofthe
definitionsandtheoremsofthatsubjectplayingaroleindatacompression.
Our main aim in the data compression part of the text, as well as in the
course it grew from, is to acquaint the students with a number of significant
lossless compression techniques, and to discuss two lossy compression meth-
ods.Ouraimisforthestudentstoemergecompetentinandbroadlyconversant
with a large range of techniques. We have striven for a “practical” style of
presentation: here is what you do and here is what it is good for. Nonethe-
less,proofsareprovided,sometimesinthetext,sometimesintheexercises,so
thattheinstructorcanhavetheoptionofemphasizingthemathematicsofdata
compressiontosomedegree.
Informationtheoryis ofa moretheoreticalnaturethandata compression.
It providesa vocabulary and a certain abstraction that can bring the power of
simplificationtomanydifferentsituations. Wethoughtitreasonabletotreatit
as a mathematical theory and to present the fundamental definitions and ele-
mentaryresultsofthattheoryinutterabstractionfromtheparticularproblems
ofcommunicationthroughnoisychannels,whichinspiredthetheoryinthefirst
place.WebringthetheorytobearonnoisychannelsinChapters3and4.
The treatment of information theory given here is extremely elementary.
The channels are memoryless and discrete, and the sources are all “zeroth-
order,” one-state sources (although more complicated source models are dis-
cussedinChapter7). We feelthatthiselementaryapproachisappropriatefor
thetargetaudience,andthat,byleavingmorecomplicatedsourcesandchannels
outofthepicture,wemoreeffectivelyimpartthegraspofInformationTheory
thatwehopeourstudentswilltakewiththem.
The exercises range from the routine to somewhat lengthy problems that
introduceadditionalmaterialorestablishmoredifficultresults. Anasteriskby
v
© 2003 by CRC Press LLC
vi Preface
anexerciseorsectionindicatesthatthematerialisoffthemainroad,sotospeak,
andmightreasonablybeskipped. Inthecaseofexercises,itmayalsoindicate
thattheproblemishardand/orunusual.
Inthedatacompressionportionofthebook,anumberofprojectsrequire
theuseofacomputer. AppendixAdocumentsOctaveandMatlabscriptswrit-
tenbytheauthorsthatcanbeusedonsomeoftheexercisesandprojectsinvolv-
ingtransformmethodsand images, andthat can also serve as buildingblocks
for other explorations. The software can be obtained from the authors’ site,
listedinAppendixC. Inaddition,thesitecontainsinformationaboutthebook,
anonlineversionofAppendixA,andlinkstoothersitesofinterest.
Organization
Here’sabriefsynopsisofeachchapterandappendix.
Chapter1 containsanintroductionto the languageandresults ofprobability
theory.
Chapter2 presentstheelementarydefinitionsofinformationtheory,ajustifi-
cationofthequantificationofinformationonwhichthetheoryisbased,
and the fundamental relations among various sorts of information and
entropy.
Chapter3 isaboutinformationflowthroughdiscretememorylessnoisychan-
nels.
Chapter4 is about coding text from a discrete source, transmitting the en-
coded text through a discrete memoryless noisy channel, and decoding
theoutput. The“classical”fundamentaltheoremsofinformationtheory,
includingtheNoisyChannelTheorem,appearinthischapter.
Chapter5 begins the material of the data compression portion of this book.
Replacementschemesare discussed and the chapter concludeswith the
Noiseless CodingTheorem, provedhere for a binary code alphabet. (It
appearsinChapter4inmoregeneralform.)
Chapter6 discussesarithmeticcoding,whichisofconsiderableinterestsince
itisoptimalinacertainwaythatthereplacementschemesarenot. Con-
siderationsforbothan “ideal”schemeandforpracticalimplementation
onacomputerarepresented.
Chapter7 focuses on the modeling aspects of Chapters 5 and 6 (Chapter 8
continuesthediscussion). Sincecodingmethodssuchasthosepresented
in Chapter 6 can (in theory) produce optimal-length output for a given
model of the source, much of the interest in improvingcompression in
statistical schemes lies in improving the model of the source. Higher-
ordermodelsattempttouselargercontextsforpredictions.Inthesecond
© 2003 by CRC Press LLC
Preface vii
edition, a section on probabilistic finite state source automata has been
added.
Chapter8 considers another approach to modeling, using statistics that are
updatedasthesourceisreadandencoded.Thesehavetheadvantagethat
nostatisticalstudyneedstobedoneinadvanceandtheschemecanalso
detectchangesinthenatureofthesource.
Chapter9 discusses popular dictionary methods. These have been widely
used,inpartduetotheirsimplicity,speed,andrelativelygoodcompres-
sion.ApplicationssuchasRossWilliams’LZRW1algorithm,Unixcom-
press,andGNUzip(gzip)areexamined.
Chapter10 develops the Fourier, cosine, and wavelet transforms, and dis-
cussestheiruseincompressionofsignalsorimages. Thelossy scheme
in JPEG is presented as a widely-usedstandardthat relies on transform
techniques.Thechapterconcludeswithanintroductiontowavelet-based
compression.
AppendixA documents the use of the “JPEGtool” collection of Octave and
MatlabscriptsinunderstandingJPEG-likeimagecompression.
AppendixB contains the source listing for Ross Williams’ LZRW1-A algo-
rithm,whichratherconciselyillustratesaviabledictionarycompression
method.
AppendixC containsmaterialthatdidn’tfitelsewhere. The firstsection lists
sources for information and code for many areas of data compression.
The second section contains a few notes on patents affecting the field.
The final section contains a semi-famous story illustrating some of the
misunderstandingsaboutcompression.
AppendixD offerssolutionsandnotesontheexercises.
Acknowledgments
We’d like to thank Ross Williams for permission to reprinthis LZRW1-A al-
gorithm, and for notes on his sources. Alistair Moffat providedpreprintsand
alertedustootherinformationconcerningarithmeticcoding.IanH.Wittenwas
kindenoughto respondto ourquestionsconcerninga detailin Text Compres-
sion. Weespeciallywishtoacknowledgethehelpoffourreviewers: Jean-loup
GaillyofferedmanyimportantsuggestionsconcerningChapter9andAppendix
C,andgrantedpermissiontouseportionsofthe“FrequentlyAskedQuestions”
documentthatheauthors;TomLanesuggestedanumberofimprovementsand
clarifications in Chapter 10; we are grateful to James R. Wall and to Isidore
FleischerforreviewingportionsofChapters1–5.
There are many folks who have made it easier for the community to un-
derstand the subject; some of their names are in this book. Others, working
© 2003 by CRC Press LLC
viii Preface
on“GNUProject”andotherfreelydistributablesoftware,madethisbookpos-
sible. The list of major contributors to this software is lengthy, and includes
those involved with AUC TeX, dvips[k], Emacs, Ghostview and Ghostscript,
GNU/Linux,theIndependentJPEG Group,Info-ZIP,LATEX, Netpbm, PICTEX,
PortableNetworkGraphics,TEX,xdvi[k],xfig,xv,XY-pic,andmanyGNUutil-
itiessuchasbash, gawk,gccandgdb, gzip,andmake. We wish to especially
thanktheprincipaldeveloperofOctave,JohnEaton.
Thanks are due to A. Scottedward Hodel, Alfred Menezes, Stan Reeves,
andGregRoelofsfor someearly adviceonthe datacompressioncourse. Our
studentswerealsogenerouswiththeiradvice. DouglasLeonardandLucTeir-
linckprovidedsomeinsightfulsuggestionsandclarifications. AlfredMenezes
gets the credit (and the blame) for setting us on the road to a course in data
compressionandthisbook.
Someofthecomputerresourcesforthecourseandbookweremadepossi-
blebyagrantfromtheNationalScienceFoundation,forwhichwearegrateful.
Our direct contacts at CRC Press were Bob Stern, Nora Konopka, Suzanne
Lassandro,TimPletscher,JamieSigal,MimiWilliams, andSylviaWood,and
itwasapleasureworkingwiththem.
© 2003 by CRC Press LLC
Contents
Preface v
PartI:InformationTheory
1 ElementaryProbability 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Conditionalprobability . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5 Bernoullitrials . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.6 Anelementarycountingprinciple . . . . . . . . . . . . . . . . 15
1.7* Ondrawingwithoutreplacement . . . . . . . . . . . . . . . . . 17
1.8 Randomvariablesandexpected,oraverage,value . . . . . . . . 18
1.9 TheLawofLargeNumbers. . . . . . . . . . . . . . . . . . . . 22
2 InformationandEntropy 25
2.1 Howisinformationquantified? . . . . . . . . . . . . . . . . . . 25
2.1.1 Namingtheunits . . . . . . . . . . . . . . . . . . . . . 27
2.1.2 Informationconnectingtwoevents . . . . . . . . . . . . 29
2.1.3 The inevitability of Shannon’s quantification of infor-
mation . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.2 Systemsofeventsandmutualinformation . . . . . . . . . . . . 33
2.3 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4 Informationandentropy . . . . . . . . . . . . . . . . . . . . . 43
3 ChannelsandChannelCapacity 47
3.1 Discretememorylesschannels . . . . . . . . . . . . . . . . . . 47
3.2 Transitionprobabilitiesandbinarysymmetricchannels . . . . . 50
3.3 Inputfrequencies . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.4 Channelcapacity . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5* ProofofTheorem3.4.3,onthecapacityequations . . . . . . . . 67
4 CodingTheory 71
4.1 Encodinganddecoding . . . . . . . . . . . . . . . . . . . . . . 71
4.2 Prefix-conditioncodesandtheKraft-McMillaninequality. . . . 75
4.3 AveragecodewordlengthandHuffman’salgorithm . . . . . . . 79
4.3.1 ThevalidityofHuffman’salgorithm . . . . . . . . . . . 86
4.4 Optimizingtheinputfrequencies . . . . . . . . . . . . . . . . . 90
4.5 Error correction, maximum likelihood decoding, nearest code
worddecoding,andreliability . . . . . . . . . . . . . . . . . . 95
ix
© 2003 by CRC Press LLC
x CONTENTS
4.6 Shannon’sNoisyChannelTheorem . . . . . . . . . . . . . . . 106
4.7 Errorcorrectionwithbinarysymmetricchannelsandequalsource
frequencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.8 Theinformationrateofacode . . . . . . . . . . . . . . . . . . 115
PartII:Data Compression
5 LosslessDataCompressionbyReplacementSchemes 119
5.1 Replacementviaencodingscheme . . . . . . . . . . . . . . . . 120
5.2 Reviewoftheprefixcondition . . . . . . . . . . . . . . . . . . 123
5.3 Choosinganencodingscheme . . . . . . . . . . . . . . . . . . 126
5.3.1 Shannon’smethod . . . . . . . . . . . . . . . . . . . . 127
5.3.2 Fano’smethod . . . . . . . . . . . . . . . . . . . . . . 130
5.3.3 Huffman’salgorithm . . . . . . . . . . . . . . . . . . . 131
5.4 TheNoiselessCodingTheoremandShannon’sbound . . . . . . 134
6 ArithmeticCoding 141
6.1 Purezeroth-orderarithmeticcoding:dfwld . . . . . . . . . . . 142
6.1.1 Rescalingwhileencoding . . . . . . . . . . . . . . . . 146
6.1.2 Decoding . . . . . . . . . . . . . . . . . . . . . . . . . 150
6.2 What’sgoodaboutdfwldcoding:thecompressionratio . . . . . 155
6.3 What’sbadaboutdfwldcodingandsomewaystofixit . . . . . 160
6.3.1 Supplyingthesourcewordlength . . . . . . . . . . . . 161
6.3.2 Computation . . . . . . . . . . . . . . . . . . . . . . . 162
6.3.3 Mustdecodingwaituntilencodingiscompleted? . . . . 164
6.4 Implementingarithmeticcoding . . . . . . . . . . . . . . . . . 167
6.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
7 Higher-orderModeling 181
7.1 Higher-orderHuffmanencoding . . . . . . . . . . . . . . . . . 182
7.2 TheShannonboundforhigher-orderencoding . . . . . . . . . . 186
7.3 Higher-orderarithmeticcoding . . . . . . . . . . . . . . . . . . 191
7.4 Statisticalmodels,statistics,andthepossiblyunknowabletruth . 193
7.5 Probabilisticfinitestatesourceautomata . . . . . . . . . . . . . 197
8 AdaptiveMethods 205
8.1 AdaptiveHuffmanencoding . . . . . . . . . . . . . . . . . . . 206
8.1.1 Compressionandreadjustment . . . . . . . . . . . . . . 209
8.1.2 Higher-orderadaptiveHuffmanencoding . . . . . . . . 210
8.2 MaintainingthetreeinadaptiveHuffmanencoding:themethod
ofKnuthandGallager . . . . . . . . . . . . . . . . . . . . . . 212
8.2.1 Gallager’smethod . . . . . . . . . . . . . . . . . . . . 215
8.2.2 Knuth’salgorithm . . . . . . . . . . . . . . . . . . . . 216
8.3 Adaptivearithmeticcoding . . . . . . . . . . . . . . . . . . . . 219
8.4 Intervalandrecencyrankencoding . . . . . . . . . . . . . . . . 221
© 2003 by CRC Press LLC