Optimal Segmentation Techniques for Piecewise Stationary Signals

126 Pages·2012·1 MB·English

Optimal Segmentation
Techniques for
Piecewise Stationary Signals
Paolo Prandoni
15 March 1999

Contents
Abstract v
Acknowledgements vii
1 Introduction 1
1.1 Beyond the (cid:12)xed-windowapproach . . . . . . . . . . . . . . . . . 3
1.1.1 The role of stationarity . . . . . . . . . . . . . . . . . . . 3
1.1.2 Piecewise stationarity . . . . . . . . . . . . . . . . . . . . 5
1.1.3 Dynamic segmentation . . . . . . . . . . . . . . . . . . . 8
1.2 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Signal Segmentation and Resource Allocation 13
2.1 Optimal allocations and rate-distortion theory . . . . . . . . . . . 16
2.2 Segmentation and allocation techniques . . . . . . . . . . . . . . 18
2.2.1 Notation and building blocks . . . . . . . . . . . . . . . . 18
2.2.2 Optimal resource allocation, independent case . . . . . . . 19
2.2.3 Optimal resource allocation, partially independent case . . 23
2.2.4 Joint segmentation and allocation: dependent case . . . . 28
2.3 About side information . . . . . . . . . . . . . . . . . . . . . . . 34
2.3.1 Upper and lower bounds . . . . . . . . . . . . . . . . . . 34
2.3.2 Coloring bits . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.A Alternative to the iteration . . . . . . . . . . . . . . . . . . . . . 40
2.A.1 Independent case . . . . . . . . . . . . . . . . . . . . . . 41
2.A.2 Partially independent case . . . . . . . . . . . . . . . . . 43
2.A.3 Dependent case . . . . . . . . . . . . . . . . . . . . . . . 44
2.A.4 Implementation Issues . . . . . . . . . . . . . . . . . . . . 47
iii
iv CONTENTS
3 Local Polynomial Modeling 51
3.1 Wavelets and compression: a brief overview . . . . . . . . . . . . 52
3.2 R/D upper bounds for piecewise polynomial functions . . . . . . . 56
3.2.1 Oracle-based local modeling . . . . . . . . . . . . . . . . 56
3.2.2 Wavelet-based approximation . . . . . . . . . . . . . . . . 60
3.2.3 Comments and experimental results . . . . . . . . . . . . 62
3.3 R/D optimal approximation . . . . . . . . . . . . . . . . . . . . . 65
3.3.1 Operational setup . . . . . . . . . . . . . . . . . . . . . . 67
3.3.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . 67
3.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.3.4 Whereto tends all this? . . . . . . . . . . . . . . . . . . . 72
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.A Local Legendre Expansion . . . . . . . . . . . . . . . . . . . . . . 74
3.B Estimate of the series truncation error . . . . . . . . . . . . . . . 75
4 Data Compression: Linear Prediction and Arithmetic Coding 77
4.1 Speech compression via Linear Prediction . . . . . . . . . . . . . 78
4.1.1 Linear Prediction . . . . . . . . . . . . . . . . . . . . . . 80
4.1.2 R/D optimal Linear Prediction . . . . . . . . . . . . . . . 82
4.1.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.2 Arithmetic coding . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.2.1 Theoretical background . . . . . . . . . . . . . . . . . . . 89
4.2.2 Rate-optimal arithmetic coding . . . . . . . . . . . . . . . 90
4.2.3 A simple example . . . . . . . . . . . . . . . . . . . . . . 94
4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5 Communications: Data Hiding for Audio Signals 97
5.1 Data hiding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.2 Perceptual audio coding . . . . . . . . . . . . . . . . . . . . . . . 99
5.2.1 Psychoacoustic modeling . . . . . . . . . . . . . . . . . . 100
5.2.2 Compression . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.3 Perceptual data hiding . . . . . . . . . . . . . . . . . . . . . . . 101
5.3.1 Discretization . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3.2 Data hiding and side information . . . . . . . . . . . . . . 104
5.4 R/D optimal data hiding . . . . . . . . . . . . . . . . . . . . . . 105
5.4.1 Problem setup . . . . . . . . . . . . . . . . . . . . . . . . 105
5.4.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . 107
5.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Abstract
Theconceptofstationarityiscentraltosignalprocessing;itindeedguarantees
that the deterministic spectral properties of linear time-invariant systems are
also applicable to realizations of stationary random processes. In almost all
practicalsettings, however, the signalat hand is just a (cid:12)nite-size vectorwhose
underlying generating process, if we are willing to admit one, is unknown; in
thiscase,invokingstationarityistantamounttostatingthatasinglelinearsys-
tem(howevercomplex)su(cid:14)cestomodelthe datae(cid:11)ectively,beitforanalysis,
processing,orcompressionpurposes. Itisintuitivelyclearthatifthe complex-
ityofthemodelcangrowunchecked,itsaccuracycanincreasearbitrarily(short
of computationallimitations); this de(cid:12)nes atradeo(cid:11)in which, foragivendata
vector, a set of complexity/accuracypairs are de(cid:12)ned for each possible model.
In general one is interested in parsimoniousmodeling; by identifying complex-
ity with \rate" and model misadjustment with \distortion", the situation be-
comesakinto anoperationalrate-distortion(R/D) problem in which, for each
possible \rate",the goalis to (cid:12)nd the model yielding the minimum distortion.
Inpractice,however,onlya(cid:12)nitepaletteofmodelsisavailable,thecom-
plexityof whichislimited bycomputationalreasons;therefore,the entiredata
vector often proves too \nonstationary" for any single model. If the process
is just slowly drifting, adaptive systems are probably the best choice; on the
other hand, a wide class of signals exhibits a series of rather abrupt transition
between locally regular portions (e.g. speech, images). In this case a common
solutionistopartitionthedatauniformlysothattheresultingpiecesaresmall
enough to appear stationary with respect to the available models. However, if
the goal is again to obtain an overall modeling which is optimal in the above
R/D sense,itisnecessarythatthe segmentationbe afreevariablein the mod-
elization process; this is howevernot the case if a (cid:12)xed-size time windowing is
used. Notethat nowthe reachablepointsinthe R/D planeareinfact indexed
notjust byamodel but byasegmentation/model-sequencepair; theirnumber
therefore grows exponentially with the size of the data vector.
v
vi
This thesis is concerned with the development of e(cid:14)cient techniques to
explore this R/D set and to determine its operational lower bound for speci(cid:12)c
signalprocessingproblems. Itwillbeshownthat, underverymildhypotheses,
many practical applications dealing with nonstationary data sets can be cast
as a R/D optimization problem involving segmentation, which can in turn be
solved using polynomial-time dynamic programming techniques. The (cid:13)exibil-
ityof the approachwillbe demonstratedbyaverydiverseset ofexamples: af-
ter ageneraloverviewof the variousfacets of the dynamic segmentationprob-
lem in Chapter 2, Chapter 3 will use the framework to determine an opera-
tionalR/Dboundfortheapproximationofpiecewisepolynomialfunctionwith
respect to wavelet-based approximation; Chapter 4 will show its relevance to
compression problems, and in particular to speech coding based on linear pre-
diction and to arithmetic coding for binary sources; (cid:12)nally, in Chapter 5, an
e(cid:14)cient data hiding scheme for PCM audio signals will be described, in which
the optimalpowerallocationforthe hiddendataisdetermined with respectto
the time-varying characteristics of the host signal.
Acknowledgements
No pain, no pain.
{ ANONYMOUS, workout T-shirt
The overall readership of a Ph.D. thesis is, to put it mildly, quite lim-
ited; the technicalpublicatlargeismorelikelytocomeincontactwith the as-
sociated feuilleton of journal and conference papers a thesis is generally borne
out of and, sometimes, milked out of even postmortem. There might however
be one or two epigonous students charged with the dubious honor of \contin-
uing the present line of research"; they will probably have to read the whole
manuscriptand to them I instantly o(cid:11)er my sincerest apologiesforany lackof
clarity and for the boring stu(cid:11) they’ll have to wade through. The only other
qualifyingreadersareof coursethe members of mythesis committee; to them,
the aboveapologiesarelikewise extended, and complemented bymy gratitude
for their patience and their time (if they pulled past Chapter 1, that is). Yet,
morepeoplewillcastaglanceonthisworkthanjustinterestedorcoercedread-
ers, for it is well known that fresh doctors’homes (and bathrooms) arestrewn
with copiesof thethesisleft infullviewforthebene(cid:12)tof visitingfriends;most
doctors(it is rumored) also succumb to an unconfessable urge and mail copies
even to their more distant and unsuspecting acquaintances. Politeness on the
readers’ part demands that they at least thumb through the pages; politeness
on the writer’s part demands that he try to put their names in print here:
that’s all they’ll be looking for anyway. But let’s proceed with order.
This thesis sums up my research e(cid:11)orts as a member ot the LCAV, the
audiovisualcommunicationlaboftheEPFL.Thelabwas(andstillis)underthe
ironhanded rule of my advisor, monsieur le professeur Martin Vetterli; a man
whoun(cid:13)inchinglysnatchedmeawayfrommyblissfullifeinCalifornia,frommy
vii
viii
sushi dinners, and from my promising career in Corporate America, and who
chainedmetothe(cid:12)fthpillaroftheLCAVdungeonstillthethesiswasdone: he
shouldn’t be getting many thanks here. But the fact is, his incredibly enthusi-
astic approachto researchand his unbelievable patience towardsthe innumer-
able self-centered little life crises of grad students such as myself really give a
meaning to the word \advising" which should become a worldwide academic
standard. So{uhm{well{thanks,boss. Andsincewe’retalkingaboutLCAV,
letmealsothankinastridemyone-of-a-kindo(cid:14)cematePina,whosepatience
Ihavetriedgalore(andshemine)butwho,amazingly,isstillonfriendlyterms
with me;andallthe restofthe gang,with whomtheremayhavebeenupsand
downs of course, but after all a lab is a microcosm that mirrors life in all its
facets and all’s well that ends well. On the other side of the ocean, my grati-
tude goesto Grace, Masoudand Vivek for sharingwith me the initial pains of
relocatingtoSwitzerland. TheylaterreturnedtoCalifornia,Ididn’t. I’llcatch
youlater,guys. Ialsoowethemalotinmoreacademicterms,ofcourse. Mike1
mentionedmein hisbook, soImust reciprocate;he’s been theonlyfellowgrad
studentI’vemetwhocoulde(cid:11)ectivelycounterbalancemyengineeringnihilism;
and his unswerving course on the road to optimality has been unquestionably
inspiring. Manythanks also to Metin and Les for the rewarding time (reward-
inginmanywaysbut, alas,notmonetarily)IwasallowedtospendatC-Cube.
This is de(cid:12)nitely not the place to exhume unresolved and unresolvable
issues of Freudian caliber with my family, so I won’t justify my refusal to fol-
low the usual custom and thank my parents. But I have just mentioned them,
haven’t I, which means I am reaching out; now it’s up to them to come for-
ward and start apologizing for not buying me that beautiful red toy I wanted
so much when I was three and a half. My sister, on the other hand, I will
thank {hopingonedayshe’ll stopchidingme forwhat sheperceivesasanim-
morallifestyle. Friendsarefortunately easiertodeal with; Jochen (lawyerand
musician),Mike2 (historianandmusician),andNicola(engineerandmusician)
have provided many an hour of respite from the soulless labor of engineering
research: the common denominator should be apparent; Jochen is also espe-
cially credited for his amazingherbal remedieswhich helped me through these
harsh Swiss winters. My old high-school buddies (Berga, Calde, Ceck, Rige,
plustheirsigni(cid:12)cantothers),thoughallbalding,kept makingmytripsbackto
Italyasure(cid:12)reblast;andMonicakeptremindingmewhyItalianwomenareso
coveted worldwide. Thanks to Edouard and Diana for their unmatched hospi-
tality during my trips to Berkeley and for their soul-searching facilities. And,
1Goodwin
2Gubser
ix
eventhough I knowshedoesn’t wanttobe mentioned, I reallycouldnot move
on to business without a huge merci to Claire.
Lausanne, April 27, 1999.
Chapter 1
Introduction
Non domandarci la formula che mondi possa aprirci
S(cid:18)(cid:16) qualche storta sillaba e secca come un ramo.
Codesto solo oggi possiamo dirti,
Cio(cid:18) che non siamo, cio(cid:18) che non vogliamo.
{ E. MONTALE, Ossi di seppia
Never trust anybody above 30.
{ Banner at Sather Gate, Berkeley
The most eventful moment in the academic journey of an engineer-to-
be iswithout doubt the study of Taylorseries;from thatmoment on, nonatu-
ral phenomenon proves too complex to escape linearization and all di(cid:14)culties
can be dismissed by virtue of su(cid:14)ciently small intervals. In signal processing,
this philosophical frame of mind (cid:12)nds its parallel in the concept of stationar-
ity and in the use of su(cid:14)ciently small analysis windows. The story goes more
or less like this.
1