Least-Mean-Square Adaptive Filters

502 Pages·2003·4.26 MB·English

by  Simon Haykin (ed.)

LEAST-MEAN-SQUARE
ADAPTIVE FILTERS
LEAST-MEAN-SQUARE
ADAPTIVE FILTERS
Edited by
S. Haykin and B. Widrow
A JOHN WILEY & SONS, INC. PUBLICATION
Thisbookisprintedonacid-freepaper.
Copyrightq2003byJohnWiley&SonsInc.Allrightsreserved.
PublishedsimultaneouslyinCanada.
Nopartofthispublicationmaybereproduced,storedinaretrievalsystemortransmittedinanyformorby
anymeans,electronic,mechanical,photocopying,recording,scanningorotherwise,exceptaspermitted
underSections107or108ofthe1976UnitedStatesCopyrightAct,withouteitherthepriorwritten
permissionofthePublisher,orauthorizationthroughpaymentoftheappropriateper-copyfeetothe
CopyrightClearanceCenter,222RosewoodDrive,Danvers,MA01923,[phone],fax(978)750-
4744.RequeststothePublisherforpermissionshouldbeaddressedtothePermissionsDepartment,John
Wiley&Sons,Inc.,111RiverStreet,Hoboken,NewJersey07030,[phone],fax[phone],
E-Mail:[email protected].
Fororderingandcustomerservice,call1-800-CALL-WILEY.
LibraryofCongressCataloging-in-PublicationData:
Least-mean-squareadaptivefilters/editedbyS.HaykinandB.Widrow
p.cm.
Includesbibliographicalreferencesandindex.
ISBN0-471-21570-8(cloth)
1. Adaptivefilters—Designandconstruction—Mathematics.2.Leastsquares.I.Widrow,
Bernard,1929-II.Haykin,Simon,1931-
TK7872.F5L43 2003
621.38150324—dc21
2003041161
PrintedintheUnitedStatesofAmerica.
10987654321
Thisbook isdedicatedto Bernard Widrowforinventingthe LMS filter
and investigatingits theoryand applications
Simon Haykin
CONTENTS
Contributors ix
Introduction: The LMS Filter (Algorithm) xi
Simon Haykin
1. On the Efficiency ofAdaptive Algorithms 1
Bernard Widrow and Max Kamenetsky
2. Traveling-Wave Model of Long LMS Filters 35
Hans J. Butterweck
3. EnergyConservation and the Learning AbilityofLMS
Adaptive Filters 79
Ali H.Sayed and V. H.Nascimento
4. On the Robustness of LMS Filters 105
Babak Hassibi
5. Dimension Analysis for Least-Mean-Square Algorithms 145
Iven M. Y. Mareels, John Homer,and Robert R. Bitmead
6. Control ofLMS-Type Adaptive Filters 175
Eberhard Ha¨nslerand Gerhard Uwe Schmidt
7. Affine Projection Algorithms 241
Steven L. Gay
8. Proportionate Adaptation: New Paradigms inAdaptive Filters 293
Zhe Chen, Simon Haykin,and StevenL. Gay
9. Steady-State Dynamic Weight Behaviorin (N)LMS Adaptive
Filters 335
A. A. (Louis) Beex and James R. Zeidler
vii
viii CONTENTS
10. Error Whitening WienerFilters:Theory and Algorithms 445
Jose C. Principe, YadunandanaN. Rao, andDeniz Erdogmus
Index 491
CONTRIBUTORS
A. A. (LOUIS) BEEX, Systems Group—DSP Research Laboratory, The Bradley
Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg,
VA 24061-0111
ROBERT R. BITMEAD, Department of Mechanical and Aerospace Engineering,
University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093-
0411
HANS BUTTERWECK, Technische Universiteit Eindhoven, Faculteit Elektrotech-
niek, EH 5.29, Postbus 513, 5600MB Eindhoven,Netherlands
ZHE CHEN, Department of Electrical and Computer Engineering, CRL 102,
McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada L8S
4K1
DENIZ ERDOGMUS, Computational NeuroEngineering Laboratory, EB 451,
Building 33, UniversityofFlorida, Gainesville, FL32611
STEVEN L. GAY, Acoustics and Speech Research Department, Bell Labs, Room
2D-531, 600 MountainAve.,Murray Hill, NJ 07974
PROF. DR.-ING. EBERHARD HA¨NSLER, Institute of Communication Technology,
Darmstadt University of Technology, Merckstrasse 25, D-64283 Darmstadt,
Germany
BABAK HASSIBI, Department of Electrical Engineering, 1200 East California
Blvd., M/C 136-93, CaliforniaInstitute ofTechnology,Pasadena, CA91101
SIMON HAYKIN, Department of Electrical and Computer Engineering, McMaster
University, 1280 Main Street West, Hamilton,Ontario, Canada L8S 4K1
JOHN HOMER, School of Computer Science and Electrical Engineering, The
UniversityofQueensland, Brisbane 4072
MAX KAMENETSKY, Stanford University, David Packard Electrical Engineering,
350 Serra Mall, Room 263, Stanford, CA 94305-9510
IVEN M.Y.MAREELS, DepartmentofElectricalandElectronicEngineering,The
UniversityofMelbourne,Melbourne Vic3010
ix
x CONTRIBUTORS
V.H.NASCIMENTO, DepartmentofElectronicSystemsEngineering,Universityof
Sa˜o Paulo, Brazil
JOSE C. PRINCIPE, Computational NeuroEngineering Laboratory, EB 451,
Building 33, UniversityofFlorida, Gainesville, FL32611
YADUNANDANA N. RAO, Computational NeuroEngineering Laboratory, EB 451,
Building 33, UniversityofFlorida, Gainesville, FL32611
ALIH.SAYED, DepartmentofElectricalEngineering,Room44-123AEngineering
IV Bldg, UniversityofCalifornia, Los Angeles, CA 90095-1594
GERHARD UWE SCHMIDT, Institute of Communication Technology, Darmstadt
UniversityofTechnology,Merckstrasse25, D-64283Darmstadt, Germany
BERNARD WIDROW, Stanford University, David Packard Electrical Engineering,
350 Serra Mall, Room 273, Stanford, CA 94305-9510
JAMES R. ZEIDLER, Department of Electrical and Computer Engineering,
UniversityofCalifornia, San Diego,La Jolla, CA92092
INTRODUCTION: THE LMS FILTER
(ALGORITHM)
SIMON HAYKIN
The earliest work on adaptive filters may be traced back to the late 1950s, during
which time a number of researchers were working independently on theories and
applications of such filters. From this early work, the least-mean-square ðLMSÞ
algorithm emerged as a simple, yet effective, algorithm for the design of adaptive
transversal(tapped-delay-line) filters.
TheLMSalgorithmwasdevisedbyWidrowandHoffin1959intheirstudyofa
pattern-recognition machine known as the adaptive linear element, commonly
referred to as the Adaline [1, 2]. The LMS algorithm is a stochastic gradient
algorithminthatititerateseachtapweightofthetransversalfilterinthedirectionof
theinstantaneousgradientofthesquarederrorsignalwithrespecttothetapweight
inquestion.
Let ww^ðnÞ denote the tap-weight vector of the LMS filter, computed at iteration
(time step) n. The adaptive operation of the filter is completely described by the
recursiveequation (assumingcomplex data)
ww^ðnþ1Þ¼ww^ðnÞþmuðnÞ½dðnÞ(cid:1)ww^HðnÞuðnÞ(cid:2)*; ð1Þ
whereuðnÞisthetap-inputvector,dðnÞisthedesiredresponse,andmisthestep-size
parameter.Thequantityenclosedinsquarebracketsistheerrorsignal.Theasterisk
denotescomplexconjugation,andthesuperscriptHdenotesHermitiantransposition
(i.e., ordinarytransposition combinedwith complexconjugation).
Equation (1) is testimony to the simplicity of the LMS filter. This simplicity,
coupledwithdesirablepropertiesoftheLMSfilter(discussedinthechaptersofthis
book)andpracticalapplications[3,4],hasmadetheLMSfilteranditsvariantsan
importantpartoftheadaptivesignalprocessingkitoftools,notjustforthepast40
yearsbutformanyyearstocome.Simplyput,theLMSfilterhaswithstoodthetest
oftime.
AlthoughtheLMSfilterisverysimpleincomputationalterms,itsmathematical
analysis is profoundly complicated because of its stochastic and nonlinear nature.
Indeed, despite the extensive effort that has been expended in the literature to
xi
xii INTRODUCTION:THELMSFILTER(ALGORITHM)
analyze the LMS filter, we still do not have a direct mathematical theory for its
stability and steady-state performance, and probably we never will. Nevertheless,
we do have a good understanding of its behavior in a stationary as well as a
nonstationary environment, asdemonstrated inthe chapters ofthis book.
The stochastic nature of the LMS filter manifests itself in the fact that in a
stationaryenvironment,andundertheassumptionofasmallstep-sizeparameter,the
filterexecutesaformofBrownianmotion.Specifically,thesmallstep-sizetheoryof
the LMS filter is almost exactly described by the discrete-time version of the
Langevinequation1 [3]:
Dn ðnÞ¼n ðnþ1Þ(cid:1)n ðnÞ
k k k ð2Þ
¼(cid:1)mln ðnÞþfðnÞ; k¼1;2;...;M;
k k k
whichisnaturallysplitintotwoparts:adampingforce(cid:1)mln ðnÞandastochastic
k k
force fðnÞ.Theterms used herein are defined as follows:
k
M ¼order (i.e., number of taps) of the transversal filter around which the
LMS filter is built
l ¼ktheigenvalueofthecorrelationmatrixoftheinputvectoruðnÞ,which
k
is denoted by R
fðnÞ¼kth component of the vector (cid:1)mQHuðnÞe*ðnÞ
k o
Q¼unitary matrix whose M columns constitute an orthogonal set of
eigerivectorsassociatedwiththeeigenvaluesofthecorrelationmatrixR
e ðnÞ¼optimum error signal produced by the corresponding Wiener filter
o
driven by the input vectoruðnÞand the desired response dðnÞ
ToillustratethevalidityofEq.(2)asthedescriptionofsmallstep-sizetheoryof
theLMSfilter,wepresenttheresultsofacomputerexperimentonaclassicexample
of adaptive equalization. The example involves an unknown linear channel whose
impulseresponse is described bythe raised cosine [3]
8 (cid:3) (cid:1) (cid:2)(cid:4)
<1 2p
1þcos ðn(cid:1)2Þ ; n¼1;2;3;
h ¼ 2 W ð3Þ
n :
0; otherwise
wheretheparameterW controlstheamountofamplitudedistortionproducedbythe
channel, with the distortion increasing with W. Equivalently, the parameter W
controlstheeigenvaluespread(i.e.,theratioofthelargesteigenvaiuetothesmallest
eigenvalue) of the correlation matrix of the tap inputs of the equalizer, with the
eigenvalue spread increasing with W. The equalizer has M ¼11 taps. Figure 1
presentsthelearningcurvesoftheequalizertrainedusingtheLMSalgorithmwith
the step-size parameter m¼0:0075 and varying W. Each learning curve was
obtainedbyaveragingthesquaredvalueoftheerrorsignaleðnÞversusthenumberof
iterations n over an ensemble of 100 independent trials of the experiment. The
1TheLangevinequationisthe“engineer’sversion”ofstochasticdifferential(difference)equations.