Table Of ContentSpringerBriefs in Electrical and Computer
Engineering
For furthervolumes:
http://www.springer.com/series/10059
Leonardo Rey Vega Hernan Rey
•
A Rapid Introduction to
Adaptive Filtering
123
Leonardo ReyVega Hernan Rey
School ofEngineering Department of Engineering
Universityof BuenosAires Universityof Leicester
Paseo Colón850 UniversityRoad
C1063ACV Leicester
BuenosAires LE17RH
Argentina UK
ISSN 2191-8112 ISSN 2191-8120 (electronic)
ISBN 978-3-642-30298-5 ISBN 978-3-642-30299-2 (eBook)
DOI 10.1007/978-3-642-30299-2
SpringerHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2012939513
(cid:2)TheAuthor(s)2013
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor
informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodology now known or hereafter developed. Exempted from this legal reservation are brief
excerpts in connection with reviews or scholarly analysis or material supplied specifically for the
purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe
work. Duplication of this publication or parts thereof is permitted only under the provisions of
theCopyrightLawofthePublisher’slocation,initscurrentversion,andpermissionforusemustalways
beobtainedfromSpringer.PermissionsforusemaybeobtainedthroughRightsLinkattheCopyright
ClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
This book is intended to be a quick reference on basic concepts of adaptive
filtering forstudents,practisingengineers,andresearchers.Inorder touseit fora
graduate level course on adaptive filtering, it should be complemented with some
introductory background material and more importantly, with exercises and
computer projects. Therefore, it is important that the reader has some familiarity
with basic concepts from probability theory and stochastic processes (first and
second order statistics, correlation, stationarity, ergodicity, etc.), algebra, matrix
analysis, and linear systems (stability, linearity, etc.)
We wanted to do more than just derive algorithms and perform simulations
showing their behavior. We prioritized to present the material under a solid the-
oretical background. However, after showing several different interpretations
associated to each algorithm, we included some intuitive thoughts in order to
improvetheunderstandingofhowtheyoperate.Webelievethisisimportantsince
equations,asexactandimportantastheyare,cansometimesobscuresomesimple
and fundamental aspects of an algorithm. We also developed a convergence
analysis using standard and non-standard approaches. Theoretical results are
important since they allow us to make predictions about the performance of an
algorithm, which is very interesting at the design stage when solving a particular
problem. In addition, we covered a few applications so the reader can see how
adaptivefilterscanbeusedinsolvingreal-worldproblems.Severalreferencesare
included so the interested reader can look for more in-depth discussions about
several important topics in adaptive filtering.
WewouldliketothankSaraTressensforgivingustheopportunitytotakeour
first steps into the world of scientific research while we were still young under-
graduates. We are also grateful to Prof. Jacob Benesty for allowing us to develop
our ideas and encourage us to write this book. Finally, we thank especially
Dr.VeronicaMasforproofreadingthemanuscriptandprovidinghelpfulcomments.
Paris, France, March 2012 Leonardo Rey Vega
Leicester, UK, March 2012 Hernan Rey
v
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Why Adaptive Filters?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Organization of the Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Wiener Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Optimal Linear Mean Square Estimation . . . . . . . . . . . . . . . . . 7
2.2 The Principle of Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Linear Optimum Filtering. . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Wiener–Hopf Equation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5 Example: Linear Prediction. . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.5.1 Forward Linear Prediction . . . . . . . . . . . . . . . . . . . . . . 12
2.5.2 Backward Linear Prediction. . . . . . . . . . . . . . . . . . . . . 14
2.6 Final Remarks on Linear Prediction. . . . . . . . . . . . . . . . . . . . . 15
2.7 Further Comments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 Iterative Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.2 Steepest Descent Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2.1 Convergence of the Steepest Descent Algorithm. . . . . . . 21
3.3 Newton-Raphson Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4 Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.5 Further Comments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4 Stochastic Gradient Adaptive Algorithms . . . . . . . . . . . . . . . . . . . 33
4.1 Least Mean Square Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . 33
4.1.1 Stochastic Gradient Approximation. . . . . . . . . . . . . . . . 34
vii
viii Contents
4.1.2 Computational Complexity of LMS. . . . . . . . . . . . . . . . 36
4.1.3 LMS with Complex Signals. . . . . . . . . . . . . . . . . . . . . 36
4.2 NLMS Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2.1 Approximation to a Variable Step Size SD Algorithm. . . 37
4.2.2 Approximation to an NR Algorithm . . . . . . . . . . . . . . . 38
4.2.3 NLMS as the Solution to a Projection Problem . . . . . . . 40
4.2.4 One More Interpretation of the NLMS . . . . . . . . . . . . . 41
4.2.5 Computational Complexity of NLMS . . . . . . . . . . . . . . 42
4.2.6 Example: Adaptive Noise Cancelation. . . . . . . . . . . . . . 43
4.3 Leaky LMS Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.4 Sign Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.4.1 Sign Error Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.4.2 Sign Data Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.4.3 Example: Adaptive Equalization. . . . . . . . . . . . . . . . . . 50
4.5 Convergence Analysis of Adaptive Filters . . . . . . . . . . . . . . . . 53
4.5.1 General Assumptions for the Convergence Analysis . . . . 55
4.5.2 Stability Analysis for a Large Class of Algorithms.. . . . . 57
4.5.3 Steady State Behavior for a Large
Family of Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.5.4 Transient and Tracking Behavior for Adaptive Filters. . . 75
4.6 Affine Projection Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.6.1 APA as the Solution to a Projection Problem. . . . . . . . . 77
4.6.2 Computational Complexity of APA. . . . . . . . . . . . . . . . 79
4.6.3 Decorrelating Property of APA. . . . . . . . . . . . . . . . . . . 79
4.7 Simulations Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.8 Further Comments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5 Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.1 The Least Squares (LS) Problem. . . . . . . . . . . . . . . . . . . . . . . 89
5.1.1 Solution to the LS Problem . . . . . . . . . . . . . . . . . . . . . 90
5.2 Orthogonal Projection Interpretation . . . . . . . . . . . . . . . . . . . . 93
5.3 Weighted Least Squares (WLS). . . . . . . . . . . . . . . . . . . . . . . . 95
5.4 Some Statistical Properties of the LS Solution . . . . . . . . . . . . . 96
5.5 Recursive Least Squares (RLS). . . . . . . . . . . . . . . . . . . . . . . . 98
5.6 Convergence and Computational Complexity Issues . . . . . . . . . 102
5.7 Example: Adaptive Beamforming . . . . . . . . . . . . . . . . . . . . . . 104
5.8 Further Comments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6 Advanced Topics and New Directions. . . . . . . . . . . . . . . . . . . . . . 113
6.1 Optimum H? Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.2 Variable Step Size/Regularization Adaptive Algorithms. . . . . . . 114
Contents ix
6.3 Sparse Adaptive Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.4 Robust Adaptive Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.5 Distributed Adaptive Filtering. . . . . . . . . . . . . . . . . . . . . . . . . 116
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Acronyms
APA Affine Projection Algorithm
BER Bit Error Rate
EMSE Excess Mean Square Error
EWRLS Exponentially Weighted RLS
LHS left hand side
LMS Least Mean Squares
LS Least Squares
MAE Mean Absolute Error
MMSE Minimum Mean Square Error
MSD Mean Square Deviation
MSE Mean Square Error
NLMS Normalized Least Mean Squares
NR Newton-Raphson
pdf probability density function
RHS right hand side
RLS Recursive Least Squares
SD Steepest Descent
SDA Sign-Data Algorithm
SEA Sign-Error Algorithm
SNR Signal to Noise Ratio
SSA Sign-Sign Algorithm
WLS Weighted Least Squares
i.i.d independent and identically distributed
RL Real L-dimensional vector space
CL Complex L-dimensional vector space
ha;bi Inner product between vectors a and b
eig½A(cid:2) ith eigenvalue of matrix A
i
eig ½A(cid:2) Maximum eigenvalue of matrix A
max
eig ½A(cid:2) Minimum eigenvalue of matrix A
min
vðAÞ Condition number of matrix A
xi
xii Acronyms
A(cid:3)1 Inverse of matrix A
y Moore-Penrose pseudoinverse of matrix A
A
NðAÞ Null space of matrix A
RðAÞ Range or column space of matrix A
Rank (A) Rank of matrix A
E½X(cid:2) or E ½X(cid:2) Expectation of random variable X
X
E [XjY] or E ½XjY Conditional expectation of X given Y
XjY
Chapter 1
Introduction
Abstract The area of adaptive filtering is a very important one in the vast field
ofSignalProcessing.Adaptivefiltersareubiquitousincurrenttechnology.System
identificaction,equalizationforcommunicationsystems,activenoisecancellation,
speech processing, sonar, seismology, beamforming, etc, are a few examples from
a large set of applications were adaptive filters are used to solve different kinds
ofproblems.Inthischapterweprovideashortintroductiontotheadaptivefiltering
problemandtothedifferentaspectsthatshouldbetakenintoaccountwhenchoosing
ordesigninganadaptivefilterforaparticularapplication.
1.1 WhyAdaptiveFilters?
The area of adaptive filtering has experienced a great growth in the last 50years,
andeventodayitisconstantlyexpanding.Thisallowedforthepossibilityofapply-
ing adaptive filters in such a variety of applications as system identification, echo
cancelation, noise cancelation, equalization, control, deconvolution, change detec-
tion,smartantennas,speechprocessing,datacompression,radar,sonar,biomedicine,
seismology,etc.[1,2].Certainly,thiswouldnothavebeenpossiblewithouttheVLSI
technologynecessarytoprovidethecomputationalrequirementsfortheseapplica-
tions[3].
A filter becomes adaptive when its coefficients are updated (using a particular
algorithm) each time a new sample of the signals is available. Conceptually, the
algorithmchosenshouldbeabletofollowtheevolutionofthesystemunderstudy.
Thisprovidestheadaptivefilterwiththeabilitytooperateinrealtimeandimproves
itsperformancewithnoneedtohaveanyusersinvolved[2].
Figure1.1showsaschemeofanadaptivefilteringproblem.Basically,theadaptive
filterreceivesavectorinputsignalx(n)andareference/desiredsignald(n).1Attime
1Unlessotherwisestated,wefocusinthisbookonfiniteimpulseresponse(FIR)filters,andrandom
signalsbeingtime-discrete,real-valued,andzero-mean.
L.ReyVegaandH.Rey,ARapidIntroductiontoAdaptiveFiltering, 1
SpringerBriefsinElectricalandComputerEngineering,
DOI:10.1007/978-3-642-30299-2_1,©TheAuthor(s)2013
