Handbook of Neural Network Signal Processing (Electrical Engineering & Applied Signal Processing Series)

384 Pages·2001·11.11 MB·English

by  Yu Hen Hu

Handbook
of
NEURAL
NETWORK
SIGNAL
PROCESSING
THE ELECTRICAL ENGINEERING
AND APPLIED SIGNAL PROCESSING SERIES
Edited by Alexander Poularikas
The Advanced Signal Processing Handbook:
Theory and Implementation for Radar, Sonar,
and Medical Imaging Real-Time Systems
Stergios Stergiopoulos
The Transform and Data Compression Handbook
K.R. Rao and P.C. Yip
Handbook of Multisensor Data Fusion
David Hall and James Llinas
Handbook of Neural Network Signal Processing
Yu Hen Hu and Jenq-Neng Hwang
Handbook of Antennas in Wireless Communications
Lal Chand Godara
Forthcoming Titles
Propagation Data Handbook for Wireless Communications
Robert Crane
The Digital Color Imaging Handbook
Guarav Sharma
Applications in Time Frequency Signal Processing
Antonia Papandreou-Suppappola
Noise Reduction in Speech Applications
Gillian Davis
Signal Processing in Noise
Vyacheslav Tuzlukov
Electromagnetic Radiation and the Human Body:
Effects, Diagnosis, and Therapeutic Technologies
Nikolaos Uzunoglu and Konstantina S. Nikita
Digital Signal Processing with Examples in MATLAB®
Samuel Stearns
Smart Antennas
Lal Chand Godara
Pattern Recognition in Speech and Language Processing
Wu Chou and Bing Huang Juang
Handbook
of
NEUR AL
NETWORK
SIGNAL
PROCESSING
Edited by
YU HEN HU
JENQ-NENG HWANG
CRC PR ESS
Boca Raton London New York Washington, D.C.
disclaimer Page 1 Wednesday, August 1, 2001 10:12 AM
Library of Congress Cataloging-in-Publication Data
Handbook of neural network signal processing / editors, Yu Hen Hu, Jenq-Neng Hwang.
p. cm.— (Electrical engineering and applied signal processing (Series))
Includes bibliographical references and index.
ISBN 0-8493-2359-2
1. Neural networks (Computer science)—Handbooks, manuals, etc. 2. Signal
processing—Handbooks, manuals, etc. I. Hu, Yu Hen. II. Hwang, Jenq-Neng. III.
Electrical engineering and signal processing series.
QA76.87 H345 2001
006.3′2—dc21 2001035674

This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with
permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish
reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials
or for the consequences of their use.
Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical,
including photocopying, microfilming, and recording, or by any information storage or retrieval system, without prior
permission in writing from the publisher.
All rights reserved. Authorization to photocopy items for internal or personal use, or the personal or internal use of specific
clients, may be granted by CRC Press LLC, provided that $1.50 per page photocopied is paid directly to Copyright clearance
Center, 222 Rosewood Drive, Danvers, MA 01923 USA. The fee code for users of the Transactional Reporting Service is
ISBN 0-8493-2359-2/01/$0.00+$1.50. The fee is subject to change without notice. For organizations that have been granted
a photocopy license by the CCC, a separate system of payment has been arranged.
The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works,
or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying.
Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation, without intent to infringe.
Visit the CRC Press Web site at www.crcpress.com
© 2002 by CRC Press LLC
No claim to original U.S. Government works
International Standard Book Number 0-8493-2359-2
Library of Congress Card Number 2001035674
Printed in the United States of America 1 2 3 4 5 6 7 8 9 0
Printed on acid-free paper
Preface
Thefieldofartificialneuralnetworkshasmadetremendousprogressinthepast20yearsinterms
of theory, algorithms, and applications. Notably, the majority of real world neural network appli-
cationshaveinvolvedthesolutionofdifficultstatisticalsignalprocessingproblems. Comparedto
conventionalsignalprocessingalgorithmsthataremainlybasedonlinearmodels, artificialneural
networks offer an attractive alternative by providing nonlinear parametric models with universal
approximation power, as well as adaptive training algorithms. The availability of such powerful
modelingtoolsmotivatednumerousresearcheffortstoexplorenewsignalprocessingapplications
ofartificialneuralnetworks. Duringthecourseoftheresearch,manyneuralnetworkparadigmswere
proposed. Some of them are merely reincarnations of existing algorithms formulated in a neural
network-likesetting, whiletheothersprovidenewperspectivestowardsolvingnonlinearadaptive
signalprocessing. Moreimportantly,thereareanumberofemergentneuralnetworkparadigmsthat
havefoundsuccessfulrealworldapplications.
The purpose of this handbook is to survey recent progress in artificial neural network theory,
algorithms (paradigms) with a special emphasis on signal processing applications. We invited a
panelofinternationallywellknownresearcherswhohaveworkedonboththeoryandapplicationsof
neuralnetworksforsignalprocessingtowriteeachchapter. Thereareatotalof12chaptersplusone
introductorychapterinthishandbook. Thechaptersarecategorizedintothreegroups. Thefirstgroup
containsin-depthsurveysofrecentprogressinneuralnetworkcomputingparadigms. Itcontainsfive
chapters, including the introduction, that deal with multilayer perceptrons, radial basis functions,
kernel-basedlearning,andcommitteemachines. Thesecondpartofthishandbooksurveystheneural
networkimplementationsofimportantsignalprocessingproblems. Thispartcontainsfourchapters,
dealing with a dynamic neural network for optimal signal processing, blind signal separation and
blinddeconvolution,aneuralnetworkforprincipalcomponentanalysis,andapplicationsofneural
networks to time series predictions. The third part of this handbook examines signal processing
applications and systems that use neural network methods. This part contains chapters dealing
withapplicationsofartificialneuralnetworks(ANNs)tospeechprocessing,learningandadaptive
characterization of visual content in image retrieval systems, applications of neural networks to
biomedicalimageprocessing,andahierarchicalfuzzyneuralnetworkforpatternclassification.
The theory and design of artificial neural networks have advanced significantly during the past
20years. Muchofthatprogresshasadirectbearingonsignalprocessing. Inparticular,thenonlinear
natureofneuralnetworks,theabilityofneuralnetworkstolearnfromtheirenvironmentsinsuper-
visedand/orunsupervisedways,aswellastheuniversalapproximationpropertyofneuralnetworks
makethemhighlysuitedforsolvingdifficultsignalprocessingproblems.
Fromasignalprocessingperspective,itisimperativetodevelopaproperunderstandingofbasic
neural network structures and how they impact signal processing algorithms and applications. A
challengeinsurveyingthefieldofneuralnetworkparadigmsistodistinguishthoseneuralnetwork
structuresthathavebeensuccessfullyappliedtosolverealworldproblemsfromthosethatarestill
under development or have difficulty scaling up to solve realistic problems. When dealing with
signalprocessingapplications,itiscriticaltounderstandthenatureoftheproblemformulationso
thatthemostappropriateneuralnetworkparadigmcanbeapplied. Inaddition,itisalsoimportant
to assess the impact of neural networks on the performance, robustness, and cost-effectiveness of
signal processing systems and develop methodologies for integrating neural networks with other
signalprocessingalgorithms.
We would like to express our sincere thanks to all the authors who contributed to this hand-
book: Michael T. Manry, Hema Chandrasekaran, and Cheng-Hsiung Hsieh (Chapter 2); Andrew
D.Back(Chapter3);Klaus-RobertMüller,SebastianMika,GunnarRätsch,KojiTsuda,andBern-
hardScholköpf(Chapter4);VolkerTresp(Chapter5);JoseC.Principe(Chapter6);ScottC.Douglas
(Chapter 7); Konstantinos I. Diamantaras (Chapter 8); Yuansong Liao, John Moody, and Lizhong
Wu(Chapter9);ShigeruKatagirig(Chapter10);PaisarnMuneesawang,Hau-SanWong,JoseLay,
and Ling Guan (Chapter 11); Tülay Adali, Yue Wang, and Huai Li (Chapter 12); and Jinshiuh
Taur,Sun-YuanKung,andShang-HungLin(Chapter13). Manyreviewershavecarefullyreadthe
manuscript and provided many constructive suggestions. We are most grateful for their efforts.
TheyareAndrewD.Back,DavidG.Brown,LaiwanChan,KonstantinosI.Diamantaras,Adriana
Dumitras,MarkGirolami,LingGuan,KuldipPaliwal,AmandaSharkey,andJinshiuhTaur.
Wewouldliketothanktheeditor-in-chiefofthisseriesofhandbooks,Dr.AlexanderD.Poularikas,
forhisencouragement. OurmostsincereappreciationtoNoraKonopkaatCRCPressforherinfinite
patienceandunderstandingthroughoutthisproject.
Editors
YuHenHureceivedaB.S.E.E.degreefromNationalTaiwanUniversity,Taipei,Taiwan,in1976.
HereceivedM.S.E.E.andPh.D.degreesinelectricalengineeringfromtheUniversityofSouthern
CaliforniainLosAngeles,in1980and1982,respectively. From1983to1987,hewasanassistant
professorintheelectricalengineeringdepartmentofSouthernMethodistUniversityinDallas,Texas.
HejoinedthedepartmentofelectricalandcomputerengineeringattheUniversityofWisconsinin
Madison,asanassistantprofessorin1987,andheiscurrentlyanassociateprofessor. Hisresearch
interestsincludemultimediasignalprocessing,artificialneuralnetworks,fastalgorithmsanddesign
methodologyforapplicationspecificmicro-architectures,aswellascomputeraideddesigntoolsfor
VLSIusingartificialintelligence. Hehaspublishedmorethan170technicalpapersintheseareas.
His recent research interests have focused on image and video processing and human computer
interface.
Dr.HuisaformerassociateeditorforIEEETransactionsofAcoustic, Speech, andSignalPro-
cessingintheareasofsystemidentificationandfastalgorithms. Heiscurrentlyassociateeditorof
theJournalofVLSISignalProcessing. HeisafoundingmemberoftheNeuralNetworkSignalPro-
cessingTechnicalCommitteeoftheIEEESignalProcessingSocietyandservedascommitteechair
from1993to1996. HeisaformermemberoftheVLSISignalProcessingTechnicalCommitteeof
theSignalProcessingSociety. Recently,heservedasthesecretaryoftheIEEESignalProcessing
Society(1996–1998).
Dr.HuisafellowoftheIEEE.
Jenq-NengHwangholdsB.S.andM.S.degreesinelectricalengineeringfromtheNationalTaiwan
University,Taipei,Taiwan. Aftercompletingtwoyearsofobligatorymilitaryservicesaftercollege,
heenrolledasaresearchassistantattheSignalandImageProcessingInstituteofthedepartmentof
electricalengineeringattheUniversityofSouthernCalifornia,wherehereceivedhisPh.D.degree
inDecember1988. HewasalsoavisitingstudentatPrincetonUniversityfrom1987to1989.
Inthesummerof1989,Dr.HwangjoinedtheDepartmentofElectricalEngineeringoftheUni-
versity of Washington in Seattle, where he is currently a professor. He has published more than
150journalandconferencepapersandbookchaptersintheareasofimage/videosignalprocessing,
computationalneuralnetworks,andmultimediasystemintegrationandnetworking. Hereceivedthe
1995IEEESignalProcessingSociety’sAnnualBestPaperAward(withShyh-RongLayandAlan
Lippman)intheareaofneuralnetworksforsignalprocessing.
Dr.HwangisafellowoftheIEEE.HeservedasthesecretaryoftheNeuralSystemsandApplica-
tionsCommitteeoftheIEEECircuitsandSystemsSocietyfrom1989to1991,andhewasamember
oftheDesignandImplementationofSignalProcessingSystemsTechnicalCommitteeoftheIEEE
SignalProcessingSociety. HeisalsoafoundingmemberoftheMultimediaSignalProcessingTech-
nicalCommitteeoftheIEEESignalProcessingSociety. HeservedasthechairmanoftheNeural
NetworksSignalProcessingTechnicalCommitteeoftheIEEESignalProcessingSocietyfrom1996
to1998,andheiscurrentlytheSociety’srepresentativetotheIEEENeuralNetworkCouncil. He
served as an associate editor for IEEE Transactions on Signal Processing from 1992 to 1994 and
currentlyistheassociateeditorforIEEETransactionsonNeuralNetworksandIEEETransactions
on Circuits and Systems for Video Technology. He is also on the editorial board of the Journal of
VLSISignalProcessingSystemsforSignal,Image,andVideoTechnology. Dr. Hwangwasthecon-
ferenceprogramchairofthe1994IEEEWorkshoponNeuralNetworksforSignalProcessingheldin
Ermioni,GreeceinSeptember1994. Hewasthegeneralco-chairoftheInternationalSymposiumon
ArtificialNeuralNetworksheldinHsinchu,TaiwaninDecember1995. Healsochairedthetutorial
committeefortheIEEEInternationalConferenceonNeuralNetworksheldinWashington,D.C.in
June1996. Hewastheprogramco-chairoftheInternationalConferenceonAcoustics,Speech,and
SignalProcessinginSeattle,Washingtonin1998.
2359/Contributors Page i Thursday, August 2, 2001 12:52 PM
Contributors
Tülay Adali Sun-Yuan Kung Jose C. Principe
University of Maryland Princeton University University of Florida
Baltimore, Maryland Princeton, New Jersey Gainesville, Florida
Andrew D. Back Jose Lay Gunnar Rätsch
Windale Technologies University of Sydney GMD FIRST and University of
Brisbane, Australia Sydney, Australia Potsdam
Berlin, Germany
Hema Chandrasekaran Huai Li
U.S. Wireless Corporation University of Maryland Bernhard Schölkopf
San Ramon, California Baltimore, Maryland Max-Planck-Institut für
Biologische Kybernetik
Konstantinos I. Yuansong Liao Tübingen, Germany
Diamantaras Oregon Graduate Institute of
Technological Education Institute Science and Technology Junshiuh Taur
of Thessaloniki Beaverton, Oregon National Chung-Hsing University
Sindos, Greece Taichung
Shang-Hung Lin Taiwan, China
Scott C. Douglas EPSON Palo Alto Laboratories
Southern Methodist University ERD Volker Tresp
Dallas, Texas Palo Alto, California Siemens AG
Corporate Technology
Ling Guan Michael T. Manry Munich, Germany
University of Sydney University of Texas
Sydney, Australia Arlington, Texas Koji Tsuda
AIST Computational Biology
Cheng-Hsiung Hsieh Sebastian Mika Research Center
Chien Kou Institute of Technology GMD FIRST Tokyo, Japan
Changwa Berlin, Germany
Taiwan, China Yue Wang
John Moody The Catholic Universtiy of
Yu Hen Hu Oregon Graduate Institute of America
University of Wisconsin Science and Technology Washington, DC
Madison, Wisconsin Beaverton, Oregon
Hau-San Wong
Jenq-Neug Hwang Klaus-Robert Müler University of Sydney
University of Washington GMD FIRST and University of Sydney, Australia
Seattle, Washington Potsdam
Berlin, Germany Lizhong Wu
Shigeru Katagiri HNC Software, Inc.
Intelligent Communication Science Paisarn Muneesawang San Diego, California
Laboratories University of Sydney
Kyoto, Japan Sydney, Australia