Table Of Content50446Haykin 9/18/06 6:24 AM Page 1
COMPUTER SCIENCE/COMPUTATIONAL NEUROSCIENCE/STATISTICS H N New Directions in
a e New Directions in
y w Statistical Signal Processing
SimonHaykinisUniversityProfessorandDirectorofthe OF RELATED INTEREST k
Adaptive Systems Laboratory at McMaster University. in D FROM SYSTEMS TO BRAINS
, i Statistical Signal Processing
JoséC.PríncipeisDistinguishedProfessorofElectrical Probabilistic Models of the Brain P re
andBiomedicalEngineeringattheUniversityofFlorida, r c
PERCEPTION AND NEURAL FUNCTION ín t edited by Simon Haykin, José C. Príncipe,
Gainesville,whereheisBellSouthProfessorandFounder edited by Rajesh P. N. Rao, Bruno A. Olshausen, and Michael S. Lewicki c io FROM SYSTEMS TO BRAINS Terrence J. Sejnowski, and John McWhirter
and Director of the Computational NeuroEngineering ip n
s
Laboratory. Terrence J. Sejnowski is Francis Crick The topics covered include Bayesian and information-theoretic models of percep- e
, i
Professor, Director of the Computational Neurobiology tion, probabilistic theories of neural coding and spike timing, computational mod- S n
Laboratory, and a Howard Hughes Medical Institute e S
els of lateral and cortico-cortical feedback connections, and the development of j t
Investigator at the Salk Institute for Biological Studies n a
receptive field properties from natural signals. o t edited by Simon Haykin, José C. Príncipe,
andProfessorofBiologyattheUniversityofCalifornia, w is
SanDiego.JohnMcWhirterisSeniorFellowatQinetiQ s t
i Terrence J. Sejnowski, and John McWhirter
Ltd.,Malvern,AssociateProfessorattheCardiffSchool Theoretical Neuroscience k c
i a
,
of Engineering, and Honorary Visiting Professor at COMPUTATIONAL AND MATHEMATICAL MODELING OF NEURAL SYSTEMS a l
S
Queen’sUniversity,Belfast. n
Peter Dayan and L. F. Abbott d ig
M n
Theoretical neuroscience provides a quantitative basis for describing what nervous a
c l
systems do, determining how they function, and uncovering the general principles
W P
by which they operate. This text introduces the basic mathematical and computa- h ro
tional methods of theoretical neuroscience and presents applications in a variety ir c
te e
of areas including vision, sensory-motor integration, development, learning, and r s
, s Signalprocessingandneuralcomputationhavesep-
memory. ed in aratelyandsignificantlyinfluencedmanydisciplines,
i g butthecross-fertilizationofthetwofieldshasbegun
t
o
only recently. Research now shows that each has
r
s much to teach the other, as we see highly sophisti-
cated kinds of signal processing and elaborate hier-
archicallevelsofneuralcomputationperformedside
by side in the brain. In New Directions in Statistical
Signal Processing, leading researchers from both
signal processing and neural computation present
new work that aims to promote interaction between
thetwodisciplines.
TheMITPress The book’s 14 chapters, almost evenly divided
MassachusettsInstituteofTechnology between signal processing and neural computation,
Cambridge,Massachusetts02142 beginwiththebrainandmoveontocommunication,
http://mitpress.mit.edu signalprocessing,andlearningsystems.Theyexam-
ine such topics as how computational models help
0-262-08348-5 us understand the brain’s information processing,
978-0-262-08348-5 how an intelligent machine could solve the “cocktail
partyproblem”with“activeaudition”inanoisyenvi-
ronment, graphical and network structure modeling
approaches,uncertaintyinnetworkcommunications,
the geometric approach to blind signal processing,
NeuralInformationProcessingseries game-theoretic learning algorithms, and observable
operator models (OOMs) as an alternative to hidden
Markovmodels(HMMs).
New Directions in Statistical Signal
Processing
Neural Information Processing Series
Michael I. Jordan and Thomas Dietterich, editors
Advances in Large Margin Classifiers
Alexander J. Smola, Peter L. Bartlett, Bernhard Sch¨olkopf, and Dale Schuurmans, eds.,
2000
Advanced Mean Field Methods: Theory and Practice
Manfred Opper and David Saad, eds., 2001
Probabilistic Models of the Brain: Perception and Neural Function
Rajesh P. N. Rao, Bruno A. Olshausen, and Michael S. Lewicki, eds., 2002
Exploratory Analysis and Data Modeling in Functional Neuroimaging
Friedrich T. Sommer and Andrzej Wichert, eds., 2003
Advances in Minimum Description Length: Theory and Applications
Peter D. Grunwald, In Jae Myung, and Mark A. Pitt, eds., 2005
New Directions in Statistical Signal Processing: From Systems to Brain
Simon Haykin, Jos´e C. Pr´ıncipe, Terrence J. Sejnowski, and John McWhirter, eds., 2006
Nearest-Neighbor Methods in Learning and Vision: Theory and Practice
Gregory Shakhnarovich, Piotr Indyk, and Trevor Darrell, eds., 2006
New Directions in Statistical Signal Processing: From Systems to Brain
Simon Haykin, Jos´e C. Pr´ıncipe, Terrence J. Sejnowski, and John McWhirter, eds., 2007
New Directions in Statistical Signal
Processing: From Systems to Brain
edited by
Simon Haykin
Jos´e C. Pr´ıncipe
Terrence J. Sejnowski
John McWhirter
The MIT Press
Cambridge, Massachusetts
London, England
(cid:2)c 2007MassachusettsInstituteofTechnology
All rights reserved. No part of this book may be reproduced in any form by any electronic
or mechanical means (including photocopying, recording, or information storage and retrieval)
withoutpermissioninwritingfromthepublisher.
PrintedandboundintheUnitedStatesofAmerica
LibraryofCongressCataloging-in-PublicationData
Newdirectionsinstatisticalsignalprocessing;fromsystemstobrain/editedbySimon
Haykin...[etal.].
p.cm. (Neuralinformationprocessingseries)
Includesbibliographicalreferencesandindex.
ISBN10:0-262-08348-5(alk.paper)
ISBN13:978-0-262-08348-5
1.Neuralnetworks(Neurobiology)2.Neuralnetworks(Computerscience)3.Signal
processing—Statisticalmethods.4.Neuralcomputers.I.Haykin,SimonS.,1931–II.
Series.
QP363.3.N52 2006
612.8’2—dc22 2005056210
10987654321
Contents
Series Foreword vii
Preface ix
1 Modeling the Mind: From Circuits to Systems 1
Suzanna Becker
2 Empirical Statistics and Stochastic Models for Visual Signals 23
David Mumford
3 The Machine Cocktail Party Problem 51
Simon Haykin and Zhe Chen
4 Sensor Adaptive Signal Processing of Biological Nanotubes
(Ion Channels) at Macroscopic and Nano Scales 77
Vikram Krishnamurthy
5 Spin Diffusion: A New Perspective in Magnetic Resonance
Imaging 119
Timothy R. Field
6 What Makes a Dynamical System Computationally Powerful? 127
Robert Legenstein and Wolfgang Maass
7 A Variational Principle for Graphical Models 155
Martin J. Wainwright and Michael I. Jordan
8 Modeling Large Dynamical Systems with Dynamical
Consistent Neural Networks 203
Hans-Georg Zimmermann, Ralph Grothmann, Anton Maximilian
Scha¨fer, and Christoph Tietz
9 Diversity in Communication: From Source Coding to
Wireless Networks 243
Suhas N. Diggavi
vi Contents
10 Designing Patterns for Easy Recognition: Information
Transmission with Low-Density Parity-Check Codes 287
Frank R. Kschischang and Masoud Ardakani
11 Turbo Processing 307
Claude Berrou, Charlotte Langlais, and Fabrice Seguin
12 Blind Signal Processing Based on Data Geometric Properties 337
Konstantinos Diamantaras
13 Game-Theoretic Learning 379
Geoffrey J. Gordon
14 Learning Observable Operator Models via the Efficient
Sharpening Algorithm 417
Herbert Jaeger, Mingjie Zhao, Klaus Kretzschmar, Tobias Oberstein,
Dan Popovici, and Andreas Kolling
References 465
Contributors 509
Index 513
Series Foreword
The yearly Neural Information Processing Systems (NIPS) workshops bring to-
getherscientistswithbroadlyvaryingbackgroundsinstatistics,mathematics,com-
puter science, physics, electrical engineering, neuroscience, and cognitive science,
unified by a common desire to develop novel computational and statistical strate-
gies for information processing, and to understand the mechanisms for information
processing in the brain. As opposed to conferences, these workshops maintain a
flexible format that both allows and encourages the presentation and discussion of
workinprogress,andthusservesasanincubatorforthedevelopmentofimportant
new ideas in this rapidly evolving field.
The series editors, in consultation with workshop organizers and members of the
NIPS Foundation Board, select specific workshop topics on the basis of scientific
excellence, intellectual breadth, and technical impact. Collections of papers chosen
and edited by the organizers of specific workshops are built around pedagogical
introductory chapters, while research monographs provide comprehensive descrip-
tions of workshop-related topics, to create a series of books that provides a timely,
authoritativeaccountofthelatestdevelopmentsintheexcitingfieldofneuralcom-
putation.
Michael I. Jordan
Thomas Dietterich
Preface
Inthecourseofsome60to65years,goingbacktothe1940s,signalprocessingand
neuralcomputationhaveevolvedintotwohighlypervasivedisciplines.Intheirown
individual ways, they have significantly influenced many other disciplines. What is
perhaps surprising to see, however, is the fact that the cross-fertilization between
signal processing and neural computation is still very much in its infancy. We only
need to look at the brain and be amazed by the highly sophisticated kinds of
signalprocessingandelaboratehierarchicallevelsofneuralcomputation,whichare
performed side by side and with relative ease.
If there is one important lesson that the brain teaches us, it is summed up
here:
There is much that signal processing can learn from neural computa-
tion, and vice versa.
It is with this aim in mind that in October 2003 we organized a one-week
workshop on “Statistical Signal Processing: New Directions in the Twentieth Cen-
tury,”whichwasheldattheFairmontLakeLouiseHotel,LakeLouise,Alberta.To
fulfill that aim, we invited some leading researchers from around the world in the
two disciplines, signal processing and neural computation, in order to encourage
interactionandcross-fertilizationbetweenthem.Needlesstosay,theworkshopwas
highly successful.
One of the most satisfying outcomes of the Lake Louise Workshop is that it
has led to the writing of this new book. The book consists of 14 chapters, divided
almost equally between signal processing and neural computation. To emphasize,
in some sense, the spirit of the above-mentioned lesson, the book is entitled New
Directions in Statistical Signal Processing: From Systems to Brain.Itisoursincere
hope that in some measurable way, the book will prove helpful in realizing the
original aim that we set out for the Lake Louise Workshop.
Finally,wewishtothankDr.ZheChen,whohadspenttremendouseffortsand
timeinLATEXeditingandproofreadingduringthepreparationandfinalproduction
of the book.
Simon Haykin
Jos´e C. Pr´ıncipe
Terrence J. Sejnowski
John McWhirter
