Table Of ContentINTELLIGENT SYSTEMS REFERENCE LIBRARY
Volume 49
Monica Bianchini
Marco Maggini
Lakhmi C. Jain (Eds.)
Handbook on Neural
Information Processing
123
Intelligent Systems Reference Library 49
Editors-in-Chief
Prof.JanuszKacprzyk Dr.LakhmiC.Jain
SystemsResearchInstitute AdjunctProfessor
PolishAcademyofSciences UniversityofCanberraACT2601
ul.Newelska6 Australia
01-447Warsaw and
Poland UniversityofSouthAustralia
E-mail:[email protected] Adelaide
SouthAustraliaSA5095
Australia
E-mail:[email protected]
Forfurthervolumes:
http://www.springer.com/series/8578
Monica Bianchini, Marco Maggini,
and Lakhmi C. Jain (Eds.)
Handbook on Neural
Information Processing
123
Editors
ProfessorMonicaBianchini Dr.LakhmiC.Jain
DipartimentodiIngegneriadell’Informazione AdjunctProfessor
Universita`degliStudidiSiena UniversityofCanberraACT2601
Siena Australia
Italy
ProfessorMarcoMaggini
DipartimentodiIngegneriadell’Informazione
Universita`degliStudidiSiena
Siena
Italy
ISSN1868-4394 ISSN1868-4408 (electronic)
ISBN978-3-642-36656-7 ISBN978-3-642-36657-4 (eBook)
DOI10.1007/978-3-642-36657-4
SpringerHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2013932221
(cid:2)c Springer-VerlagBerlinHeidelberg2013
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub-
lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany
errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect
tothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Foreword
Neuralinformationprocessingis aninterdisciplinaryfield ofresearchwithcontri-
butionsfromstatistics,computerscience,neuroscience,electricalengineering,cog-
nitivescience,mathematics,andphysics.Researchersinthefieldshareacommon
desireto developandunderstandcomputationalandstatistical strategiesforinfor-
mationprocessing,inthebrainorinartifactsthereof.
Thefieldofneuralinformationprocessingsystemshasseenatremendousgrowth
inthelastcoupleofdecades,bothinvolumeand,moreimportantly,inreputation.It
helpsthatthereisnowaquiteclearseparationbetweencomputationalneuroscience
(modeling the brain) and machine learning (developing artifacts that can “learn”,
possibly but notnecessarily inspired by the way the brain works). This handbook
focusesonthelatter,anddoessowithgreatsuccess.
Althoughtheeditorsdonotclaimtobefullycomprehensive,thehandbookhas
anamazingcoverageofthe importantparadigmsin machinelearning,bysome of
thebestexpertsavailable.Itcontainschaptersgoingallthewayfromrelevantmath-
ematical theory, to state-of-the-artarchitecturesand learning algorithms, to actual
applications.
The nice thing about the chapters is that the authors can take a bit more space
to introduce and explain the relevant concepts than they are normally allowed or
expected to do in journal and conference papers. You can tell that they enjoyed
writingtheircontributions,whichmakesthechaptersapleasuretoread.
In short,the editorsand authorsare to be congratulatedand thankedfor a very
interestingandreadablecontemporaryperspectiveonthestate-of-the-artinthema-
chine learning part of neuralinformationprocessing systems. This handbookwill
be a valuableasset for graduatestudents, research,as well as practitionersfor the
nextyearstocome,untiltherewillhopefullybeanotheredition.Icanhardlywait!
Prof.dr.TomHeskes
InstituteforComputingandInformationSciences
RadboudUniversityNijmegen
Preface
Thishandbookisinspiredbytwofundamentalquestions:”Canintelligentlearning
machinesbe built?” and”Can theybe applied to face problemsotherwise unsolv-
able?”.Asimpleuniqueanswercertainlydoesnotexisttothefirstquestion.Instead
inthelastthreedecades,thegreatamountofresearchinmachinelearninghassuc-
ceededinansweringmanyrelated,butfarmorespecific,questions.Inotherwords,
manyautomatictoolsabletolearninparticularenvironmentshavebeenproposed.
They do not show an ”intelligent” behavior, in the human sense of the term, but
certainly they can help in addressing problemsthat involve a deep perceptualun-
derstandingofsuchenvironments.Therefore,theanswertothesecondquestionis
alsopartialandnotfullysatisfactory,evenifalotofchallengingproblems(compu-
tationallytoohardtobefacedintheclassicalgorithmicframework)canactuallybe
tackledwithmachinelearningtechniques.
Inthisview,thehandbookcollectsbothwell-establishedandnewmodelsincon-
nectionism,togetherwiththeirlearningparadigms,andproposesadeepinspection
of theoreticalpropertiesand advancedapplicationsusing a plain language,partic-
ularly tailored to non experts. Not pretending to be exhaustive, this chapter and
the whole book delineate an evolving picture of connectionism, in which neural
information systems are moving towards approaches that try to keep most of the
informationunalteredandtospecializethemselves,sometimesbasedonbiological
inspiration,tocopeexpertlywithdifficultreal–worldapplications.
Chapters Included inthe Book
Chapterscomposingthisbookcanbelogicallygroupedintothreesections:
(cid:129) NeuralNetworkArchitectures
(cid:129) LearningParadigms
(cid:129) ReasoningandApplications
VIII Preface
Neural NetworkArchitectures
Chapter1,byYoshuaBengioandAaronCourville,givesanoverviewondeeplearn-
ing algorithms, that are able to better capture invariances, and discover non–local
structureindatadistributions.Actually,deeparchitecturesareobtainedbycompos-
ing several levels of distributed representations, as in neural networks with many
layersandinthemammaliancortex.Unsupervisedlearningofrepresentations,ap-
pliedindeeparchitectures,hasbeenfoundusefulinmanyapplicationsandbenefits
from several advantages, e.g. when there are many unlabeled examples and few
labeledones(semi-supervisedlearning),orwhentheexamplescomefromadistri-
bution differentbut related to the one of interest (self–taughtlearning, multi–task
learning,anddomainadaptation).Learningalgorithmsfordeeparchitectureshave
recentlybeproventobeimportantinmodelingrichdata,withthekindofcomplex-
ityexhibitedinimages,video,naturallanguageandotherAItasks.
In Chapter 2, by Sajid A. Marhon, Chistopher J. F. Cameron, and Stefan C.
Kremer,the readersare introducedto thebasic principlesofrecurrentnetworks,a
variantoffeedforwardnetworkswhichadditionallyincludesrecurrentconnections
(loops) between processing elements. Recurrence forces a cyclic temporal depen-
dencybetweentheactivationvaluesoftheprocessingelements,thatinjectsatem-
poraldynamicsintheirevaluation.Theabilitytoevolvetheactivationvaluesover
time makes recurrent networks capable of processing temporal input patterns (as
opposedtothespatialpatternsthatlimitconventionalmodels),generatingtemporal
outputpatterns,andperforminginternal,iterativecomputations.Computationalca-
pabilitiesandlearningabilitiesofrecurrentnetworksareassessed, andreal–world
state–of–the–artapplicationsexplored.
In the supervised framework, Graph Neural Networks (GNNs) are a powerful
toolforprocessingstructureddatain theformofgeneralgraphs.InChapter3, by
MonicaBianchiniandMarcoMaggini,aunifiedframeworkisproposedforlearning
inneuralnetworks,basedontheBackPropagationalgorithm,thatstartsfromGNNs
and graphs, and views each other type of architecture/learning environment (f.i.,
recursivenetworks/trees,recurrentnetworks/sequences,staticnetworks/arrays)asa
particular and simpler subcase of the BackPropagationThroughGraph algorithm.
The derivation of the learning algorithm is described in details, showing how the
adaptationisdrivenbyanerrordiffusionprocessonthegraphstructure.
Chapter4,byLiviuGoras¸,IonVornicu,andPaulUngureanu,introducestheCel-
lular Neural Network (CNN) paradigm, and several analog parallel architectures
inspired by CNNs. CNNs are massive parallel computing architectures defined
in discrete N–dimensional spaces, and represented by regular arrays of elements
(cells). Cells are multipleinput–singleoutputprocessors,describedbyone orfew
parametric functionals. The spatio–temporal dynamics of CNNs, possibly associ-
atedwithimagesensors,canbeusedforhighspeed1Dand2Dsignalprocessing,
including linear and nonlinear filtering, and feature extraction. The VLSI imple-
mentabilityofCNNsinstandardCMOStechnologyisalsoexplored.
InChapter5,byPaulC.Kainen,VeˇraKu˚rkova´,andMarcelloSanguineti,tools
fromnonlinearapproximationtheoryareusedtoderiveusefulestimatesofnetwork
Preface IX
complexity,whichdependsonthetypeofcomputationalunitsandtheinputdimen-
sion.Majortoolsaredescribedforestimatingtheratesofdecreaseofapproximation
errorswithincreasingmodelcomplexity.Propertiesofbestapproximationarealso
discussed. Finally, recent results concerningthe dependenceof modelcomplexity
ontheinputdimensionaredescribed,togetherwithexamplesofmultivariablefunc-
tionsthatcanbetractablyapproximated.
Aneuralnetworkutilizesdatatofindafunctionconsistentwiththedataandwith
furtherconceptualdesiderata,suchasdesiredsmoothness,boundednessorintegra-
bility.Theweightsandthefunctionsembodiedin thehiddenunitscan bethought
ofasdeterminingafinite sumthatapproximatessomefunction.Thisfinitesumis
a kind of quadraturefor an integralformulathat would representthe functionex-
actly.Inthelastchapterofthissection(Chapter6),authoredbyPaulC.Kainenand
AndrewVogt,Bochnerintegrationisappliedtoneuralnetworks,inordertounder-
standtheirapproximationcapabilitiesandtomakeeffectivechoicesofthenetwork
parameters. By modifying the traditional focus on the data to the construction of
a family of functions able to approximate such data, gives not only a deep theo-
reticalinsight,butalsohelpsinconstructingnetworkscapableofperformingmore
sophisticatedtasks.
Learning Paradigms
Intraditionalsupervisedlearning,labeleddataareusedtobuildamodel.However,
labeling training data for real–world applications is difficult, expensive and time
consuming,asitrequirestheeffortsofhumanswithaspecificdomainexperience.
This is especially true for applications that involve learning with a large number
ofclasses,andsometimeswithsimilaritiesamongthem.Semi–supervisedlearning
addresses this inherent bottleneck by allowing the model to combine labeled and
unlabeled data to boost the performance. In Chapter 7, by Mohamed Farouk Ab-
del Hady and Friedhelm Schwenker, an overview of recent research advances in
semi–supervisedlearningisprovided,whereasitscombinationwithactivelearning
techniquesisexplored.
Relational learning refers to learning from data that have a complex structure.
Thisstructuremaybeeitherinternal(adatainstancemaybecomplex)orexternal
(relationshipsamongdata).Statisticalrelationallearningreferstotheuseofstatisti-
callearningmethodsinarelationallearningcontext.InChapter8,byHendrikBloc-
keel,statisticalrelationallearningispresentedinitsconcreteforms(learningfrom
graphs,learningfromlogicalinterpretations,learningfromrelationaldatabases)and
state–of–the–art methods in this framework — such as inductive logic program-
ming,relationalneuralnetworks,andprobabilisticlogicalmodels—areextensively
reviewed.
Kernel methods are a class of non–parametric learning techniques relying on
kernels.Akernelgeneralizesdotproductstoarbitrarydomainsandcanthusbeseen
as a similarity measure between data points with complex structures. Key to the
successofanykernelmethodisthedefinitionofanappropriatekernelforthedata
at hand. A well–designed kernelshould capture the aspects characterizingsimilar
X Preface
instances, while being computationally efficient. Kernels have been designed for
sequences,treesandgraphs,aswellasarbitraryrelationaldatarepresentedinfirstor
higherorderlogic.InChapter9,byAndreaPasserini,thebasicprinciplesunderlying
kernelmachinesarerevised,togetherwithsomeofthemostpopularapproachesthat
haverecentlybeendeveloped.Finally,kernelmethodsforpredictingstructuresare
alsopresented.Thesealgorithmsdealwithstructured–outputprediction,alearning
frameworkinwhichtheoutputisitselfastructurethathastobepredictedfromthe
inputone.
InChapter10,byFrancescoGargiulo,ClaudioMazzariello,andCarloSansone,
asurveyonmultipleclassifiersystemsispresented.IntheareaofPatternRecogni-
tion,thisideahasbeenproposedbasedontherationalethattheconsensusofaset
ofclassifiersmaycompensatefortheweaknessofa singleone,whileeachclassi-
fierpreservesitsownstrength.Classifierscouldactuallybeconstructedfollowinga
varietyofclassificationmethodologies,andtheycouldachievedifferentratesofcor-
rectlyclassifiedsamples.Moreover,differentcombiningrulesandstrategies,inde-
pendentoftheadoptedclassificationmodel,havebeenproposedinliterature.Some
of the currently available tools (followingdifferentapproaches) for implementing
multipleclassifiersystemsarereviewedinthischapter,togetherwithataxonomyof
real–worldapplications.
Modallearninginneuralcomputingreferstothestrategiccombinationsofmodes
of adaptation and learning within a single artificial neural network architecture.
Modes,inthiscontext,arelearningmethods,andtwoormoremodesmayproceed
inparallelindifferentpartsoftheneuralcomputingstructure(layersandneurons)
or, alternatively, can be applied to the same part of the structure with a mecha-
nism for allowingthe networkto switch betweenmodes.Chapter 11,by Dominic
Palmer–BrownandChrisinaDraganova,presentsanintroductiontomodallearning
techniques,withaparticularfocusonmodalself–organizationmethods,appliedto
groupinglearners’responsestomultiplechoicequestions,naturallanguagephrase
recognitionandpatternclassification.Algorithms,datasetdescriptions,pseudocode
andMatlabcodeareincluded.
Reasoning and Applications
Bayesian networks represent a widely accepted model for reasoning with uncer-
tainty. In Chapter 12, by Wim Wiegerink, Willem Burgers, and Bert Kappen,
Bayesiannetworksareapproachedfromapracticalpointofview,puttingemphasis
onmodellingandpracticalapplications.AshorttheoreticintroductiontoBayesian
networksisprovidedandsomeoftheirtypicalusagesarereported,e.g.forreason-
ing and diagnostics. Furthermore, peculiar network behaviors are described, such
astheexplainingawayphenomenon.Finally,practicalapplicationstovictimiden-
tification by kinship analysis based on DNA profiles, and to the construction of a
petrophysicaldecision support system are provided, in order to illustrate the data
modellingprocessindetail.
In content based image retrieval (CBIR), relevance feedback is an interactive
process,thatbuildsabridgebetweenusersandasearchengine.Infact,itleadstoa
Preface XI
muchimprovedretrievalperformancebyupdatingqueriesandsimilaritymeasures
according to a user’s preference. Chapter 13, by Jing Li and Nigel M. Allinson,
introducesthebasicelementsofCBIR(fromlow–levelfeatureextractiontoclassi-
ficationmethods)andreviewsrecentlyproposedCBIRtechniques,withaparticular
focusonlong–termlearning,thatallowsthesystemtorecordandcollectfeedback
knowledgefrom differentusers over a variety of querysessions, and it is particu-
larlytailoredformultimediainformationsearching.Somerepresentativeshort–term
learningtechniquesare also presented, in orderto providethe reader with a com-
prehensivereferencesourceforCBIR.
In Chapter 14, authored by Ah Chung Tsoi, Markus Hagenbuchner,Milly Kc,
and ShiJua Zhang, an application of learning in structured domains is presented,
namelyclassification/regressionproblemsontextdocumentsfromlargedocument
collections.Infact,whiletextdocumentsareoftenprocessedasunstructureddata,
the performanceandproblemsolvingcapabilityofmachinelearningmethodscan
be enhanced through the use of suitable structured representations. In particular,
forclassificationproblems,theincorporationoftherelatednessinformation,asex-
pressed by Concept Link Graphs, allows the developmentof tighter clusters with
respecttothebag–of–wordsrepresentation,whereassuchgraphscanalsobeuseful
intheregressionframework,inordertoranktheitemsinalargetextcorpus.
Finally,Chapter15,byMasoodZamaniandStefanC.Kremer,constitutesasur-
vey on the application of connectionist models to bioinformatics. Actually, many
problemsin bioinformaticsinvolve predicting later stages in the informationflow
from earlier ones. Bioinformatics methods capable of such predictions can often
eliminatecostly,difficultortime–consumingtasksinimportantbiologicalresearch.
Forexample,predictingproteinstructureandfunctionbasedontheaminoacidse-
quenceisanessentialcomponentofmoderndrugdesign,andcanreplaceexpensive
wet–labwork.Thischapterisintendedtoprovideanintroductiontothepredominant
researchareasandsomeneuralnetworkapproachesusedwithinbioinformatics.
Conclusions
This chapter has presented an introduction to recent research in neural informa-
tionprocessingparadigms,startingfromconnectionistmodelsrecentlydevelopedto
learngraphs—andshowinghowtheyrepresentapowerfultooltoaddressallthose
problemswheretheinformationisnaturallyorganizedinentitiesandrelationships
amongentities—toclassicalparadigms,theoreticallyinspectedinorderto estab-
lish new significantproperties,and rivisitedfor guaranteeingbetter performances,
and to advanced applications, derived from real–world problems and strongly bi-
ologogically inspired. Not having the presumption to exhaust a so extensive and
evolvingargument,ifeversomethingshouldbeconcludedontheuniverseofcon-
nectionism is that neural information paradigmsare moving towards approaching
avastvarietyofproblems,derivedfromevery–daylifeand,insodoing,theyboth
try to keep most of the information unaltered and to specialize themselves, to at-
tackeachproblemwiththeweaponsofanexpertintheparticularfield.Thisbook
