Artificial Neural Networks: Methods and Applications in Bio-/Neuroinformatics

487 Pages·2015·19.43 MB·English

by  Petia Koprinkova-Hristova

Springer Series in Bio-/Neuroinformatics 4
Petia Koprinkova-Hristova
Valeri Mladenov
Nikola K. Kasabov Editors
Artificial Neural
Networks
Methods and Applications
in Bio-/Neuroinformatics
Springer Series in Bio-/Neuroinformatics
Volume 4
Serieseditor
NikolaK.Kasabov,AucklandUniversityofTechnology,Auckland,NewZealand
e-mail:[e-mail]
Aims&Scope
SpringerSeriesinBio-/Neuroinformaticspublishescontentonfundamentalprinci-
plesaswellasstate-of-arttheories,methodsandapplicationsintherapidlyevolv-
ing fields of bioinformatics and neuroinformatics. The series is unique in that it
integrates three interdependent disciplines, namely information science, bioinfor-
matics and neuroinformatics. The series covers general informatics methods and
techniques,likedatamining,databasemanagementsystems,machinelearning,ar-
tificial neuralnetworks, evolutionarycomputation,chaostheory,quantumcompu-
tation,statistical methods,andimageandsignalprocessing,aswellastheirappli-
cationstodifferentareasofresearch,includingbigdata,functionalanalysisofthe
brain,computationalsystemsbiology,computationalsystemsneuroscience,health
informatics, personalizedmedicine, rehabilitationmedicine, industrial biotechnol-
ogy, and many more. The series publishes monographs, contributed volumes and
conferenceproceedings,aswellasadvancedtextbooks.
AdvisoryBoard
Shun-ichiAmari,RIKENBrainScienceInstitute,Japan
PaoloAvesani,FondazioneBrunoKessler,Trento,Italy
LubicaBenuskova,UniversityofOtago,NewZealand
ChrisM.Brown,UniversityofOtago,NewZealand
RichardJ.Duro,UniversityofCoruña,Spain
PetiaGeorgieva,UniversityofAveiro,Portugal
KaizhuHaung,ChineseAcademyofSciences,Beijing,China
Zeng-GuangHou,ChineseAcademyofSciences,Beijing,China
GiacomoIndiveri,ETHandUniversityofZurich,Switzerland
IrwinKing,TheChineseUniversityofHongKong
HiroshiKojima,TamagawaUniversity,Japan
RobertKozma,TheUniversityofMemphis,USA
AndreasKönig,TechnicalUniversityofKeiserlautern,Germany
DaniloMandic,ImperialCollegeLondon,UnitedKingdom
FrancescoMasulli,UniversityofGenova,Italy
MartinMcGinnity,UniversityofUlster,N.Ireland
HeikeSichtig,UniversityofFlorida,Gainesville,Florida,USA
Jean-PhilippeThivierge,UniversityofOttawa,Canada
ShiroUsui,RIKENBrainScienceInstitute,Japan
AlessandroE.P.Villa,UniversityofLausanne,Switzerland
JieYang,ShanghaiJiaotongUniversity,China
Moreinformationaboutthisseriesathttp://www.springer.com/series/10088
·
Petia Koprinkova-Hristova Valeri Mladenov
Nikola K. Kasabov
Editors
Artificial Neural Networks
Methods and Applications
in Bio-/Neuroinformatics
ABC
Editors
PetiaKoprinkova-Hristova NikolaK.Kasabov
InstituteofInformationandCommunication AucklandUniversityofTechnology
Technologies Auckland
BulgarianAcademyofSciences NewZealand
Sofia
Bulgaria
ValeriMladenov
TechnicalUniversitySofia
Sofia
Bulgaria
ISSN2193-9349 ISSN2193-9357 (electronic)
ISBN978-3-319-09902-6 ISBN978-3-319-09903-3 (eBook)
DOI10.1007/978-3-319-09903-3
LibraryofCongressControlNumber:2014945765
SpringerChamHeidelbergNewYorkDordrechtLondon
(cid:2)c SpringerInternationalPublishingSwitzerland2015
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub-
lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany
errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect
tothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
The edited bookincludeschaptersthatpresentselected and extendedpapersfrom
theInternationalConferenceonArtificialNeuralNetworks(ICANN)2013.Founded
in1991,theICANNbecomethePremierannualconferenceoftheEuropeanNeural
NetworkSociety.Itsmaingoalistobringtogetherandtofacilitatecontactsbetween
researchersfrominformationsciencesandneurosciencesandtoprovideahigh-level
internationalforumforbothacademicandindustrialcommunities.
The selected and invited to the present book chapters were selected among
ICANN papers that received highest scores during the strong peer review assess-
ment(almost40%rejectionrate)andamongselectedbysessionchairsbestpresen-
tations.
The collected in the book chapters are presented in topical sections that cover
wide range of contemporary topics varying from neural network theory and
models, machine learning and learning algorithms, brain-machine interaction and
bio-inspiredsystems,patternrecognitionandclassificationaswellasvariousappli-
cations.
Thebookchaptersincludenewtheoreticaldevelopmentsinrecurrentneuralnet-
works and reservoir computing, new and improved training algorithms for Deep
Boltzmann Machines (DBM), tapped delay feedforward architectures and kernel
machines, reinforcement learning and Adaptive Critic Designs (ACD), new bio-
inspired models and architectures related to cell assembly mechanisms, visual
perception and natural language understanding,new and improvedalgorithms for
pattern recognition with applications to gesture classification, handwritten digit
recognitionandtimeseriesforecasting.
The book will be of interest to all researchersand postgraduatestudents in the
area of computational intelligence, applied mathematics, computer science, engi-
neering,neuroscience,andotherrelatedareas.
June2014 Editors
SofiaandAuckland PetiaKoprinkova-Hristova
ValeriMladenov
NikolaK.Kasabov
Contents
Neural Networks Theory and Models
Recurrent Neural Networks and Super-Turing Interactive
Computation .................................................. 1
JérémieCabessa,AlessandroE.P.Villa
ImageClassificationwithNonnegativeMatrixFactorizationBasedon
SpectralProjectedGradient ..................................... 31
RafałZdunek,AnhHuyPhan,AndrzejCichocki
Energy-TimeTradeoffinRecurrentNeuralNets .................... 51
JirˇíŠíma
AnIntroductiontoDelay-CoupledReservoirComputing ............. 63
JohannesSchumacher,HazemToutounji,GordonPipa
Double-LayerVectorPerceptronforBinaryPatternsRecognition...... 91
VladimirKryzhanovskiy,IrinaZhelavskaya
Local Detectionof Communities by AttractorNeural-Network
Dynamics ..................................................... 115
HiroshiOkamoto
LearningGestaltFormationsforOscillatorNetworks ................ 127
MartinMeier,RobertHaschke,HelgeJ.Ritter
Analysingthe Multiple TimescaleRecurrentNeuralNetworkfor
EmbodiedLanguageUnderstanding .............................. 149
StefanHeinrich,SvenMagg,StefanWermter
LearningtoLookandLookingtoRemember:ANeural-Dynamic
EmbodiedModelforGenerationofSaccadicGazeShiftsandMemory
Formation .................................................... 175
YuliaSandamirskaya,TobiasStorck
VIII Contents
New Machine Learning Algorithms forNeural Networks
HowtoPretrainDeepBoltzmannMachinesinTwoStages ............ 201
KyunghyunCho,TapaniRaiko,AlexanderIlin,JuhaKarhunen
TrainingDynamicNeuralNetworksUsingthe ExtendedKalman
FilterforMulti-Step-AheadPredictions............................ 221
ArtemChernodub
LearningasConstraintReactions................................. 245
GiorgioGnecco,MarcoGori,StefanoMelacci,MarcelloSanguineti
Baseline-FreeSamplinginParameterExploringPolicyGradients:
SuperSymmetricPGPE......................................... 271
FrankSehnke,TingtingZhao
Sparse Approximations to Value Functions in Reinforcement
Learning...................................................... 295
HunorS.Jakab,LehelCsató
NeuralNetworksSolutionofOptimalControlProblemswithDiscrete
TimeDelaysandTime-DependentLearningofInfinitesimalDynamic
System ....................................................... 315
TiborKmet,MariaKmetova
PatternRecognition, Classificationand OtherNeural
NetworkApplications
Applying PrototypeSelection and Abstraction Algorithms for
EfficientTime-SeriesClassification................................ 333
StefanosOugiaroglou,LeonidasKaramitopoulos,Christos Tatoglou,
GeorgiosEvangelidis,DimitrisA.Dervos
EnforcingGroupStructurethroughtheGroupFusedLasso........... 349
CarlosM.Alaíz,ÁlvaroBarbero,JoséR.Dorronsoro
IncrementalAnomaly Identificationin Flight Data Analysis by
AdaptedOne-ClassSVMMethod................................. 373
DenisKolev,MikhailSuvorov,EvgeniyMorozov,GareginMarkarian,
PlamenAngelov
InertialGestureRecognitionwithBLSTM-RNN .................... 393
Grégoire Lefebvre, Samuel Berlemont, Franck Mamalet,
ChristopheGarcia
OnlineRecognitionofFixations,Saccades,andSmoothPursuitsfor
AutomatedAnalysisofTrafficHazardPerception ................... 411
Enkelejda Kasneci, Gjergji Kasneci, Thomas C. Kübler,
WolfgangRosenstiel
Contents IX
Input Transformationand Output Combination for Improved
HandwrittenDigitRecognition ................................... 435
Juan M. Alonso-Weber, M. Paz Sesmero, German Gutierrez,
AgapitoLedezma,AraceliSanchis
FeatureSelectionforIntervalForecastingofElectricityDemandTime
SeriesData.................................................... 445
MashudRana,IrenaKoprinska,AbbasKhosravi
Stacked Denoising Auto-Encoders for Short-Term Time Series
Forecasting.................................................... 463
PabloRomeu,FranciscoZamora-Martínez,PalomaBotella-Rocamora,
JuanPardo
AuthorIndex ..................................................... 487
Recurrent Neural Networks
and Super-Turing Interactive Computation
Je´re´mieCabessaandAlessandroE.P.Villa
Abstract.Wepresentacompleteoverviewofthecomputationalpowerofrecurrent
neural networks involved in an interactive bio-inspired computational paradigm.
More precisely, we recall the results stating that interactive rational- and real-
weightedneuralnetworksareTuring-equivalentandsuper-Turing,respectively.We
furtherprovethatinteractiveevolvingneuralnetworksaresuper-Turing,irrespective
of whethertheir synaptic weights are modeledby rationalor real numbers. These
resultsshowthatthecomputationalpowersofneuralnetsinvolvedinaclassicalor
in an interactivecomputationalframeworkfollowsimilar patternsof characteriza-
tion.Theysuggestthatsomeintrinsiccomputationalcapabilitiesofthebrainmight
liebeyondthescopeofTuring-equivalentmodelsofcomputation,hencesurpassthe
potentialitieseverycurrentstandardartificialmodelsofcomputation.
1 Introduction
Understanding the computational and dynamical capabilities of biological neural
networks represents an issue of central importance. In this context, much interest
has been focused on comparing the computational powers of diverse theoretical
neuralmodelswith those of abstract computingdevices.Nowadays, the computa-
tional capabilitiesof neuralmodels is knownto be tightly related to the nature of
the activation functionof the neurons,to the nature of their synaptic connections,
totheeventualpresenceofnoiseinthemodel,tothepossibilityforthenetworksto
evolveovertime,andtothecomputationalparadigmperformedbythenetworks.
Je´re´mieCabessa
LaboratoryofMathematicalEconomics(LEMMA),UniversityofParis2–Panthe´on-Assas,
4RueBlaiseDesgoffe,75006Paris,France
e-mail:[e-mail]
AlessandroE.P.Villa
NeuroheuristicResearchGroup,FacultyofBusinessandEconomics,UniversityofLausanne,
CH-1015Lausanne,Switzerland
e-mail:[e-mail]
(cid:2)c SpringerInternationalPublishingSwitzerland2015 1
P.Koprinkova-Hristovaetal.(eds.),ArtificialNeuralNetworks,
SpringerSeriesinBio-/Neuroinformatics4,DOI:10.1007/978-3-319-09903-3_1