Table Of ContentLeonardoFranco,DavidA.Elizondo,andJose´M.Jerez(Eds.)
ConstructiveNeuralNetworks
StudiesinComputationalIntelligence,Volume258
Editor-in-Chief
Prof.JanuszKacprzyk
SystemsResearchInstitute
PolishAcademyofSciences
ul.Newelska6
01-447Warsaw
Poland
E-mail:[email protected]
Furthervolumesofthisseriescanbefoundonour
homepage:springer.com Vol.248.CheePengLim,LakhmiC.Jain,and
SatchidanandaDehuri(Eds.)
Vol.237.GeorgeA.PapadopoulosandCostinBadica(Eds.) InnovationsinSwarmIntelligence,2009
IntelligentDistributedComputingIII,2009 ISBN978-3-642-04224-9
ISBN978-3-642-03213-4
Vol.249.WesamAshourBarbakh,YingWu,andColinFyfe
Vol.238.LiNiu,JieLu,andGuangquanZhang
Non-StandardParameterAdaptationforExploratoryData
Cognition-DrivenDecisionSupportforBusinessIntelligence,
Analysis,2009
2009
ISBN978-3-642-04004-7
ISBN978-3-642-03207-3
Vol.239.ZongWooGeem(Ed.) Vol.250.RaymondChiongandSandeepDhakal(Eds.)
HarmonySearchAlgorithmsforStructuralDesign NaturalIntelligenceforScheduling,PlanningandPacking
Optimization,2009 Problems,2009
ISBN978-3-642-03449-7 ISBN978-3-642-04038-2
Vol.240.DimitriPlemenosandGeorgiosMiaoulis(Eds.) Vol.251.ZbigniewW.RasandWilliamRibarsky(Eds.)
IntelligentComputerGraphics2009,2009 AdvancesinInformationandIntelligentSystems,2009
ISBN978-3-642-03451-0 ISBN978-3-642-04140-2
Vol.241.Ja´nosFodorandJanuszKacprzyk(Eds.)
Vol.252.NgocThanhNguyenandEdwardSzczerbicki(Eds.)
AspectsofSoftComputing,IntelligentRoboticsandControl,
IntelligentSystemsforKnowledgeManagement,2009
2009
ISBN978-3-642-04169-3
ISBN978-3-642-03632-3
Vol.242.CarlosArtemioCoelloCoello, Vol.253.AkitoshiHanazawa,TsutomMiki,and
SatchidanandaDehuri,andSusmitaGhosh(Eds.) KeiichiHorio(Eds.)
SwarmIntelligenceforMulti-objectiveProblemsinData Brain-InspiredInformationTechnology,2009
Mining,2009 ISBN978-3-642-04024-5
ISBN978-3-642-03624-8
Vol.254.KyandoghereKyamakya,WolfgangA.Halang,
Vol.243.ImreJ.Rudas,Ja´nosFodor,and
HerwigUnger,JeanChamberlainChedjou,
JanuszKacprzyk(Eds.)
NikolaiF.Rulkov,andZhongLi(Eds.)
TowardsIntelligentEngineeringandInformationTechnology,
RecentAdvancesinNonlinearDynamicsand
2009
Synchronization,2009
ISBN978-3-642-03736-8
ISBN978-3-642-04226-3
Vol.244.NgocThanhNguyen,Rados lawPiotrKatarzyniak,
andAdamJaniak(Eds.) Vol.255.CatarinaSilvaandBernardeteRibeiro
NewChallengesinComputationalCollectiveIntelligence, InductiveInferenceforLargeScaleTextClassification,2009
2009 ISBN978-3-642-04532-5
ISBN978-3-642-03957-7
Vol.256.PatriciaMelin,JanuszKacprzyk,and
Vol.245.OlegOkunandGiorgioValentini(Eds.) WitoldPedrycz(Eds.)
ApplicationsofSupervisedandUnsupervisedEnsemble Bio-inspiredHybridIntelligentSystemsforImageAnalysis
Methods,2009 andPatternRecognition,2009
ISBN978-3-642-03998-0 ISBN978-3-642-04515-8
Vol.246.ThanasisDaradoumis,SantiCaballe´,
Vol.257.OscarCastillo,WitoldPedrycz,and
JoanManuelMarque`s,andFatosXhafa(Eds.)
JanuszKacprzyk(Eds.)
IntelligentCollaborativee-LearningSystemsand
EvolutionaryDesignofIntelligentSystemsinModeling,
Applications,2009
SimulationandControl,2009
ISBN978-3-642-04000-9
ISBN978-3-642-04513-4
Vol.247.MonicaBianchini,MarcoMaggini,FrancoScarselli,
andLakhmiC.Jain(Eds.) Vol.258.LeonardoFranco,DavidA.Elizondo,and
InnovationsinNeuralInformationParadigmsand Jose´M.Jerez(Eds.)
Applications,2009 ConstructiveNeuralNetworks,2009
ISBN978-3-642-04002-3 ISBN978-3-642-04511-0
Leonardo Franco,DavidA.Elizondo,
and Jose´ M.Jerez (Eds.)
Constructive Neural Networks
123
LeonardoFranco Jose´M.Jerez
Dept.ofComputerScience Dept.ofComputerScience
UniversityofMalaga UniversityofMalaga
CampusdeTeatinosS/N CampusdeTeatinosS/N
Ma´laga29071 Ma´laga29071
Spain Spain
E-mail:[email protected] E-mail:[email protected]
DavidA.Elizondo
CentreforComputationalIntelligence
SchoolofComputing
DeMontfortUniversity
TheGateway
LeicesterLE19BH
UK
E-mail:[email protected]
ISBN 978-3-642-04511-0 e-ISBN 978-3-642-04512-7
DOI 10.1007/978-3-642-04512-7
Studiesin Computational Intelligence ISSN1860-949X
Library of Congress Control Number:2009937150
(cid:2)c 2009 Springer-VerlagBerlin Heidelberg
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpart
of the material is concerned, specifically therights of translation, reprinting,reuse
ofillustrations, recitation,broadcasting, reproductiononmicrofilm orinanyother
way, and storage in data banks. Duplication of this publication or parts thereof is
permittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,
1965, in its current version, and permission for use must always be obtained from
Springer. Violations are liable to prosecution undertheGerman Copyright Law.
The use of general descriptive names, registered names, trademarks, etc. in this
publication does not imply, even in the absence of a specific statement, that such
names are exempt from the relevant protective laws and regulations and therefore
free for general use.
Typeset&CoverDesign:ScientificPublishing ServicesPvt. Ltd., Chennai, India.
Printed in acid-free paper
9 8 7 6 5 4 3 2 1
springer.com
Preface
This book presents a collection of invited works that consider constructive
methods for neural networks, taken primarily from papers presented at a special
session held during the 18th International Conference on Artificial Neural
Networks (ICANN 2008) in September 2008 in Prague, Czech Republic.
The book is devoted to constructive neural networks and other incremental
learning algorithms that constitute an alternative to the standard method of finding
a correct neural architecture by trial-and-error. These algorithms provide an
incremental way of building neural networks with reduced topologies for
classification problems. Furthermore, these techniques produce not only the
multilayer topologies but the value of the connecting synaptic weights that are
determined automatically by the constructing algorithm, avoiding the risk of
becoming trapped in local minima as might occur when using gradient descent
algorithms such as the popular back-propagation. In most cases the convergence
of the constructing algorithms is guaranteed by the method used.
Constructive methods for building neural networks can potentially create more
compact and robust models which are easily implemented in hardware and used
for embedded systems. Thus a growing amount of current research in neural
networks is oriented towards this important topic.
The purpose of this book is to gather together some of the leading investigators
and research groups in this growing area, and to provide an overview of the most
recent advances in the techniques being developed for constructive neural
networks and their applications.
The first chapter of the book presents a review of existing constructive neural
network algorithms (M. Nicoletti, J. Bertini, D. Elizondo and L. Franco). Next,
four different constructing approaches to solving classification problems are
presented: Muselli and Ferrari introduce a constructing method for switching
functions, Grochowski and Duck focus on a new method for highly complex
functions, Anthony presents a constructive method based on decision lists and the
SONN3 model is analyzed by Horzyk in chapter 5. Nguiph presents in chapter 6
concept lattice-based neural networks, followed in chapter 7 by the work of
Sussner and Esmi who discuss the theory and experiments using morphological
neural networks. Two extensions of constructive algorithms to multiclass
problems are introduced in chapters 8 and 9 by Bertini and Nicoletti and by
Elizondo and Ortiz de Lazcano respectively .
VI Preface
The application of constructive algorithms is used by Franco, Jerez, and
Subirats for active learning in chapter 10, while Ollington,Vamplew and Swanson
explore the use of constructive algorithms in a reinforcement learning framework
in chapter 11. Chapter 12 by Huemer, Elizondo and Góngora shows the
application of a constructive neural network for evolving a machine controller,
followed by the contribution from Satizábal, Pérez-Uribe and Tomassini about
avoiding prototype proliferation. The volume ends with two works on self
organizing neural networks: chapter 14 by Barreto et al. where the parameter
setting in a fuzzy growing network is analysed, while chapter 15 discusses the
method of Inoue on Self-Organizing Neural Grove.
The editors wish to thank all the authors who contributed with their research in
this volume and hope that the current snapshot of some of the latest work in the
field of constructive neural network algorithms help in the further development of
the field.
February, 2009 Leonardo Franco
Málaga, Spain
David A. Elizondo
Leicester, UK
José M. Jerez
Málaga, Spain
Contents
Constructive Neural Network Algorithms for Feedforward
Architectures Suitable for Classification Tasks ............... 1
Maria do Carmo Nicoletti, Joa˜o R. Bertini Jr., David Elizondo,
Leonardo Franco, Jos´e M. Jerez
Efficient Constructive Techniques for Training Switching
Neural Networks ............................................ 25
Enrico Ferrari, Marco Muselli
Constructive Neural Network Algorithms That Solve
Highly Non-separable Problems ............................. 49
Marek Grochowski, Wl(cid:2)odzisl(cid:2)aw Duch
On Constructing Threshold Networks for Pattern
Classification ................................................ 71
Martin Anthony
Self-Optimizing Neural Network 3 ........................... 83
Adrian Horzyk
M-CLANN: Multiclass Concept Lattice-Based Artificial
Neural Network ............................................. 103
Engelbert Mephu Nguifo, Norbert Tsopze, Gilbert Tindo
Constructive Morphological Neural Networks: Some
Theoretical Aspects and Experimental Results in
Classification ................................................ 123
Peter Sussner, Estev˜ao Laureano Esmi
A Feedforward Constructive Neural Network Algorithm
for Multiclass Tasks Based on Linear Separability ........... 145
Jo˜ao Roberto Bertini Jr., Maria do Carmo Nicoletti
VIII Contents
Analysis and Testing of the m-Class RDP Neural Network... 171
David A. Elizondo, Juan M. Ortiz-de-Lazcano-Lobato,
Ralph Birkenhead
Active Learning Using a Constructive Neural Network
Algorithm ................................................... 193
Jos´e L. Subirats, Leonardo Franco, Ignacio Molina, Jos´e M. Jerez
Incorporating Expert Advice into Reinforcement Learning
Using Constructive Neural Networks ........................ 207
Robert Ollington, Peter Vamplew, John Swanson
A Constructive Neural Network for Evolving a Machine
Controller in Real-Time ..................................... 225
Andreas Huemer, David Elizondo, Mario Gongora
Avoiding Prototype Proliferation in Incremental Vector
Quantization of Large Heterogeneous Datasets............... 243
H´ector F. Satiza´bal, Andres P´erez-Uribe, Marco Tomassini
Tuning Parameters in Fuzzy Growing Hierarchical
Self-Organizing Networks.................................... 261
Miguel Arturo Barreto-Sanz, Andr´es P´erez-Uribe,
Carlos-Andres Pen˜a-Reyes, Marco Tomassini
Self-Organizing Neural Grove: Efficient Multiple Classifier
System with Pruned Self-Generating Neural Trees........... 281
Hirotaka Inoue
Author Index................................................ 293
Constructive Neural Network Algorithms for
Feedforward Architectures Suitable for
Classification Tasks
Maria do Carmo Nicoletti, João R. Bertini Jr., David Elizondo,
Leonardo Franco, and José M. Jerez*
Abstract. This chapter presents and discusses several well-known constructive
neural network algorithms suitable for constructing feedforward architectures
aiming at classification tasks involving two classes. The algorithms are divided
into two different groups: the ones directed by the minimization of classification
errors and those based on a sequential model. In spite of the focus being on two-
class classification algorithms, the chapter also briefly comments on the multiclass
versions of several two-class algorithms, highlights some of the most popular
constructive algorithms for regression problems and refers to several other
alternative algorithms.
1 Introduction
Conventional neural network (NN) training algorithms (such as Backpropagation )
require the definition of the NN architecture before learning starts. The common
way for developing a neural network that suits a task consists of defining several
different architectures, training and evaluating each of them, and then choosing
the one most appropriate for the problem based on the error produced between the
target and actual output values. Constructive neural network (CoNN) algorithms,
Maria do Carmo Nicoletti
CS Dept, UFSCar, S. Carlos, SP, Brazil
[email protected]
João R. Bertini Jr.
ICMC, USP, S. Carlos, SP, Brazil
[email protected]
David Elizondo
De Monfort University, Leicester, UK
[email protected]
Leonardo Franco and José M. Jerez
University of Málaga, Málaga, Spain
[email protected],[email protected]
L. Franco et al. (Eds.): Constructive Neural Networks, SCI 258, pp. 1–23.
springerlink.com © Springer-Verlag Berlin Heidelberg 2009
2 M. do Carmo Nicoletti et al.
however, define the architecture of the network along with the learning process.
Ideally CoNN algorithms should efficiently construct small NNs that have good
generalization performance. As commented in (Muselli, 1998), “…the possibility
of adapting the network architecture to the given problem is one of the advantages
of constructive techniques… [This] has also important effects on the convergence
speed of the training process. In most constructive methods, the addition of a new
hidden unit implies the updating of a small portion of weights, generally only
those regarding the neuron to be added”.
The automated design of appropriate neural network architectures can be
approached by two different groups of techniques : evolutionary and non-
evolutionary. In the evolutionary approach, a NN can be evolved by means of an
evolutionary technique, i.e. a population-based stochastic search strategy such as a
GA (see (Schaffer et al., 1992) (Yao, 1999)). In the non-evolutionary approach,
the NN is built not as a result of an evolutionary process, but rather as the result of
a specific algorithm designed to automatically construct it, as is the case with a
constructive algorithm.
CoNN algorithms, however, are not the only non-evolutionary approach to
the problem of defining a suitable architecture for a neural network. The
strategy implemented by the so called pruning methods can also be used and
consists in training a larger than necessary network (which presumably is an easy
task) and then, pruning it by removing some of its connections and/or nodes
(see (Reed, 1993)).
As approached by Lahnajärvi and co-workers in (Lahnajärvi et al., 2002),
pruning algorithms can be divided into two main groups. Algorithms in the first
group estimate the sensitivity of an error function to the removal of an element
(neuron or connection); those with the least effect can be removed. Algorithms in
the second group generally referred to as penalty-term as well as regularization
algorithms, “add terms to the objective function that reward the network for
choosing efficient and small solutions”. The same group of authors also detected
in the literature what they call combined algorithms that take advantage of the
properties of both, constructive and pruning algorithms, in order to determine the
network size in a flexible way (see (Fiesler, 1994), (Gosh & Tumer, 1994)).
There are many different kinds of neural network; new algorithms and
variations of already known algorithms are constantly being published in the
literature. Similarly to other machine learning techniques, neural network
algorithms can also be characterized as supervised, when the target values are
known and the algorithm uses the information, or as unsupervised, when such
information is not given and/or used by the algorithm. The two main classes of
NN architecture are feedforward, where the connections between neurons do not
form cycles, and feedback (or recurrent), where the connections may form cycles.
NNs may also differ in relation to the type of data they deal with; the two more
popular being categorical and quantitative. Both, supervised and unsupervised
learning with categorical targets are referred to as classification. Supervised
learning with quantitative target values is known as regression. Classification
problems can be considered a particular type of regression problems.
