Multi-Objective Machine Learning

655 Pages·2006·22.615 MB·English

by  Mohua Banerjee, Sushmita Mitra, Ashish Anand (auth.), Yaochu Jin Dr. (eds.)

YaochuJin(Ed.)
Multi-ObjectiveMachineLearning
StudiesinComputationalIntelligence,Volume 16
Editor-in-chief
Prof.JanuszKacprzyk
SystemsResearchInstitute
PolishAcademyofSciences
ul.Newelska6
01-447Warsaw
Poland
E-mail:[email protected]
Furthervolumesofthisseries Vol.8.SrikantaPatnaik,LakhmiC.Jain,
canbefoundonourhomepage: SpyrosG.Tzafestas,GermanoResconi,
AmitKonar(Eds.)
springer.com
InnovationsinRobotMobilityandControl,
2006
Vol.1.TetsuyaHoya ISBN3-540-26892-8
ArtificialMindSystem–KernelMemory
Vol.9.TsauYoungLin,SetsuoOhsuga,
Approach,2005
Churn-JungLiau,XiaohuaHu(Eds.)
ISBN3-540-26072-2
FoundationsandNovelApproachesinData
Vol.2.SamanK.Halgamuge,LipoWang Mining,2005
(Eds.) ISBN3-540-28315-3
ComputationalIntelligenceforModelling
Vol.10.AndrzejP.Wierzbicki,Yoshiteru
andPrediction,2005
Nakamori
ISBN3-540-26071-4
CreativeSpace,2005
Vol.3.Boz˙enaKostek ISBN3-540-28458-3
Perception-BasedDataProcessingin
Vol.11.AntoniLigeza
Acoustics,2005 c
LogicalFoundationsforRule-Based
ISBN3-540-25729-2
Systems,2006
Vol.4.SamanK.Halgamuge,LipoWang ISBN3-540-29117-2
(Eds.)
Vol.12.JonathanLawry
ClassificationandClusteringforKnowledge
ModellingandReasoningwithVague
Discovery,2005
Concepts,2006
ISBN3-540-26073-0
ISBN0-387-29056-7
Vol.5.DaRuan,GuoqingChen,EtienneE.
Vol.13.NadiaNedjah,AjithAbraham,
Kerre,GeertWets(Eds.)
LuizadeMacedoMourelle(Eds.)
IntelligentDataMining,2005
GeneticSystemsProgramming,2006
ISBN3-540-26256-3
ISBN3-540-29849-5
Vol.6.TsauYoungLin,SetsuoOhsuga,
Vol.14.SpirosSirmakessis(Ed.)
Churn-JungLiau,XiaohuaHu,Shusaku
AdaptiveandPersonalizedSemanticWeb,
Tsumoto(Eds.)
2006
FoundationsofDataMiningandKnowledge
ISBN3-540-30605-6
Discovery,2005
ISBN3-540-26257-1 Vol.15.LeiZhiChen,SingKiongNguang,
XiaoDongChen
Vol.7.BrunoApolloni,AshishGhosh,Ferda
ModellingandOptimizationof
Alpaslan,LakhmiC.Jain,SrikantaPatnaik
BiotechnologicalProcesses,2006
(Eds.)
ISBN3-540-30634-X
MachineLearningandRobotPerception,
2005 Vol.16.YaochuJin(Ed.)
ISBN3-540-26549-X Multi-ObjectiveMachineLearning,2006
ISBN3-540-30676-5
Yaochu Jin
(Ed.)
Multi-Objective
Machine Learning
ABC
Dr.YaochuJin
HondaResearchInstitute
EuropeGmbH
Carl-Legien-Str.30
63073Offenbach
Germany
E-mail:[email protected]
LibraryofCongressControlNumber:2005937505
ISSNprintedition:1860-949X
ISSNelectronicedition:1860-9503
ISBN-10 3-540-30676-5SpringerBerlinHeidelbergNewYork
ISBN-13 978-3-540-30676-4SpringerBerlinHeidelbergNewYork
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broadcasting,
reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,
1965,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Violationsare
liableforprosecutionundertheGermanCopyrightLaw.
SpringerisapartofSpringerScience+BusinessMedia
springer.com
(cid:1)c Springer-VerlagBerlinHeidelberg2006
PrintedinTheNetherlands
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnotimply,
evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotectivelaws
andregulationsandthereforefreeforgeneraluse.
Typesetting:bytheauthorsandtechbooksusingaSpringerLATEXmacropackage
Coverdesign:design&productionGmbH,Heidelberg
Printedonacid-freepaper SPIN:11399346 89/techbooks 543210
To Fanhong, Robert and Zewei
Preface
Featureselectionandmodelselectionaretwomajorelementsinmachinelearn-
ing. Both feature selection and model selection are inherently multi-objective
optimization problems where more than one objective has to be optimized.
For example in feature selection, minimization of the number of features and
maximization of feature quality are two common objectives that are likely
conflictingwitheachother.Itisalsowidelyrealizedthatonehastodealwith
thetrade-offbetweenapproximationorclassificationaccuracyandmodelcom-
plexity in model selection.
Traditional machine learning algorithms try to satisfy multiple objectives
by combining the objectives into a scalar cost function. A good example is
the training of neural networks, where the main target is to minimize a cost
function that accounts for the approximation or classification error on given
training data. However, reducing the training error often leads to overfitting,
which means that the error on unseen data will become very large, though
the neural network performs perfectly on the training data. To improve the
generalization capability of neural networks, i.e., to improve their ability to
perform well on unseen data, a regularization term, e.g., the complexity of
neural networks weighted by a hyper-parameter (regularization coefficient)
has to be included in the cost function. One major challenge to implement
the regularization technique is how to choose the regularization coefficient
appropriately, which is non-trivial for most machine learning problems.
Severalotherexamplesexistinmachineandhumanlearningwhereatrade-
offbetweenconflictingobjectiveshastobetakenintoaccount.Forexamplein
object recognition, a learning system should learn as many details of a given
object as possible; on the other hand, it should also be able to abstract the
generalfeaturesofdifferentobjects.Anotherexampleisthestabilityandplas-
ticitydilemma,wherethelearningsystemshavetotradeoffbetweenlearning
new information without forgetting old information. From the viewpoint of
multi-objective optimization, there is no single learning model that can sat-
isfy different objectives at the same time. In this sense, Pareto-based multi-
objective optimization is the only way to deal with the conflicting objectives
VIII Preface
in machine learning. However, this seemingly straightforward idea has not
been implemented in machine learning until late 1990’s. Liu and Kadirka-
manathan [1] considered three criteria in designing neural networks for sys-
tem identification. They used a genetic algorithm to minimize the maximum
of the three normalized objectives. Similar work has been presented in [2].
KottathraandAttikiouzel[3]employedabranch-and-boundsearchalgorithm
to determine the structure of neural networks by trading off the mean square
error and the number of hidden nodes. The trade-off between sum of squared
errorandthenormofweightsofneuralnetworkswasreportedin[4],whereas
thetrade-offbetweentrainingerrorandtesterrorhasbeenconsideredin [5].
An interesting work on Pareto-based neural network learning is reported by
Kupinski and Anastasio [6], where a multi-objective genetic algorithm is im-
plemented to generate the receiver operating characteristics curve of neural
network classifiers. In generating fuzzy systems, Ishibuchi et al [7] used a
multi-objective genetic algorithm to minimize the classification error and the
number of rules. Gomez-Skarmeta et al suggested the idea of using Pareto-
based multi-objective genetic algorithms to optimize multiple objectives in
fuzzy modeling [8], though no simulation results were provided. A genetic al-
gorithm is used to minimize approximation error, complexity, sensitivity to
noise and continuity of rules [9].
With the boom of the research on evolutionary multi-objective optimiza-
tion,Pareto-optimalitybasedmulti-objectivemachinelearninghasgainednew
impetus. Compared to the early works, not only more sophisticated multi-
objective algorithms are used, but new research areas are also being opened
up. For example, it is found that one is able to determine the number of clus-
ters by analyzing the shape of the Pareto front obtained by a multi-objective
clustering method [10], see Chapter 2 for more detailed results. In [11], it
is found that interpretable rules can be extracted from the simple, Pareto-
optimal neural networks. Further research reveals that neural networks with
good generalization capability can also be identified by analyzing the Pareto
front, as reported in Chapter 13 in this book. Our most recent work sug-
gests that multi-objective machine learning provides an efficient approach to
addressing catastrophic forgetting [12]. Besides, Pareto-based multi-objective
learning has shown particularly powerful in generating ensembles, support
vector machines (SVMs), and interpretable fuzzy systems.
Thiseditedbookpresentsacollectionofmostrepresentativeresearchwork
on multi-objective machine learning. The book is structured into five parts.
PartIdiscussesmulti-objectivefeatureextractionandselection,suchasrough
set based feature selection, clustering and cluster validation, supervised and
unsupervised feature selection for ensemble based handwritten digit recog-
nition, and edge detection in image processing. In the second part, multi-
objectivemodelselectionispresentedforimprovingtheperformanceofsingle
objective learning algorithms in generating various machine learning mod-
els, including linear and nonlinear regression models, multi-layer perceptrons
(MLPs), radial-basis-function networks (RBFNs), support vector machines
Preface IX
(SVMs),decisiontrees,andlearningclassifiersystems.Multi-objectivemodel
selection for creating interpretable models is described in Part III. Generat-
ing interpretable learning models plays an important rule in data mining and
knowledge extraction, where the preference is put on gaining insights into
unknown systems. From the work presented, the reader can see that how un-
derstandable symbolic or fuzzy rules can be extracted from trained neural
networks, or from data directly within the framework of multi-objective opti-
mization. The merit of multi-objective optimization is fully demonstrated in
PartIV,wheretechniquesforgeneratingensemblesofmachinelearningmod-
elsareconcerned.Diversememberofneuralnetworksorfuzzysystemscanbe
generated by trading off between training and test errors, between accuracy
andcomplexity,orbetweenaccuracyanddiversity.Comparedtosingleobjec-
tive based ensemble generation methods, diversity is imposed more explicitly
inmulti-objectivelearningsothatstructuralorfunctionaldiversityofensem-
ble members can be guaranteed. To conclude the book, Part V presents a
numberofsuccessfulapplicationsofmulti-objectivemachinelearning,suchas
multi-class receiver operating curve analysis, mobile robot navigation, dock-
ing maneuver of automated ground vehicles, information retrieval and object
detection.
Iamconfidentthatbyreadingthisbook,thereaderisabletobringhomea
complete view of the emerging research area and to gain hands-on experience
of a variety of multi-objective machine learning approaches. Furthermore, I
do hope that this book, which is the first book dedicated to multi-objective
machinelearningtothebestofmyknowledge,willinspiremorecreativeideas
to further promote the research in this research area.
I would like to thank all contributors who prepared excellent chapters for
this book. Many thanks go to Prof. Janusz Kacprzyk for his interest in this
book.IwouldalsoliketothankDr.ThomasDitzingerandMs.HeatherKing
at Springer for their kind assistance. I am most grateful to Mr. Tomohiko
Kawanabe, Prof. Dr. Edgar Ko¨rner, Dr. Bernhard Sendhoff and Mr. Andreas
Richter at the Honda Research Institute Europe for their full understanding
and kind support.
Offenbach am Main
October 2005 Yaochu Jin
References
[1] G.P. Liu and V. Kadirkamanathan. Learning with multi-objective criteria. In
IEE Conference on Artificial Neural Networks, pages 53–58, 1995.
[2] S. Park, D. Nam, and C. H. Park. Design of a neural controller using multi-
objectiveoptimizationfornonminimumphasesystems. InProceedings of 1999
IEEE Int. Conf. on Fuzzy Systems, volume I, pages 533–537, 1999.
X Preface
[3] K.KottathraandY.Attikiouzel. Anovelmulticriteriaoptimizationalgorithm
forthestructuredeterminationofmultilayerfeedforwardneuralnetworks.Jour-
nal of Network and Computer Applications, 19:135–147, 1996.
[4] R.deA.Teixeira,A.P.Braga,R.H.C.Takahashi,andR.R.Saldanha. Improv-
inggeneralizationofMLPwithmulti-objectiveoptimization. Neurocomputing,
35:189–194, 2000.
[5] H.A. Abbass. A memetic Pareto approach to artificial neural networks. In
Proceedings of the 14th Australian Joint Conference on Artificial Intelligence,
pages 1–12, 2001.
[6] M.A. Kupinski and M Anastasio. Multiobjective genetic optimization of diag-
nostic classifiers with implementations for generating receiver operating char-
acteristiccurves. IEEETransactionsonMedicalImaging,18(8):675–685,1999.
[7] H.Ishibuchi,T.Murata,andB.Turksen.Single-objectiveandtwo-objectivege-
neticalgorithmsforselectinglinguisticrulesforpatternclassificationproblems.
Fuzzy Sets and Systems, 89:135–150, 1997.
[8] A. Gomez-Skameta, F. Jimenez, and J. Ibanez. Pareto-optimality in fuzzy
modeling. InProc.ofthe6thEuropeanCongressonIntelligentTechniquesand
Soft Computing, pages 694–700, 1998.
[9] T. Suzuhi, T. Furuhashi, S. Matsushita, and H. Tsutsui. Efficient fuzzy mod-
eling under multiple criteria by using genetic algorithm. In IEEE Conf. on
Systems, Man and Cybernetics, volume 5, pages 314–319, 1999.
[10] J.HandlandJ.Knowles. Exploitingthetrade-off–thebenefitsofmultipleob-
jectivesindataclustering. InEvolutionaryMulti-CriteriaOptimization,LNCS
3410, pages 547–560. Springer, 2005.
[11] Y. Jin, B. Sendhoff, and E. K¨orner. Evolutionary multi-objective optimiza-
tion for simultaneous generation of signal-type and symbol-type representa-
tions.InEvolutionaryMulti-CriteriaOptimization,LNCS3410,pages752–766.
Springer, 2005.
[12] Y. Jin and B. Sendhoff. Avoiding catastrophic forgetting via multi-objective
learning. Congress of Evolutionary Computation. 2006
Contents
Part I Multi-Objective Clustering, Feature Extraction
and Feature Selection
1 Feature Selection Using Rough Sets
Mohua Banerjee, Sushmita Mitra, Ashish Anand..................... 3
2 Multi-Objective Clustering and Cluster Validation
Julia Handl, Joshua Knowles...................................... 21
3 Feature Selection for Ensembles Using the Multi-Objective
Optimization Approach
Luiz S. Oliveira, Marisa Morita, Robert Sabourin .................... 49
4 Feature Extraction Using Multi-Objective
Genetic Programming
Yang Zhang, Peter I Rockett ...................................... 75
Part II Multi-Objective Learning for Accuracy Improvement
5 Regression Error Characteristic Optimisation of Non-Linear
Models
Jonathan E. Fieldsend............................................103
6 Regularization for Parameter Identification Using
Multi-Objective Optimization
Tomonari Furukawa, Chen Jian Ken Lee, John G. Michopoulos........125
7 Multi-Objective Algorithms for Neural Networks Learning
Antˆonio P´adua Braga, Ricardo H. C. Takahashi, Marcelo Azevedo
Costa, Roselito de Albuquerque Teixeira.............................151