Multidimensional Particle Swarm Optimization for Machine Learning and Pattern Recognition

343 Pages·2014·14.606 MB·English

by  Serkan Kiranyaz, Turker Ince, Moncef Gabbouj (auth.)

5
, 1
G e
N m
I
N u
R l
o
A
V
E
L N
N, O
Serkan Kiranyaz O TI
I A
Turker Ince AT IZ
T M
P
Moncef Gabbouj ADTI
DNP
AAO
Multidimensional Particle
Swarm Optimization for
Machine Learning and
Pattern Recognition
Adaptation, Learning, and Optimization
Volume 15
Editors-in-Chief
Meng-Hiot Lim
DivisionofCircuitsandSystems,SchoolofElectricalandElectronicEngineering,
Nanyang Technological University, Nanyang 639798, Singapore
Yew-Soon Ong
School of Computer Engineering, Nanyang Technological University, Block N4,
2b-39 Nanyang Avenue, Nanyang, 639798, Singapore
For furthervolumes:
http://www.springer.com/series/8335
Serkan Kiranyaz Turker Ince
•
Moncef Gabbouj
Multidimensional Particle
Swarm Optimization
for Machine Learning
and Pattern Recognition
123
Serkan Kiranyaz TurkerInce
MoncefGabbouj Department of Electricaland Electronics
Department of Signal Processing Engineering
Tampere Universityof Technology Izmir Universityof Economics
Tampere Balcova, Izmir
Finland Turkey
ISSN 1867-4534 ISSN 1867-4542 (electronic)
ISBN 978-3-642-37845-4 ISBN 978-3-642-37846-1 (eBook)
DOI 10.1007/978-3-642-37846-1
SpringerHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2013938212
ACMComputingClassification(1998):I.2,I.4,J.2
(cid:2)Springer-VerlagBerlinHeidelberg2014
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor
informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodology now known or hereafter developed. Exempted from this legal reservation are brief
excerpts in connection with reviews or scholarly analysis or material supplied specifically for the
purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe
work. Duplication of this publication or parts thereof is permitted only under the provisions of
theCopyright Law of the Publisher’s location, in its current version, and permission for use must
always be obtained from Springer. Permissions for use may be obtained through RightsLink at the
CopyrightClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Thedefinitionofsuccess—Tolaughmuch;towinrespectofintelligentpersonsandthe
affectionsofchildren;toearntheapprobationofhonestcriticsandendurethebetrayalof
falsefriends;toappreciatebeauty;tofindthebestinothers;togiveone’sself;toleavethe
world a little better, whether by a healthy child, a garden patch, or a redeemed social
condition;tohaveplayedandlaughedwithenthusiasm,andsungwithexultation;toknow
evenonelifehasbreathedeasierbecauseyouhavelived—thisistohavesucceeded.
RalphWaldoEmerson
The research work presented in this book has been carried out at the Depart-
mentofSignalProcessingofTampereUniversityofTechnology,Finlandasapart
oftheMUVISproject. ThisbookcontainsarichsoftwarecompilationofC/C??
projects with open source codes, which can be requested from the authors via the
email address: [email protected].
Over the years the authors have had the privilege to work with a wonderful
groupofresearchers,students,andcolleagues,manyofwhomareourfriends.The
amount of our achievements altogether is much more than any individual
achievementandwestronglybelievethattogetherwehavereallybuiltsomething
significant. We thank all of them so deeply. Our special thanks and acknowl-
edgment go to Jenni Raitoharju and Stefan Uhlmann for their essential
contributions.
Last but not least, the authors wish to express their love and gratitude to their
beloved families; for their understanding and endless love, and the vital role they
played in our lives and through the completion of this book. We would like to
dedicatethisbooktoourchildren:thenew-bornbabygirl,AlyaNickoleKiranyaz,
the baby boy, Doruk Ince, Selma, and Sami Gabbouj.
Tampere, December 2012 Prof. Dr. Serkan Kiranyaz
Doc. Dr. Turker Ince
Prof. Dr. Moncef Gabbouj
v
Abstract
The field of optimization consists of an elegant blend of theory and applications.
This particular field constitutes the essence of engineering and it was founded,
developed, and extensively used by a certain group of creative people, known as
Engineers.Theyinvestigateandsolveagivenrealworldortheoreticalproblemas
besttheycanandthatiswhyoptimizationiseverywhereinhumanlife,fromtools
and machinery we use daily in engineering design, computer science, IT tech-
nology, and even economics. It is also true that many optimization problems are
multi-modal, which presents further challenges due to deceiving local optima.
Earlier attempts such as gradient descent methods show drastic limitations and
often get trapped into a local optimum, thus yielding a sub-optimum solution.
During the last few decades, such deficiencies turned attention toward stochastic
optimization methods and particularly to Evolutionary Algorithms. Genetic
Algorithms and Particle Swarm Optimization have been studied extensively and
the latter particularly promises much. However, the peculiar nature of many
engineering problems also requires dynamic adaptation, seeking the dimensionof
the search space where the optimum solution resides, and especially robust tech-
niques to avoid getting trapped in local optima.
Thisbookexploresarecentoptimizationtechniquedevelopedbytheauthorsof
the book, called Multi-dimensional Particle Swarm Optimization (MD PSO),
whichstrivestoaddresstheaboverequirementsfollowinganalgorithmicapproach
tosolveimportantengineeringproblems.Someofthemorecomplexproblemsare
formulated in a multi-dimensional search space where the optimum dimension is
alsounknown.Inthiscase,MDPSOcanseekforbothpositionalanddimensional
optima. Furthermore, two supplementary enhancement methods, the Fractional
Global-Best Formation and Stochastic Approximation with Simultaneous Pertur-
bation,areintroducedasanefficientcuretoavoidgettingtrapped inlocaloptima
especially in multi-modal search spaces defined over high dimensions. The book
covers a wide range of fundamental application areas, which can particularly
benefit from such a unified framework. Consider for instance a data clustering
application where MD PSO can be used to determine the true number of clusters
and accurate cluster centroids, in a single framework. Another application in the
field of machine intelligence is to determine the optimal neural network config-
uration for a particular problem. This might be a crucial step, e.g., for robust and
vii
viii Preface
accurate detection of electrocardiogram (ECG) heartbeat patterns for a specific
patient. The reader will see that this system can adapt to significant inter-patient
variations in ECG patterns by evolving the optimal classifier and thus achieves a
high accuracy over large datasets.
The proposed unified framework is then explored in a set of challenging
application domains, namely data mining and content-based multimedia classifi-
cation. Although there are numerous efforts for the latter, we are still in the early
stages of the development to guarantee a satisfactory level of efficiency and
accuracy. To accomplish this for content-based image retrieval (CBIR) and clas-
sification,thebookpresentsaglobalframeworkdesignthatembodiesacollective
network of evolutionary classifiers. This is a dynamic and adaptive topology,
which allows thecreation anddesignofadedicatedclassifierfordiscriminatinga
certain image class from the othersbased on a single visual descriptor. During an
evolution session, new images, classes, or features can be introduced whilst sig-
naling the classifier networkto create new correspondingnetworksand classifiers
within, to dynamically adapt to the change. In this way the collective classifier
network will be able to scale itself to the indexing requirements of the image
content datareserve whilststrivingfor maximizingtheclassification andretrieval
accuracies for better user experience. However one obstacle still remains: low-
level features play the most crucial role in CBIR but they usually lack the dis-
crimination power needed for accurate visual description and representation
especiallyinthecaseoflargeanddynamicimagedatareserves.Finally,thebook
tacklesthismajorresearchobjectiveandpresentsanevolutionaryfeaturesynthesis
framework, which aims to significantly improve the discrimination power by
synthesizing highly discriminative features. This is obviously not limited to only
CBIR, but can be utilized to synthesize enhanced features for any application
domain where features or feature extraction is involved.
Thesetofdiverseapplicationspresentedinthebookpointsthewaytoexplorea
wide range of potential applications in engineering as well as other disciplines.
ThebookissupplementedwithC/C??sourcecodesforallapplicationsandmany
sample datasets to illustrate the major concepts presented in the book. This will
allow practitioners and professionals to comprehend and use the presented tech-
niques and adapt them to their own applications immediately.
Contents
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Optimization Era . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Key Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Synopsis of the Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Optimization Techniques: An Overview . . . . . . . . . . . . . . . . . . . 13
2.1 History of Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Deterministic and Analytic Methods. . . . . . . . . . . . . . . . . . . 29
2.2.1 Gradient Descent Method . . . . . . . . . . . . . . . . . . . . 29
2.2.2 Newton–Raphson Method . . . . . . . . . . . . . . . . . . . . 30
2.2.3 Nelder–Mead Search Method. . . . . . . . . . . . . . . . . . 32
2.3 Stochastic Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.3.1 Simulated Annealing. . . . . . . . . . . . . . . . . . . . . . . . 33
2.3.2 Stochastic Approximation . . . . . . . . . . . . . . . . . . . . 35
2.4 Evolutionary Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4.1 Genetic Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4.2 Differential Evolution. . . . . . . . . . . . . . . . . . . . . . . 41
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2 Basic PSO Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.3 Some PSO Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.3.1 Tribes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3.2 Multiswarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4 Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.1 Nonlinear Function Minimization. . . . . . . . . . . . . . . 55
3.4.2 Data Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.4.3 Artificial Neural Networks. . . . . . . . . . . . . . . . . . . . 61
3.5 Programming Remarks and Software Packages . . . . . . . . . . . 74
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
ix
x Contents
4 Multi-dimensional Particle Swarm Optimization . . . . . . . . . . . . . 83
4.1 The Need for Multi-dimensionality. . . . . . . . . . . . . . . . . . . . 83
4.2 The Basic Idea. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.3 The MD PSO Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4 Programming Remarks and Software Packages . . . . . . . . . . . 92
4.4.1 MD PSO Operation in PSO_MDlib Application. . . . . 92
4.4.2 MD PSO Operation in PSOTestApp Application. . . . . 94
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5 Improving Global Convergence. . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.1 Fractional Global Best Formation. . . . . . . . . . . . . . . . . . . . . 102
5.1.1 The Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.1.2 PSO with FGBF. . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.1.3 MD PSO with FGBF . . . . . . . . . . . . . . . . . . . . . . . 104
5.1.4 Nonlinear Function Minimization. . . . . . . . . . . . . . . 104
5.2 Optimization in Dynamic Environments . . . . . . . . . . . . . . . . 116
5.2.1 Dynamic Environments: The Test Bed . . . . . . . . . . . 116
5.2.2 Multiswarm PSO . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.2.3 FGBF for the Moving Peak Benchmark for MPB. . . . 118
5.2.4 Optimization over Multidimensional MPB. . . . . . . . . 119
5.2.5 Performance Evaluation on Conventional MPB . . . . . 120
5.2.6 Performance Evaluation on Multidimensional
MPB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.3 Who Will Guide the Guide? . . . . . . . . . . . . . . . . . . . . . . . . 128
5.3.1 SPSA Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5.3.2 SA-Driven PSO and MD PSO Applications. . . . . . . . 131
5.3.3 Applications to Non-linear Function Minimization. . . 134
5.4 Summary and Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . 141
5.5 Programming Remarks and Software Packages . . . . . . . . . . . 142
5.5.1 FGBF Operation in PSO_MDlib Application. . . . . . . 143
5.5.2 MD PSO with FGBF Application Over MPB. . . . . . . 144
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6 Dynamic Data Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
6.1 Dynamic Data Clustering via MD PSO with FGBF . . . . . . . . 152
6.1.1 The Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
6.1.2 Results on 2D Synthetic Datasets. . . . . . . . . . . . . . . 155
6.1.3 Summary and Conclusions. . . . . . . . . . . . . . . . . . . . 160
6.2 Dominant Color Extraction . . . . . . . . . . . . . . . . . . . . . . . . . 160
6.2.1 Motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
6.2.2 Fuzzy Model over HSV-HSL Color Domains . . . . . . 163
6.2.3 DC Extraction Results. . . . . . . . . . . . . . . . . . . . . . . 164
6.2.4 Summary and Conclusions. . . . . . . . . . . . . . . . . . . . 170
6.3 Dynamic Data Clustering via SA-Driven MD PSO. . . . . . . . . 171
Contents xi
6.3.1 SA-Driven MD PSO-Based Dynamic Clustering
in 2D Datasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.3.2 Summary and Conclusions. . . . . . . . . . . . . . . . . . . . 174
6.4 Programming Remarks and Software Packages . . . . . . . . . . . 176
6.4.1 FGBF Operation in 2D Clustering . . . . . . . . . . . . . . 176
6.4.2 DC Extraction in PSOTestApp Application . . . . . . . . 179
6.4.3 SA-DRIVEN Operation in PSOTestApp
Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7 Evolutionary Artificial Neural Networks . . . . . . . . . . . . . . . . . . . 187
7.1 Search for the Optimal Artificial Neural Networks:
An Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
7.2 Evolutionary Neural Networks by MD PSO. . . . . . . . . . . . . . 190
7.2.1 PSO for Artificial Neural Networks:
The Early Attempts. . . . . . . . . . . . . . . . . . . . . . . . . 190
7.2.2 MD PSO-Based Evolutionary Neural Networks . . . . . 191
7.2.3 Classification Results on Synthetic Problems. . . . . . . 193
7.2.4 Classification Results on Medical Diagnosis
Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
7.2.5 Parameter Sensitivity and Computational
Complexity Analysis. . . . . . . . . . . . . . . . . . . . . . . . 203
7.3 Evolutionary RBF Classifiers for Polarimetric SAR Images. . . 205
7.3.1 Polarimetric SAR Data Processing . . . . . . . . . . . . . . 207
7.3.2 SAR Classification Framework. . . . . . . . . . . . . . . . . 209
7.3.3 Polarimetric SAR Classification Results . . . . . . . . . . 211
7.4 Summary and Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . 217
7.5 Programming Remarks and Software Packages . . . . . . . . . . . 218
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
8 Personalized ECG Classification . . . . . . . . . . . . . . . . . . . . . . . . . 231
8.1 ECG Classification by Evolutionary Artificial
Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
8.1.1 Introduction and Motivation. . . . . . . . . . . . . . . . . . . 233
8.1.2 ECG Data Processing . . . . . . . . . . . . . . . . . . . . . . . 235
8.1.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 239
8.2 Classification of Holter Registers. . . . . . . . . . . . . . . . . . . . . 244
8.2.1 The Related Work . . . . . . . . . . . . . . . . . . . . . . . . . 245
8.2.2 Personalized Long-Term ECG Classification:
A Systematic Approach. . . . . . . . . . . . . . . . . . . . . . 246
8.2.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 250
8.3 Summary and Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . 253
8.4 Programming Remarks and Software Packages . . . . . . . . . . . 255
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257