Algorithms for Fuzzy Clustering: Methods in c-Means Clustering with Applications

244 Pages·2008·4.89 MB·English

by  Sadaaki Miyamoto

SadaakiMiyamoto,HidetomoIchihashi,KatsuhiroHonda
AlgorithmsforFuzzyClustering
Studiesin Fuzzinessand Soft Computing,Volume229
Editor-in-Chief
Prof.JanuszKacprzyk
SystemsResearchInstitute
PolishAcademyofSciences
ul.Newelska6
01-447Warsaw
Poland
E-mail:[email protected]
Furthervolumesofthisseriescanbefoundonourhomepage:springer.com
Vol.214.IrinaGeorgescu Vol.222.JamesJ.Buckley,
FuzzyChoiceFunctions,2007 LeonardJ.Jowers
ISBN978-3-540-68997-3 MonteCarloMethodsinFuzzy
Optimization,2008
Vol.215.PaulP.Wang,DaRuan,
ISBN978-3-540-76289-8
EtienneE.Kerre(Eds.)
FuzzyLogic,2007 Vol.223.OscarCastillo,PatriciaMelin
ISBN978-3-540-71257-2 Type-2FuzzyLogic:Theoryand
Vol.216.RudolfSeising Applications,2008
TheFuzzificationofSystems,2007 ISBN978-3-540-76283-6
ISBN978-3-540-71794-2
Vol.224.RafaelBello,RafaelFalcón,
Vol.217.MasoudNikravesh,JanuszKacprzyk, WitoldPedrycz,JanuszKacprzyk(Eds.)
LoftiA.Zadeh(Eds.) ContributionstoFuzzyandRoughSets
ForgingNewFrontiers:Fuzzy TheoriesandTheirApplications,2008
PioneersI,2007 ISBN978-3-540-76972-9
ISBN978-3-540-73181-8
Vol.225.TerryD.Clark,JenniferM.Larson,
Vol.218.MasoudNikravesh,JanuszKacprzyk, JohnN.Mordeson,JoshuaD.Potter,
LoftiA.Zadeh(Eds.) MarkJ.Wierman
ForgingNewFrontiers:Fuzzy ApplyingFuzzyMathematicstoFormal
PioneersII,2007 ModelsinComparativePolitics,2008
ISBN978-3-540-73184-9 ISBN978-3-540-77460-0
Vol.219.RolandR.Yager,LipingLiu(Eds.) Vol.226.BhanuPrasad(Ed.)
ClassicWorksoftheDempster-ShaferTheory SoftComputingApplicationsinIndustry,2008
ofBeliefFunctions,2007 ISBN978-3-540-77464-8
ISBN978-3-540-25381-5
Vol.227.EugeneRoventa,TiberiuSpircu
Vol.220.HumbertoBustince, ManagementofKnowledgeImperfectionin
FranciscoHerrera,JavierMontero(Eds.) BuildingIntelligentSystems,2008
FuzzySetsandTheirExtensions: ISBN978-3-540-77462-4
Representation,AggregationandModels,2007
Vol.228.AdamKasperski
ISBN978-3-540-73722-3
DiscreteOptimizationwithIntervalData,2008
Vol.221.GlebBeliakov,TomasaCalvo, ISBN978-3-540-78483-8
AnaPradera
AggregationFunctions:AGuide Vol.229.SadaakiMiyamoto,
forPractitioners,2007 HidetomoIchihashi,KatsuhiroHonda
ISBN978-3-540-73720-9 AlgorithmsforFuzzyClustering,2008
ISBN978-3-540-78736-5
Sadaaki Miyamoto,
Hidetomo Ichihashi, Katsuhiro Honda
Algorithms for Fuzzy
Clustering
Methods in c-Means Clustering with
Applications
ABC
Authors
Dr.SadaakiMiyamoto Dr.KatsuhiroHonda
UniversityofTsukuba OsakaPrefectureUniversity
Inst.InformationSciences GraduateSchoolofEngineering
andElectronics 1-1Gakuen-cho
Ibaraki Sakai
305-8573Japan Osaka,599-8531Japan
Email:[email protected]
Dr.HidetomoIchihashi
OsakaPrefectureUniversity
GraduateSchoolofEngineering
1-1Gakuen-cho
Sakai
Osaka,599-8531Japan
ISBN978-3-540-78736-5 e-ISBN978-3-540-78737-2
DOI10.1007/978-3-540-78737-2
StudiesinFuzzinessandSoftComputing ISSN1434-9922
LibraryofCongressControlNumber:2008922722
(cid:2)c 2008Springer-VerlagBerlinHeidelberg
Thisworkissubject tocopyright. Allrightsarereserved, whetherthewholeorpart ofthematerial
isconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broad-
casting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of
thispublicationorpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLaw
ofSeptember 9, 1965, initscurrent version, andpermission for use must always be obtained from
Springer.ViolationsareliabletoprosecutionundertheGermanCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnot
imply, even in the absence of a specific statement, that such names are exempt from the relevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Typeset&CoverDesign:ScientificPublishingServicesPvt.Ltd.,Chennai,India.
Printedinacid-freepaper
987654321
springer.com
Preface
Recently many researchers are working on cluster analysis as a main tool for
exploratory data analysis and data mining. A notable feature is that specialists
in different fields of sciences are considering the tool of data clustering to be
useful. A major reasonis that clustering algorithms and software are flexible in
thesensethatdifferentmathematicalframeworksareemployedinthealgorithms
and a user can select a suitable method according to his application. Moreover
clusteringalgorithmshavedifferentoutputsrangingfromtheolddendrogramsof
agglomerativeclusteringtomorerecentself-organizingmaps.Thus,aresearcher
orusercanchooseanappropriateoutputsuitedtohispurpose,whichisanother
flexibility of the methods of clustering.
An old and still most popular method is the K-means which use K cluster
centers. A group of data is gathered around a cluster center and thus forms a
cluster. The main subject of this book is the fuzzy c-means proposed by Dunn
andBezdekandtheir variationsincluding recentstudies.Amainreasonwhywe
concentrate on fuzzy c-means is that most methodology and application studies
infuzzyclusteringusefuzzyc-means,andfuzzyc-meansshouldbeconsideredto
beamajortechniqueofclusteringingeneral,regardlesswhetheroneisinterested
in fuzzy methods or not. Moreover recent advances in clustering techniques are
rapidandwerequireanewtextbookthatincludesrecentalgorithms.Weshould
also note that several books have recently been published but the contents do
not include some methods studied herein.
Unlike most studies in fuzzy c-means, what we emphasize in this book is a
familyofalgorithmsusingentropyorentropy-regularizedmethodswhichareless
known, but we consider the entropy-basedmethod to be another useful method
of fuzzy c-means. For this reasonwe call the method of fuzzy c-means by Dunn
and Bezdek as the standard method to distinguish it from the entropy-based
method. Throughout this book one of our intentions is to uncover theoretical
and methodological differences between the standard method and the entropy-
based method. We do note claim that the entropy-basedmethod is better than
the standardmethod,butwebelievethatthe methodsoffuzzyc-meansbecome
complete by adding the entropy-based method to the standard one by Dunn
VI Preface
and Bezdek, since we can observe natures of the both methods more deeply by
contrasting these two methods.
Readerswillobservethattheentropy-basedmethodissimilartothestatistical
model of Gaussian mixture distribution since both of them are using the error
functions, while the standard method is very different from a statistical model.
For this reason the standard method is purely fuzzy while the entropy-based
method connects a statistical model and a fuzzy model.
Thewholetextisdividedintotwoparts:ThefirstpartthatconsistsofChap-
ters 1∼5 is theoretical and discusses basic algorithms and variations. This part
has been written by Sadaaki Miyamoto.
The second part is application-oriented. Chapter 6 which has been written
by Hidetomo Ichihashi studies classifier design; Katsuhiro Honda has written
Chapters 7∼9 where clustering algorithms are applied to a variety of methods
in multivariate analysis.
The authors are grateful to Prof. Janusz Kacprzyk, the editor, for his en-
couragement to contribute this volume to this series and helpful suggestions
throughout the publication process. We also thank Dr. Mika Sato-Ilic and Dr.
Yasunori Endo for their valuable comments to our works.
We finally note that studies related to this book have partly been supported
by the Grant-in-Aid for Scientific Research,Japan Society for the Promotionof
Science, No.16300065.
January 2008 Sadaaki Miyamoto
Hidetomo Ichihashi
Katsuhiro Honda
Contents
1 Introduction................................................ 1
1.1 Fuzziness and Neural Networks in Clustering................. 3
1.2 An Illustrative Example ................................... 4
2 Basic Methods for c-Means Clustering...................... 9
2.1 A Note on Terminology ................................... 9
2.2 A Basic Algorithm of c-Means ............................. 11
2.3 Optimization Formulation of Crisp c-Means Clustering ........ 12
2.4 Fuzzy c-Means ........................................... 16
2.5 Entropy-BasedFuzzy c-Means ............................. 20
2.6 Addition of a Quadratic Term.............................. 23
2.6.1 Derivation of Algorithm in the Method of the
Quadratic Term.................................... 24
2.7 Fuzzy Classification Rules ................................. 25
2.8 Clustering by Competitive Learning ........................ 29
2.9 Fixed Point Iterations – General Consideration............... 30
2.10 Heuristic Algorithms of Fixed Point Iterations ............... 31
2.11 Direct Derivation of Classification Functions ................. 33
2.12 Mixture Density Model and the EM Algorithm............... 36
2.12.1 The EM Algorithm ................................. 37
2.12.2 Parameter Estimation in the Mixture Densities......... 39
3 Variations and Generalizations - I .......................... 43
3.1 Possibilistic Clustering .................................... 43
3.1.1 Entropy-BasedPossibilistic Clustering ................ 44
3.1.2 Possibilistic Clustering Using a Quadratic Term ........ 46
3.1.3 ObjectiveFunction for Fuzzyc-Means andPossibilistic
Clustering......................................... 46
3.2 Variables for Controlling Cluster Sizes ...................... 47
3.2.1 Solutions for J (U,V,A)........................... 50
efca
3.2.2 Solutions for J (U,V,A) .......................... 50
fcma
VIII Contents
3.3 Covariance Matrices within Clusters ........................ 51
3.3.1 Solutions for FCMAS by the GK(Gustafson-Kessel)
Method ........................................... 53
3.4 The KL (Kullback-Leibler) Information Based Method ........ 55
3.4.1 Solutions for FCMAS by the Method of KL
Information Method ................................ 55
3.5 Defuzzified Methods of c-Means Clustering .................. 56
3.5.1 Defuzzified c-Means with Cluster Size Variable ......... 57
3.5.2 Defuzzification of the KL-Information Based Method ... 58
3.5.3 Sequential Algorithm ............................... 58
3.5.4 Efficient Calculation of Variables ..................... 59
3.6 Fuzzy c-Varieties ......................................... 60
3.6.1 Multidimensional Linear Varieties .................... 62
3.7 Fuzzy c-Regression Models................................. 62
3.8 Noise Clustering.......................................... 65
4 Variations and Generalizations - II ......................... 67
4.1 Kernelized Fuzzy c-Means Clustering and Related Methods .... 67
4.1.1 Transformation into High-Dimensional Feature Space ... 68
4.1.2 Kernelized Crisp c-Means Algorithm.................. 71
4.1.3 Kernelized Learning Vector Quantization Algorithm .... 73
4.1.4 An Illustrative Example............................. 74
4.2 Similarity Measure in Fuzzy c-Means........................ 77
4.2.1 Variable for Controlling Cluster Sizes ................. 80
4.2.2 Kernelization Using Cosine Correlation................ 81
4.2.3 Clustering by Kernelized Competitive Learning Using
Cosine Correlation.................................. 84
4.3 Fuzzy c-Means Based on L Metric ......................... 86
1
4.3.1 Finite Termination Property of the L Algorithm....... 88
1
4.3.2 Classification Functions in the L Case................ 89
1
4.3.3 Boundary between Two Clusters in the L Case........ 90
1
4.4 Fuzzy c-Regression Models Based on Absolute Deviation ...... 91
4.4.1 Termination of Algorithm Based on Least Absolute
Deviation ......................................... 93
4.4.2 An Illustrative Example............................. 96
5 Miscellanea................................................. 99
5.1 More on Similarity and Dissimilarity Measures ............... 99
5.2 Other Methods of Fuzzy Clustering ......................... 100
5.2.1 Ruspini’s Method .................................. 100
5.2.2 Relational Clustering ............................... 101
5.3 Agglomerative Hierarchical Clustering....................... 102
5.3.1 The Transitive Closure of a Fuzzy Relation and the
Single Link ........................................ 106
5.4 A Recent Study on Cluster Validity Functions................ 108
5.4.1 Two Types of Cluster Validity Measures............... 108
Contents IX
5.4.2 Kernelized Measures of Cluster Validity ............... 110
5.4.3 Traces of Covariance Matrices........................ 110
5.4.4 Kernelized Xie-Beni Index........................... 111
5.4.5 Evaluation of Algorithms............................ 111
5.5 Numerical Examples ...................................... 112
5.5.1 The Number of Clusters............................. 112
5.5.2 Robustness of Algorithms ........................... 117
6 Application to Classifier Design............................. 119
6.1 Unsupervised Clustering Phase............................. 119
6.1.1 A Generalized Objective Function .................... 120
6.1.2 Connections with k-Harmonic Means ................. 123
6.1.3 Graphical Comparisons ............................. 125
6.2 Clustering with Iteratively Reweighted Least Square
Technique ............................................... 130
6.3 FCM Classifier........................................... 133
6.3.1 Parameter Optimization with CV Protocol and
Deterministic Initialization .......................... 134
6.3.2 Imputation of Missing Values ........................ 136
6.3.3 Numerical Experiments ............................. 139
6.4 Receiver Operating Characteristics ......................... 144
6.5 Fuzzy Classifier with Crisp c-Means Clustering ............... 150
6.5.1 Crisp Clustering and Post-supervising................. 150
6.5.2 Numerical Experiments ............................. 153
7 Fuzzy Clustering and Probabilistic PCA Model............. 157
7.1 Gaussian Mixture Models and FCM-Type Fuzzy Clustering.... 157
7.1.1 Gaussian Mixture Models ........................... 157
7.1.2 Another Interpretation of Mixture Models ............. 159
7.1.3 FCM-Type Counterpart of Gaussian Mixture Models ... 160
7.2 Probabilistic PCA Mixture Models and Regularized Fuzzy
Clustering ............................................... 162
7.2.1 Probabilistic Models for Principal Component
Analysis........................................... 162
7.2.2 Linear Fuzzy Clustering with Regularized Objective
Function .......................................... 164
7.2.3 An Illustrative Example............................. 167
8 Local Multivariate Analysis Based on Fuzzy Clustering ..... 171
8.1 Switching Regression and Fuzzy c-Regression Models ......... 171
8.1.1 Linear RegressionModel ............................ 171
8.1.2 Switching Linear Regression by Standard Fuzzy
c-RegressionModels ................................ 174
8.1.3 Local Regression Analysis with Centered Data Model ... 175
8.1.4 Connection of the Two Formulations.................. 177
8.1.5 An Illustrative Example............................. 177