Table Of ContentStudies in Computational Intelligence 659
Jayadeva
Reshma Khemchandani
Suresh Chandra
Twin Support
Vector
Machines
Models, Extensions and Applications
Studies in Computational Intelligence
Volume 659
Series editor
Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland
e-mail: [email protected]
About this Series
The series “Studies in Computational Intelligence” (SCI) publishes new develop-
mentsandadvancesinthevariousareasofcomputationalintelligence—quicklyand
with a high quality. The intent is to cover the theory, applications, and design
methods of computational intelligence, as embedded in the fields of engineering,
computer science, physics and life sciences, as well as the methodologies behind
them. The series contains monographs, lecture notes and edited volumes in
computational intelligence spanning the areas of neural networks, connectionist
systems, genetic algorithms, evolutionary computation, artificial intelligence,
cellular automata, self-organizing systems, soft computing, fuzzy systems, and
hybrid intelligent systems. Of particular value to both the contributors and the
readership are the short publication timeframe and the worldwide distribution,
which enable both wide and rapid dissemination of research output.
More information about this series at http://www.springer.com/series/7092
Jayadeva Reshma Khemchandani
(cid:129)
Suresh Chandra
Twin Support Vector
Machines
Models, Extensions and Applications
123
Jayadeva SureshChandra
Department ofElectrical Engineering Department ofMathematics
Indian Institute of Technology (IIT) Indian Institute of Technology (IIT)
NewDelhi NewDelhi
India India
ReshmaKhemchandani
Department ofComputer Science
Faculty of Mathematics andComputer
Science
SouthAsianUniversity (SAU)
NewDelhi
India
ISSN 1860-949X ISSN 1860-9503 (electronic)
Studies in Computational Intelligence
ISBN978-3-319-46184-7 ISBN978-3-319-46186-1 (eBook)
DOI 10.1007/978-3-319-46186-1
LibraryofCongressControlNumber:2016951985
©SpringerInternationalPublishingSwitzerland2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
foranyerrorsoromissionsthatmayhavebeenmade.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
To my parents, Shri. R. Sampat Kumaran and
Smt. M.C. Ramananda, who have been a
lifelong source of inspiration and support.
—Jayadeva
To my loving mother
Mrs. Shilpa Khemchandani, whose beautiful
memories will always stay with me,
my family and my teachers.
—Reshma Khemchandani
To my wife Shashi, daughter Surabhi and
ever present son Suyash for their continuous
support and encouragement.
—Suresh Chandra
Preface
Supportvectormachines(SVMs),introducedbyVapnikin1998,haveprovenvery
effective computational tool in Machine Learning. SVMs have already outper-
formed most other computational intelligence methodologies mainly because they
are based on sound mathematical principles of statistical learning theory and
optimization theory. SVMs have been applied successfully to a wide spectrum of
areas, ranging from pattern recognition, text categorization, biomedicine, bioin-
formatics and brain–computer interface to financial time series forecasting.
In2007,Jayadeva,KhemchandaniandChandrahaveproposedanovelclassifier
called twin support vector machine (TWSVM) for binary data classification.
TWSVMgeneratestwonon-parallelhyperplanesbysolvingapairofsmaller-sized
quadraticprogrammingproblems(QPPs)suchthateachhyperplaneisclosertoone
class and as far as possible from the other class. The strategy of solving a pair of
smaller-sized QPPs, rather than a single large one, makes the learning speed of
TWSVM approximately four times faster than the standard SVM.
Overtheyears,TWSVMhasbecomeapopularmachinelearningtoolbecauseof
its low computational complexity. Not only TWSVM has been applied to a wide
spectrum of areas, many researchers have proposed new variants of TWSVM, for
classification, clustering (TWSVC) and regression (TWSVR) scenarios.
This monograph presents a systematic and focused study of the various aspects
of TWSVM and related developments for classification, clustering and regression.
ApartfrompresentingmostofthebasicmodelsofTWSVM,TWSVCandTWSVR
available in the literature, a special effort has been made to include important and
challenging applications of the tool. A chapter on “Some Additional Topics” has
beenincludedtodiscusstopicsofkerneloptimizationandsupporttensormachines
which are comparatively new, but have great potential in applications.
After presenting an overview of support vector machines in Chap. 1 and
devoting an introductory Chapter (Chap. 2) on generalized eigenvalue proximal
supportvectormachines(GEPSVM),themaincontentsrelatedwithTWSVMsare
presented in Chaps. 3–8. Here Chap. 8 is fully devoted to “applications”.
This monograph is primarily addressed to graduate students and researchers in
the area of machine learning and related topics in computer science, mathematics
vii
viii Preface
and electrical engineering. Senior undergraduate students having reasonable
mathematical maturity will also benefit from this monograph. In writing a mono-
graph of this kind, there is always a difficulty in identifying what to include, or
rather what not to include. We have tried our best to include all major variants
of the basic TWSVM formulation but still some may have been left out. We hope
thatthereaderswillappreciatethatamonographcanneverbemadeexhaustiveand
some such omissions are bound to happen. Although every care has been taken to
make the presentation error free, some errors may still remain and we hold our-
selvesresponsibleforthatandrequestthaterrors,ifany,beintimatedbye-mailing
at [email protected] (e-mail address of S. Chandra).
In the long process of writing this monograph, we have been encouraged and
helpedbymanyindividuals.WewouldfirstandforemostliketothankProf.Janusz
Kacprzyk for accepting our proposal and encouraging us to write this monograph.
We would specially like to thank Prof. O.L. Mangasarian whose research work
hasbeenthestartingpointofourinterestintheareaofmachinelearning.Inanera
when we had no access to the Internet, he has been very kind in sending his
reprints/preprints and clarifying many subtle points. In our research, we have been
mostly following the work of Prof. Mangasarian and in that sense we have always
regarded him as our mentor.
We are highly grateful to Profs. X.J. Peng, Y. Tian and Y.H. Shao and their
research collaborators for sending their reprints/preprints and answering to our
queriesattheearliest.Wewouldalsoliketothankandacknowledgetheeditorsand
the publishers of the journals IEEE Transactions on Pattern Analysis and Machine
Intelligence, Neurocomputing, Neural Networks, International Journal of Machine
Learning and Cybernetics, Expert System with Applications, OptimizationLetters,
Information Sciences, European Journal of Operational Research, Neural
ComputinganditsApplications,AppliedIntelligence,andAppliedSoftComputing
for publishing our papers on TWSVM-related work which constitutes the core of
this monograph.
OurspecialthanksaretotheauthoritiesofIndianInstituteofTechnology,Delhi,
andSouthAsianUniversity,NewDelhi,fortheirencouragementandprovidingall
facilities for the preparation of this monograph.
We would like to thank our friends and colleagues Profs. Aparna Mehra,
B.Chandra,K.K.Biswas,R.KSharma,S.DharmarajaandShantanuChaudharyat
IIT Delhi, and Danish Lohani, Pranab Muhuri and R.K. Mohanty at South Asian
University (SAU) for their interest and encouragement. Professors Amit Bhaya,
C.A. Murthy, C.R. Bector, Jeyakumar, Pankaj Gupta, Ravi Kothari, M. Arun
Kumar, S. Balasundaram and S.K. Neogy have always encouraged us and took
keen interest in the preparation of this monograph.
We appreciate our students Anuj Karpatne, Keshav Goel and Sumit Soman of
IIT Delhi, and Pooja Saigal, Aman Pal, Sweta Sharma and Pritam Anand of SAU
for their help in the preparation of this manuscript in LATEX and reading the
manuscript from a student point of view. Dr. Abha Aggarwal took a major
responsibility of organizing the material and her efforts are duly acknowledged.
Preface ix
We are obliged to Dr. Thomas Ditzinger of International Engineering
Department,SpringerVerlag,forallhishelp,cooperationandunderstandinginthe
publication of this monograph.
New Delhi, India Jayadeva
Reshma Khemchandani
Suresh Chandra
Contents
1 Introduction.... .... .... ..... .... .... .... .... .... ..... .... 1
1.1 Support Vector Machines: An Overview .. .... .... ..... .... 1
1.2 The Classical L -Norm SVM ... .... .... .... .... ..... .... 2
1
1.2.1 Linear SVM: Hard Margin Classifier ... .... ..... .... 2
1.2.2 Linear SVM: Soft Margin Classifier.... .... ..... .... 4
1.2.3 Nonlinear/Kernel SVM.. .... .... .... .... ..... .... 6
1.3 Least Squares SVM and Proximal SVM... .... .... ..... .... 9
1.4 Support Vector Regression. .... .... .... .... .... ..... .... 12
1.5 Efficient Algorithms for SVM and SVR... .... .... ..... .... 15
1.6 Other Approaches to Solving the SVM QPP ... .... ..... .... 18
1.6.1 The Relaxed SVM . .... .... .... .... .... ..... .... 18
1.6.2 The Relaxed LSSVM ... .... .... .... .... ..... .... 20
1.6.3 Solving the Relaxed SVM and the Relaxed LSSVM .... 21
1.7 Conclusions ... .... ..... .... .... .... .... .... ..... .... 23
References.. .... .... .... ..... .... .... .... .... .... ..... .... 23
2 Generalized Eigenvalue Proximal Support Vector Machines... .... 25
2.1 Introduction ... .... ..... .... .... .... .... .... ..... .... 25
2.2 GEPSVM for Classification .... .... .... .... .... ..... .... 25
2.2.1 Linear GEPSVM Classifier... .... .... .... ..... .... 26
2.2.2 Nonlinear GEPSVM Classifier .... .... .... ..... .... 31
2.3 Some Variants of GEPSVM for Classification .. .... ..... .... 32
2.3.1 ReGEPSVM Formulation .... .... .... .... ..... .... 33
2.3.2 Improved GEPSVM Formulation .. .... .... ..... .... 34
2.4 GEPSVR: Generalized Eigenvalue Proximal Support
Vector Regression... ..... .... .... .... .... .... ..... .... 36
2.4.1 GEPSVR Formulation... .... .... .... .... ..... .... 37
2.4.2 Regularized GEPSVR Formulation. .... .... ..... .... 39
2.4.3 Experimental Results.... .... .... .... .... ..... .... 40
2.5 Conclusions ... .... ..... .... .... .... .... .... ..... .... 42
References.. .... .... .... ..... .... .... .... .... .... ..... .... 42
xi
