Table Of ContentSpringerBriefs in Computer Science
M. N. Murty · M. Avinash
Representation in
Machine Learning
SpringerBriefs in Computer Science
SeriesEditors
StanZdonik,BrownUniversity,Providence,RI,USA
ShashiShekhar,UniversityofMinnesota,Minneapolis,MN,USA
XindongWu,UniversityofVermont,Burlington,VT,USA
LakhmiC.Jain,UniversityofSouthAustralia,Adelaide,SA,Australia
DavidPadua,UniversityofIllinoisUrbana-Champaign,Urbana,IL,USA
XueminShermanShen,UniversityofWaterloo,Waterloo,ON,Canada
BorkoFurht,FloridaAtlanticUniversity,BocaRaton,FL,USA
V.S.Subrahmanian,UniversityofMaryland,CollegePark,MD,USA
MartialHebert,CarnegieMellonUniversity,Pittsburgh,PA,USA
KatsushiIkeuchi,UniversityofTokyo,Tokyo,Japan
BrunoSiciliano,UniversitàdiNapoliFedericoII,Napoli,Italy
SushilJajodia,GeorgeMasonUniversity,Fairfax,VA,USA
NewtonLee,InstituteforEducation,ResearchandScholarships,LosAngeles,CA,
USA
SpringerBriefs present concise summaries of cutting-edge research and practical
applicationsacrossawidespectrumoffields.Featuringcompactvolumesof50to
125pages,theseriescoversarangeofcontentfromprofessionaltoacademic.
Typicaltopicsmightinclude:
(cid:129) Atimelyreportofstate-of-theartanalyticaltechniques
(cid:129) A bridge between new research results, as published in journal articles, and a
contextualliteraturereview
(cid:129) Asnapshotofahotoremergingtopic
(cid:129) Anin-depthcasestudyorclinicalexample
(cid:129) A presentationofcoreconceptsthatstudentsmustunderstandinordertomake
independentcontributions
Briefs allow authors to present their ideas and readers to absorb them with
minimal time investment. Briefs will be published as part of Springer’s eBook
collection, with millions of users worldwide. In addition, Briefs will be available
forindividualprintandelectronicpurchase.Briefsarecharacterizedbyfast,global
electronic dissemination, standard publishing contracts, easy-to-use manuscript
preparation and formatting guidelines, and expedited production schedules. We
aim for publication 8–12 weeks after acceptance. Both solicited and unsolicited
manuscriptsareconsideredforpublicationinthisseries.
**Indexing:ThisseriesisindexedinScopus,Ei-Compendex,andzbMATH**
M. N. Murty (cid:129) M. Avinash
Representation in Machine
Learning
M.N.Murty M.Avinash
DepartmentofCSandAutomation IndianInstituteofTechnologyMadras
IndianInstituteofScienceBangalore Chennai,India
Bangalore,India
ISSN2191-5768 ISSN2191-5776 (electronic)
SpringerBriefsinComputerScience
ISBN978-981-19-7907-1 ISBN978-981-19-7908-8 (eBook)
https://doi.org/10.1007/978-981-19-7908-8
©TheAuthor(s),underexclusivelicensetoSpringerNatureSingaporePteLtd.2023
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewhole orpart ofthematerial isconcerned, specifically therights oftranslation, reprinting, reuse
ofillustrations, recitation, broadcasting, reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
Overview
Thisbookdealswiththemostimportantissueofrepresentationinmachinelearning
(ML). While learning class/cluster abstractions from the data using a machine,
it is important to represent the data in a form suitable for effective and efficient
machinelearning.Inthisbook,weproposetocoverawidevarietyofrepresentation
techniquesthatareimportantinboththeoryandpractice.
Inpracticalapplicationsofcurrentinterest,thedatatypicallyishighdimensional.
These applications include image classification, information retrieval, problem
solving in AI, biological and chemical structure analysis, and social network
analysis.Amajorproblemwithsuchhigh-dimensionaldataanalysisisthatmostof
thepopulartoolslike thek-nearestneighborclassifier, decisiontree classifier, and
severalclusteringalgorithmsthatdependoninterpatterndistancecomputationsfail
toworkwell.So,representingthedatainalower-dimensionalspaceisinevitable.
Popularly used dimensionality reduction techniques may be categorized as
follows:
1. Featureselectionschemes:Hereanappropriatesubsetofthegivenfeaturesetis
identifiedandusedinlearning.
2. Featureextractionschemes:Herelinearornonlinearcombinationsofthegiven
featuresareusedinlearning.
Someofthepopularlinearfeatureextractorsarebasedonprincipalcomponents,
randomprojections,andnonnegativematrixfactorization.Wecoverallthesetech-
niquesinthebook.Therearesomemisconceptionsintheliteratureonrepresenting
thedatausingasubsetofprincipalcomponents.Itistypicallybelievedthatthefirst
fewprincipalcomponentsmaketherightchoiceforclassifyingthedata.Weargue
andshowpractically,inthebook,howsuchapracticemaynotbecorrect.
It is argued in the literature that deep learning tools are the ideal choices for
nonlinear feature selection; also they can learn the representations automatically.
These tools include autoencoders and convolutional neural networks. We discuss
v
vi Preface
thesetoolsinthebook.Further,wearguethatitisdifficultevenforthedeeplearners
toautomaticallylearntherepresentations.
Wepresentexperimentalresultsonsomebenchmarkdatasetstoillustratevarious
ideas.
Audience
The coverage is meant for both students and teachers and helps practitioners in
implementingMLalgorithms.Itisintendedforseniorundergraduateandgraduate
students and researchers working in machine learning, data mining, and pattern
recognition. We present material in this book so that it is accessible to a wide
variety of readers with some basic exposure to undergraduate-levelmathematics.
Thepresentationisintentionallymadesimplertomakethereaderfeelcomfortable.
Organization
This book is organized as follows: Chapter 1 deals with a generic introduction
to machine learning (ML) and various concepts including feature engineering,
modelselection,modelestimation,modelvalidation,and modelexplanation.Two
importanttasksinMLareclassificationandclustering.So,Chap.2dealswiththe
representationofdataitems,classes,andclusters.
Nearest neighbor finding algorithms play an important role in several ML
tasks. However,findingnearestneighborsin high-dimensionalspaces can be both
time consumingandinaccurate.InChap.3, we dealwith nearestneighborfinding
algorithms using fractional norms and approximate nearest neighborcomputation
usinglocality-sensitivehashing.Weillustrateusingseveralbenchmarkdatasets.
Chapter 4 deals with feature selection and linear feature extractionschemes. It
includesdiscussiononprincipalcomponents,randomprojections,andnonnegative
matrix factorization.Nonlinear featureextractionschemes are gainingimportance
because of the deep learning architectures based on autoencoders and multilayer
perceptrons.ThesetopicsareexaminedinChap.5.
Bangalore,India M.N.Murty
Chennai,India M.Avinash
Contents
1 Introduction ................................................................... 1
1.1 MachineLearning(ML)System........................................ 1
1.2 MainStepsinanMLSystem ........................................... 3
1.2.1 DataCollection/Acquisition..................................... 3
1.2.2 FeatureEngineeringandRepresentation ....................... 7
1.2.3 ModelSelection.................................................. 14
1.2.4 ModelEstimation................................................ 14
1.2.5 ModelValidation................................................. 14
1.2.6 ModelExplanation............................................... 15
1.3 DataSetsUsed ........................................................... 15
1.4 Summary ................................................................. 16
References...................................................................... 16
2 Representation ................................................................ 17
2.1 Introduction .............................................................. 17
2.2 RepresentationinProblemSolving ..................................... 18
2.3 RepresentationofDataItems............................................ 19
2.4 RepresentationofClasses................................................ 25
2.5 RepresentationofClusters............................................... 26
2.6 Summary ................................................................. 28
References...................................................................... 28
3 NearestNeighborAlgorithms ............................................... 29
3.1 Introduction .............................................................. 29
3.2 NearestNeighborsinHigh-DimensionalSpaces ...................... 30
3.3 FractionalNorms......................................................... 38
3.4 LocalitySensitiveHashing(LSH)andApplications.................. 41
3.5 Summary ................................................................. 44
References...................................................................... 45
vii
viii Contents
4 RepresentationUsingLinearCombinations............................... 47
4.1 Introduction .............................................................. 47
4.2 FeatureSelection......................................................... 47
4.3 PrincipalComponentAnalysis.......................................... 52
4.4 RandomProjections...................................................... 56
4.5 Non-negativeMatrixFactorization...................................... 58
4.6 Summary ................................................................. 61
References...................................................................... 62
5 Non-linearSchemesforRepresentation.................................... 63
5.1 Introduction .............................................................. 63
5.2 OptimizationSchemesforRepresentation.............................. 63
5.3 Visualization.............................................................. 64
5.4 AutoencodersforRepresentation........................................ 74
5.5 ExperimentalResults:ORLDataSet ................................... 79
5.6 ExperimentalResults:MNISTDataSet ............................... 80
5.7 Summary ................................................................. 85
References...................................................................... 85
6 Conclusions.................................................................... 87
References...................................................................... 89
Index................................................................................ 91
