Improving Classifier Generalization. Real-Time Machine Learning based Applications

181 Pages·2023·4.329 MB·English

by  Rahul Kumar Sevakula, Nishchal K. Verma

Studies in Computational Intelligence 989
Rahul Kumar Sevakula
Nishchal K. Verma
Improving
Classifier
Generalization
Real-Time Machine Learning based
Applications
Studies in Computational Intelligence
Volume 989
SeriesEditor
JanuszKacprzyk,PolishAcademyofSciences,Warsaw,Poland
The series “Studies in Computational Intelligence” (SCI) publishes
new developments and advances in the various areas of computational
intelligence—quickly and with a high quality. The intent is to cover the theory,
applications, and design methods of computational intelligence, as embedded in
the fields of engineering, computer science, physics and life sciences, as well as
themethodologiesbehindthem.Theseriescontainsmonographs,lecturenotesand
editedvolumesincomputationalintelligencespanningtheareasofneuralnetworks,
connectionist systems, genetic algorithms, evolutionary computation, artificial
intelligence, cellular automata, self-organizing systems, soft computing, fuzzy
systems,andhybridintelligentsystems.Ofparticularvaluetoboththecontributors
and the readership are the short publication timeframe and the world-wide
distribution,whichenablebothwideandrapiddisseminationofresearchoutput.
IndexedbySCOPUS,DBLP,WTIFrankfurteG,zbMATH,SCImago.
AllbookspublishedintheseriesaresubmittedforconsiderationinWebofScience.
·
Rahul Kumar Sevakula Nishchal K. Verma
Improving Classifier
Generalization
Real-Time Machine Learning based
Applications
RahulKumarSevakula NishchalK.Verma
SignalProcessing DepartmentofElectricalEngineering
WhoopInc IndianInstituteofTechnologyKanpur
Boston,MA,USA Kanpur,India
ISSN 1860-949X ISSN 1860-9503 (electronic)
StudiesinComputationalIntelligence
ISBN 978-981-19-5072-8 ISBN 978-981-19-5073-5 (eBook)
https://doi.org/10.1007/978-981-19-5073-5
©SpringerNatureSingaporePteLtd.2023
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
Classificationalgorithmsformthebasisofdecision-makinginmostpatternrecog-
nitionproblems,e.g.imagerecognition,speechandspeakerrecognition,irisrecog-
nition, and spam mail detection. With the horizon of their applications expanding
at a fast pace, the need for further research has only increased. This fact becomes
particularly true because (a) each application poses its own set of challenges and
(b)onewouldalwaysfindaclassifierwithaparticularimprovisationthatbestsuits
thesituation.Nomatterwhichclassificationapproachisused,generalizationisan
importantaspect.Generalizationessentiallyindicateshowwellthetrainedclassifier
worksinrealtime,i.e.onunseentestdata.
This monograph begins with the fundamentals of classifiers, bias-variance
tradeoff, statistical learning theory (SLT), probably approximate correct (PAC)
framework,maximummarginclassifiers,andpopularmethodswhichimprovegener-
alization like regularization, boosting, transfer learning, dropout in deep learning,
etc.Furthermore,themonographsolvesfourindependentproblemsthathavegreat
relevanceforcertainreal-timeapplications.
Thefirstpartofthemonographaimsatfindingclassifierswhichexhibitextremely
lowvariance.Classificationalgorithmsaretraditionallydesignedtosimultaneously
reduce errors caused by bias as well as variance. In many situations, low variance
becomesextremelycrucialforgettingtangibleclassificationsolutionsandevenslight
overfittingcanhaveseriousconsequencesonthetestresults.Classifierswithlowvari-
ancehavetwomainadvantages:(1)theclassifierstatisticallymanagestokeepthe
testerrorsclosetothetrainingerror,and(2)theclassifierlearnseffectivelyevenwith
asmallnumberofsamples.Themonographintroducesaclassofclassifierscalled
MajorityVotePointClassifier(MVPC),whichonaccountofthelowerVapnikCher-
vonenkis(VC)dimensioncanexhibitlowervariancethanevenlinearclassifiers.The
monographcontributesbyestimatingatrendfortheMVPclassifier’sVCdimension,
andvalidatesitslowvarianceontworeal-timeproblems.
The monograph then focuses on the real-time application of condition-based
monitoring of machines using acoustic and vibration measurements. Signal data
acquired from machines are often found to change with time, wear and tear, and
subsequent repair of the machine. Classifiers are typically trained to perform the
v
vi Preface
decision-makingprocedureduringfaultdiagnosis/detection.Sincedatamaychange
withtime,lowgeneralizationerrorisessentialtoavoidoverfittingduringclassifica-
tion.Therefore,MVPCisseentobebestsuitedforthissituation.However,MVPC
hasalimitationthatitmaynotbeabletofitthedatasufficiently,andmayhavehigh
trainingerrors.Themonographpresentsanovelframeworkforpatternrecognition,
where novel procedures for optimal data source (sensitive position) identification,
dataacquisition,andfeatureselectionaretailoredtogivethebestpossibletraining
performancewiththeMVPclassifier.TheunderstandinghereisthatifMVPgiveslow
trainingerror,real-timefaultdiagnosisofmachinesbecomesfeasiblewithconsistent
accuracy.Theintroducedframeworkwasexperimentallyimplementedandtestedfor
anaircompressorconditionmonitoringapplication;associatedreal-timeexperiments
showedasignificantimprovementinthereliabilityoffaultdetection.
ThethirdpartofthemonographfocusesondealingwithclassnoiseintheFuzzy
SupportVectorMachine(FSVM)classifier.FSVMisconsideredtobeasignificant
additionoversoftmarginSVMlikeC-SVM,fortheformercanguardagainstoutlier
sensitivity and the latter cannot. The ability of FSVM to absorb outliers strongly
depends on how well the training samples are assigned fuzzy membership values
(MVs).Traditionally,themembershipfunctions(MFs)usedforFSVMwerecustom-
madeforapplications,andMFsusedforonecouldingeneralnotbeusedforothers.
Toovercomethelimitation,GeneralPurposeMembershipFunctions(GPMFs)are
definedinthismonographasthoseMFswhichcanuniversallybeusedformultiple
applications, and which allow FSVM to statistically perform better than C-SVM.
ThemonographcontributestotheGPMFliteratureintwostages.Firstlywithhelp
of convex hulls, it presents a few limitations that FSVM faces while treating all
samplesofaclasswithasingleMF.Further,itrecommendsdifferentialtreatmentof
databycategorizingthemintotwofuzzysets:onecontainingpossiblenon-outliers
andtheothercontainingpossibleoutliers.Whilepossibleoutliersaremodeledwith
anormalMF,possiblenon-outliersarerecommendedtohaveaconstantMVof‘1’.
The chapter then introduces novel GPMFs which use clustering-based techniques
to detect possible outliers, and use Hausdorff Distance and pt-set to characterize
those possible outliers. To establish conclusions, the introduced GPMFs are thor-
oughlyevaluatedandstatisticallycomparedwithearlierGPMFsonnumerousreal-
worldbenchmarkdatasets.TheresultsshowthatproposedGPMFsnotonlyperform
significantly better in treating class noise, but also execute with efficient run time
complexity.
Finally, a novel scheme to introduce deep learning in Fuzzy Rule-based classi-
fiers(FRCs)ispresented.FRCshavegainedprominencefortheiruniqueabilityof
givinggoodclassificationperformance,andallowingexistingexpertknowledgetobe
usedconjointlywithtrainingdata.RecentinnovationsinDeepNeuralNetworksare
allowingresearcherstotacklesomeverycomplexproblemswithimprovedtheoret-
icalandempiricaljustifications,e.g.imageclassificationandaudioclassification.The
monographpresentsaschemetoincorporatestackeddenoisingsparseautoencoders
withintheFRCframework.Whilestackingofdenoisingsparseautoencodershelps
learnthecomplexnon-linearrelationshipsamongdataandrepresenttheinputdata
inareduced compact feature space, theframework builttoward FRC allows users
Preface vii
to input expert knowledge to the system. To make denoising sparse autoencoders
learn more effectively, data pre-processing strategies have been proposed. Further,
to improve the classification performance and rule reduction performance of the
FRC,threefine-tuningstrategieshavealsobeenproposed.Theproposedframework
istestedacrossreal-worldbenchmarkdatasets,andanelaboratecomparisonacross
literature shows that proposed methods are capable of building FRCs that provide
state-of-the-artaccuraciesand/orafewrules,aspertheuser’sdemand.
The monograph ends with an epilogue, on the use of autoencoders in transfer
learning, tumor classification, and condition monitoring problems. Furthermore,
pertainingtotheresearchcontributionsmadeherein,thedirectionsforpossiblefuture
workhavealsobeendiscussed.
Boston,USA RahulKumarSevakula
Kanpur,India NishchalK.Verma
Contents
1 Introduction ................................................... 1
1.1 Basics ................................................... 1
1.2 StatisticalLearningTheoryandVCDimension ................ 3
1.2.1 VCDimension ..................................... 3
1.2.2 Probably Approximate Correct (PAC) Learning
Framework ......................................... 4
1.2.3 GrowthFunction .................................... 6
1.3 Bias-VarianceTrade-Off .................................... 6
1.3.1 Occam’sRazor ..................................... 7
1.4 Outline .................................................. 8
References ..................................................... 9
2 MethodsUsedtoImproveGeneralizationPerformance ............ 11
2.1 MaximumMarginClassifier ................................ 12
2.1.1 UpperBoundonVCDimensionofLinearClassifiers .... 12
2.1.2 SupportVectorMachine(SVM) ....................... 13
2.2 BaggingandBoosting ...................................... 15
2.3 Semi-SupervisedLearning,ActiveLearning,andTransfer
Learning ................................................. 16
2.4 Regularization ............................................ 17
2.5 Stochastic Gradient Descent, Momentum, Adaptive
LearningRates,Dropout,SkipConnections,Etc. ............... 18
2.6 Conclusions .............................................. 20
References ..................................................... 20
3 MVPC—AClassifierwithVeryLowVCDimension ............... 23
3.1 MajorityVoteClassifier .................................... 23
3.1.1 TheMajorityVotePoint(MVP)Classifier .............. 24
3.1.2 ImplementationofMVPClassifier .................... 25
ix
x Contents
3.2 EvaluatingandComparingVCDimension .................... 25
3.2.1 UpperBoundonVCDimensionofMVPClassifier ...... 25
3.2.2 EmpiricalEstimationofVCDimension ................ 29
3.2.3 ComparingwithVCDimensionofLinearClassifiers ..... 32
3.3 CaseStudies .............................................. 32
3.3.1 RotatingMachineFaultDiagnosis ..................... 32
3.3.2 BiologicalData ..................................... 35
3.4 Summary ................................................ 38
References ..................................................... 39
4 FrameworkforReliableFaultDetectionwithSensorData ......... 41
4.1 LiteratureSurveyonFaultDiagnosisinRotatingMachines ...... 41
4.2 ProblemFormulationandSummary .......................... 44
4.3 Data Acquisition Framework to Simulate Real-Time
Environment .............................................. 44
4.4 DataPre-processing ....................................... 46
4.4.1 NormalizationRobusttoOutliers ...................... 47
4.5 FeatureExtraction ......................................... 51
4.6 FeatureSelectionAlgorithmUsingGraphicalIndices ........... 54
4.6.1 FeatureRanking .................................... 55
4.6.2 DatasetRejection ................................... 60
4.6.3 DatasetRetrieval .................................... 62
4.6.4 FeatureSelectionArchitecture ........................ 62
4.7 Classification ............................................. 63
4.8 SensitivePositionAnalysis(SPA) ............................ 64
4.8.1 ParameterRangeIdentification(PRI) .................. 65
4.9 CaseStudyonAirCompressorFaultDetection ................ 66
4.9.1 LeakageInletValve(LIV)FaultDetection .............. 66
4.9.2 LeakageOutletValve(LOV)FaultDetection ........... 69
4.9.3 OfflineTesting ..................................... 70
4.9.4 Real-TimeTesting .................................. 72
4.10 Summary ................................................ 73
References ..................................................... 74
5 MembershipFunctionsforFuzzySupportVectorMachine
inaNoisyEnvironment ......................................... 77
5.1 DealingwithClassNoise ................................... 78
5.2 FuzzySupportVectorMachine(FSVM) ...................... 79
5.2.1 AvailableMembershipFunctionsforFSVM ............ 80
5.2.2 LimitationsofEarlierMembershipFunctions ........... 81
5.2.3 ConvexHullAnalysisonFSVM ...................... 82
5.3 Set Measures—Distance Measure Between Points
andNon-emptySets ....................................... 85
5.3.1 DistanceBetweenaPointandaNon-emptySet ......... 86
5.3.2 HausdorffDistance(HD) ............................. 87