Table Of ContentTaeho Jo
Machine
Learning
Foundations
Supervised, Unsupervised, and
Advanced Learning
Machine Learning Foundations
Taeho Jo
Machine Learning
Foundations
Supervised, Unsupervised, and Advanced
Learning
TaehoJo
HongikUniversity
GarosuroCheongju,Korea(Republicof)
ISBN978-3-030-65899-1 ISBN978-3-030-65900-4 (eBook)
https://doi.org/10.1007/978-3-030-65900-4
©SpringerNatureSwitzerlandAG2021
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
This book is concerned with the concepts, the theories, and the algorithms of
machinelearning.InPartI,weprovidethefundamentalknowledgeaboutmachine
learning by exploring learning theories, evaluation schemes, and simple machine
learning algorithms. In Parts II and III, we describe the supervised learning
algorithms as the approaches to the classification and regression tasks, and the
unsupervisedonesasthosetotheclusteringtasks.InPartIV,wecoverthespecial
types of learning algorithms, and the mixture of the supervised and unsupervised
algorithmsasthefurtherstudy.Readersneedthebasiclevelofknowledgeaboutlin-
earalgebraandvectorcalculusforunderstandingthemachinelearningalgorithms
whereinputdataisalwaysgivenasanumericalvector.
We mention some agenda that provide motivations for writing this book as the
presidentinthecompanyforconsultingtheartificialintelligencetechniques,Alpha
LabAI.Themachinelearningalgorithmsbecameverypopularapproachestodata
mining tasks, because of their flexibility and adaptability, compared with the rule
based approaches. In the rule based systems, symbolic rules must be provided,
manually,whereasthemachinelearningalgorithmsgeneratetherulesautomatically
oncepreviousexamplesaregiven;themachinelearningbasedapproachesaremore
automatic than the rule based ones. The machine learning algorithms are applied
to tasks in various disciplines such as control system engineering, chemical engi-
neering, and business administration and management. In this book, we describe
themachinelearningalgorithmswiththeassumptionthattheinputdataisgivenas
anumericalvector.
This book consists of four parts: foundation, supervised learning, unsupervised
learning,andadvancedtopics.Inthefirstpart,wecoverthelinearalgebra,theraw
datatypes,andtheprimitivemachinelearningalgorithmsasthegeneralaspectsof
machinelearning.Inthesecondpart,wedescribefourkindsofsupervisedmachine
learning algorithms: K Nearest Neighbor, Probabilistic Learning, the Decision
Tree, and the SVM (Support Vector Machine). In the third part, we focus on
threekindsofunsupervisedlearningalgorithms:AHC(AgglomerativeHierarchical
Clustering) Algorithm, K Means Algorithm, and EM (Expectation Maximization)
Algorithm.Inthelastpart,weexploretheadvancedtopicsonthemachinelearning:
v
vi Preface
Fig.1 Organizationofthe
fourparts
the combination of machine learning algorithms, the mixture of the supervised
and unsupervised learning, the temporal sequence learning, and the reinforcement
learning.
This book is written intended for three groups of readers: students, professors,
and researchers. Senior undergraduate and graduate students will be able to study
the contents of machine learning by themselves with this book. For lecturers and
professorsinuniversities,thisbookmaybeusedasthemainmaterialforproviding
lectures on machine learning. This book is useful for researchers and system
developers in industrial organizations for developing intelligent systems using the
machine learning algorithms as tools. Therefore, this book is practical for people
in both worlds: the academic world and the industrial one, to study the machine
learning.
ThefourpartsofthebookareoutlinedinFig.1.
PartI:Foundation
The first part of this book is concerned with the linear algebra, the raw data types
before encoding them into numerical vectors, and the simple machine learning
algorithms. Because this part provides the foundation for studying the machine
learning, we need to understand the linear algebra. We study the raw data types,
suchastherelationaldata,thetextualdata,andtheimagedata,andtheprocessof
encoding them into numerical vectors. We study some primitive machine learning
algorithmsforwarminguptostudythemainonesthatarecoveredinPartsIIandIII.
Preface vii
Part I is intended to provide the basic foundations for understanding the machine
learningalgorithms.
Chapter1isconcernedwiththeoverviewofmachinelearningalgorithmsinthe
generalaspect.Webeginwiththeoverviewoftasks,suchasthedataclassification,
the data clustering, and the regression, to which we able to apply the machine
learning algorithms in the functional view. We describe briefly the four types of
machinelearningalgorithms:thesupervised,theunsupervised,thereinforced,and
thesemi-supervisedlearning.Weexploreotherareasthatarerelatedtothecurrent
area, by comparing it with them. Therefore, Chap.1 covers the entire aspect of
machinelearningalgorithmswiththeirfunctions,types,andrelatedareas.
Chapter 2 is concerned with the linear algebra, which is the mathematics that
provides the foundation for understanding the machine learning algorithms. It is
assumedthattheinputdataisgivenasanumericalvector,whileapplyingmachine
learning algorithms to real tasks. Chapter 2 covers mainly the operations and the
mathematicalpropertiesofthevectorandthematrix,andtherelationbetweenthem.
We will mention the two schemes of reducing the dimension of the input vector:
thesingularvaluedecomposition andtheprincipalcomponent analysis.Ifyouare
familiarwiththelinearalgebra,studyonlythetwodimensionreductionschemes.
Chapter 3 is concerned with the raw data types, which are in the status before
encoding them into numerical vectors. The relational data that consists of records
isthetypicalrawdatatype,anditisrelativelyeasytoencodethemintonumerical
vectors. We mention the textual data as the most popular raw data type in the real
world and study the process of indexing a text into a list of words and encoding
it into a numerical vector. We also study the image data, which is one more kind
of raw data, and the process of encoding them into numerical vectors. Because it
is assumed that a numerical vector is given as the input, while applying machine
learning algorithms to real tasks, it is very important to encode the raw data into
numericalvectors.
Chapter 4 is concerned with the binary classification task and some simple
machine learning algorithms. While studying the machine learning algorithms, it
is assumed that the given task is a binary classification, and the regression or the
multipleclassificationmaybedecomposedintobinaryclassifications.Somesimple
machine learning algorithms, which are given as threshold rules or hypercubes,
will be mentioned for helping understanding the machine learning algorithms. As
atypicaltypeofmachinelearningalgorithm,thelinearclassifierisexpressedasa
linearequation.Thischapterisintendedtoprovidethewarm-upforunderstanding
themainmachinelearningalgorithms,whicharecoveredinPartsIIandIII.
PartII:Supervised Learning
Part II is concerned with the four representative supervised machine learning
algorithms: the KNN (K Nearest Neighbor), the decision tree, the probabilistic
learning, and the SVM (Support Vector Machine). The concept learning that was
viii Preface
mentioned in the previous part is the only toy machine learning algorithm that
provides the basic concept, rather than a real one, so we need to study the real
versions that are applicable to real tasks. This part begins with the KNN as the
simple and practical machine learning algorithm in Chap.5. In the subsequent
chapter,wedescribethemoreadvancedmachinelearningalgorithms:thedecision
tree, the probabilistic learning, and the SVM. Therefore, this part is intended to
studythefoursupervisedlearningalgorithms,chapterbychapter.
Chapter 5 is concerned with the KNN (K Nearest Neighbor) as the simple
and practical supervised learning algorithm. As its background, we mention the
case based reasoning and lazy learning, which underlie in the KNN. Based on the
background,weexplainindetailtheprocessofcarryingouttheclassificationsand
regressions of data items by the KNN. We present also the modified versions of
KNN as its variants. In this chapter, we assume that the KNN is the supervised
learningalgorithm,butwecoveritsunsupervisedversioninthenextpart.
Chapter6isconcernedwiththeprobabilisticlearning,whichisbasedoncondi-
tional probabilities of categories, given an example. We mention the probability
theory, which is called Bayes Theorem, in order to provide the background for
understanding the chapter. We describe in detail some probabilistic classifiers
such as Bayes Classifier and Naive Bayes as the popular and simple machine
learningalgorithms.WecovertheBayesianLearningasthemoreadvancedlearning
method than the two probabilistic learning algorithms. Among the probabilistic
learning algorithms, the Naive Bayes is used as the most popular classifier in real
classificationTasks,includingtextcategorization.
Chapter 7 is concerned with the decision tree that is characterized as one of
the symbolic machine learning algorithms. We begin this chapter by explaining
the information gain as the important measure for implementing the decision
tree. Afterward, we describe in detail the process of learning training examples
for building the decision tree and classifying novice examples. We present some
decisiontreevariants,suchastherandomforests,thedecisionlist,andthedecision
graph.Thedecisiontreeisthemostpopularamongmachinelearningalgorithmsin
spiteofitslessperformancethanothermachinelearningalgorithmsbecauseofits
symboliccharacterization.
Chapter8isconcernedwiththeSVMthatdefinesdualparallellinearboundaries
among classes. We study the Perceptron that is a typical linear classifier and the
basis for deriving the SVM. In the main part, we cover the classification process,
the constraints, and the learning process of the SVM. We survey some variants of
the SVM that are expansions of the standard SVM. The SVM is applicable to a
nonlinear classification problem, robustly, as the most popular machine learning
algorithm.
Preface ix
PartIII:UnsupervisedLearning
PartIIIisconcernedwiththeunsupervisedmachinelearningalgorithmsascluster-
ingtools.Assimpleclusteringalgorithms,westudytheAHCalgorithm,thedivisive
clusteringalgorithm,andtheonlineclusteringalgorithm.Wecontinuethestudyof
the unsupervised machine learning algorithms with the k means algorithm and its
variants. The EM algorithm is considered as a clustering paradigm, rather than a
specific algorithm, and some specific versions will be mentioned. We mention the
clustering algorithms as the unsupervised learning algorithms and the clustering
metric,whichisthemetricforevaluatingthem.
Chapter9isconcernedwiththethreesimpleclusteringalgorithms.Wemention
the AHC algorithm where the clustering proceeds in the bottom-up direction. We
mentionthedivisivealgorithmwheretheclusteringproceedsinthetop-downdirec-
tion as another kind of simple clustering algorithm. The online linear clustering,
which is a fast and simple clustering algorithm, has almost linear complexity in
caseofmakingasmallnumberofclusters.Weexploreothersophisticatedclustering
algorithmsinthesubsequentchapters.
Chapter 10 is concerned with the k means algorithm as the most popular
clustering algorithm. This chapter begins with the unsupervised version of the
KNN algorithm. With respect to the clustering process, we study the two main
versions of the k means algorithms: the crisp k means algorithm and the fuzzy k
meansalgorithm.Thekmedoidalgorithmismentionedasavariantofthekmeans
algorithm,andthestrategiesofselectingrepresentativeitemsarefocused.Notethat
thekmeansalgorithmisthesimplestversionofEMalgorithm,anditiscoveredin
thenextchapter.
Chapter11isconcernedwiththeEMalgorithm,whichisaframeofclustering
data items, rather than a specific algorithm. We will survey some probabilistic
distributionsasthebasisforunderstandingtheEMalgorithm.Assumingthateach
cluster has its own normal distribution, we describe in detail the EM algorithm,
which is extended from the k means algorithm. We mention the variations of EM
algorithmandthesemi-supervisedlearningalgorithmwithitscombinationwiththe
Naive Bayes. In Chaps.9–11, we cover the kinds of clustering algorithms, and in
Chap.12,wecovertheclusteringindexasaclusteringevaluationmetric.
Chapter 12 is concerned with the clustering index, which is the metric for
evaluating clustering algorithms. There are two kinds of similarities in evaluating
clustering algorithms: the intra-cluster similarity that should be maximized and
the inter-cluster similarity that should be minimized. The clustering index is the
combinationofthesetwokindsofsimilarities,followingthestyleofcombiningthe
recallandtheprecisionintotheF1measure.Theclusteringindexmaybeusedfor
tuning the external parameters of clustering algorithms, as well as for evaluating
them. The clustering index was initially proposed for evaluating the current data
organization.
x Preface
PartIV:AdvancedTopics
PartIVisconcernedwiththeadvancedtopicsinthemachinelearning.Weexplore
variousschemesofcombiningmultiplemachinelearningalgorithms.Weconsider
some advanced supervised learning including the semi-supervised learning. We
mention the two additional kinds of machine learning: the temporal learning and
the reinforced learning. Part IV is intended to describe in detail the combination
schemesandthespecialtypeofmachinelearningalgorithms.
Chapter 13 is concerned with the schemes of combining multiple machine
learning algorithms with each other. We mention the four main combination
schemes, depending on the role of the coordinator as the start of this chapter.
Afterward,wedescribethemeta-learningforreinforcingthecombinationschemes.
We present some partitions for implementing the ensemble learning. This chapter
focuses on the combination of supervised learning algorithms for classifying data
items.
Chapter 14 is concerned with the advanced supervised learning, including the
semi-supervised learning. We start with the three versions of the Kohonen Net-
works: the unsupervised version, the supervised version, and the semi-supervised
version. Afterward, as another kind of semi-supervised learning algorithm, we
presentsomemodelswiththecombinationofthesupervisedlearningalgorithmand
the unsupervised learning algorithm. We explore some techniques for advancing
the supervised learning, including the resampling and the co-learning. There are
two kinds of semi-supervised learning: a single model that is derived from an
unsupervisedlearningalgorithmandacombinedmodelthatconsistsofasupervised
learningalgorithmandanunsupervisedone.
Chapter 15 is concerned with the Hidden Markov Model as a special type of
machinelearningalgorithm.WedescribeDiscreteMarkovModelasthebackground
for understanding the Hidden Markov Model. In Section 3, we describe entirely
the Hidden Markov Model that performs the following three tasks: computing the
probability of a state sequence, generating state sequence given an observation
sequence, and estimating its parameters from a training set. We mention the text
topicanalysisasatypicaltasktowhichtheHiddenMarkovModelisapplied.This
chapter covers the special machine learning algorithm that considers the temporal
sequenceofinputvalues,differentlyfromothermachinelearningalgorithms.
Chapter 16 is concerned with the reinforced learning, which is the alternative
kind of machine learning to both the supervised learning and the unsupervised
learning. The reinforced learning is one where the critics is given after making
the decision to the input, instead of its target label. The table of entries, each of
whichconsistsofaninputvectorandrewardscorescorrespondingtodecisions,is
called Q table. To an initial input, a decision is made at random, and it is done
subsequently based on the Q table. We consider the case where the reward is
variabletothesameinput,dependingontheapplicationarea.Thispartprovidesthe
foundationforunderstandingthemachinelearningalgorithms.Inthispart,weneed
tounderstandthelinearalgebrathatisconcernedwiththevectorandthematrixas
