Introduction to machine learning

639 Pages·2014·3.44 MB·English

by  Ethem Alpaydin

Introduction
to
Machine
Learning
Third
Edition
AdaptiveComputationandMachineLearning
ThomasDietterich,Editor
ChristopherBishop,DavidHeckerman,MichaelJordan,andMichael
Kearns,AssociateEditors
AcompletelistofbookspublishedinTheAdaptiveComputationand
MachineLearningseriesappearsatthebackofthisbook.
Introduction
to
Machine
Learning
Third
Edition
Ethem Alpaydın
The MIT Press
Cambridge, Massachusetts
London, England
©2014MassachusettsInstituteofTechnology
Allrightsreserved. Nopartofthisbookmaybereproducedinanyformbyany
electronicormechanicalmeans(includingphotocopying,recording,orinforma-
tionstorageandretrieval)withoutpermissioninwritingfromthepublisher.
Forinformationaboutspecialquantitydiscounts,pleaseemail
[email protected].
Typesetin10/13LucidaBrightbytheauthorusingLATEX2ε.
PrintedandboundintheUnitedStatesofAmerica.
LibraryofCongressCataloging-in-PublicationInformation
Alpaydin,Ethem.
Introductiontomachinelearning/EthemAlpaydin—3rded.
p. cm.
Includesbibliographicalreferencesandindex.
ISBN978-0-262-02818-9(hardcover: alk. paper)
1. Machinelearning. I.Title
Q325.5.A462014
006.3’1—dc23 2014007214
CIP
10987654321
Brief Contents
1 Introduction 1
2 SupervisedLearning 21
3 BayesianDecisionTheory 49
4 ParametricMethods 65
5 MultivariateMethods 93
6 DimensionalityReduction 115
7 Clustering 161
8 NonparametricMethods 185
9 DecisionTrees 213
10 LinearDiscrimination 239
11 MultilayerPerceptrons 267
12 LocalModels 317
13 KernelMachines 349
14 GraphicalModels 387
15 HiddenMarkovModels 417
16 BayesianEstimation 445
17 CombiningMultipleLearners 487
18 ReinforcementLearning 517
19 DesignandAnalysisofMachineLearningExperiments 547
A Probability 593

Contents
Preface xvii
Notations xxi
1 Introduction 1
1.1 WhatIsMachineLearning? 1
1.2 ExamplesofMachineLearningApplications 4
1.2.1 LearningAssociations 4
1.2.2 Classification 5
1.2.3 Regression 9
1.2.4 UnsupervisedLearning 11
1.2.5 ReinforcementLearning 13
1.3 Notes 14
1.4 RelevantResources 17
1.5 Exercises 18
1.6 References 20
2 SupervisedLearning 21
2.1 LearningaClassfromExamples 21
2.2 Vapnik-ChervonenkisDimension 27
2.3 ProbablyApproximatelyCorrectLearning 29
2.4 Noise 30
2.5 LearningMultipleClasses 32
2.6 Regression 34
2.7 ModelSelectionandGeneralization 37
2.8 DimensionsofaSupervisedMachineLearningAlgorithm 41
2.9 Notes 42
viii Contents
2.10 Exercises 43
2.11 References 47
3 BayesianDecisionTheory 49
3.1 Introduction 49
3.2 Classification 51
3.3 LossesandRisks 53
3.4 DiscriminantFunctions 55
3.5 AssociationRules 56
3.6 Notes 59
3.7 Exercises 60
3.8 References 64
4 ParametricMethods 65
4.1 Introduction 65
4.2 MaximumLikelihoodEstimation 66
4.2.1 BernoulliDensity 67
4.2.2 MultinomialDensity 68
4.2.3 Gaussian(Normal)Density 68
4.3 EvaluatinganEstimator: BiasandVariance 69
4.4 TheBayes’Estimator 70
4.5 ParametricClassification 73
4.6 Regression 77
4.7 TuningModelComplexity: Bias/VarianceDilemma 80
4.8 ModelSelectionProcedures 83
4.9 Notes 87
4.10 Exercises 88
4.11 References 90
5 MultivariateMethods 93
5.1 MultivariateData 93
5.2 ParameterEstimation 94
5.3 EstimationofMissingValues 95
5.4 MultivariateNormalDistribution 96
5.5 MultivariateClassification 100
5.6 TuningComplexity 106
5.7 DiscreteFeatures 108
5.8 MultivariateRegression 109
5.9 Notes 111
5.10 Exercises 112
Contents ix
5.11 References 113
6 DimensionalityReduction 115
6.1 Introduction 115
6.2 SubsetSelection 116
6.3 PrincipalComponentAnalysis 120
6.4 FeatureEmbedding 127
6.5 FactorAnalysis 130
6.6 SingularValueDecompositionandMatrixFactorization 135
6.7 MultidimensionalScaling 136
6.8 LinearDiscriminantAnalysis 140
6.9 CanonicalCorrelationAnalysis 145
6.10 Isomap 148
6.11 LocallyLinearEmbedding 150
6.12 LaplacianEigenmaps 153
6.13 Notes 155
6.14 Exercises 157
6.15 References 158
7 Clustering 161
7.1 Introduction 161
7.2 MixtureDensities 162
7.3 k-MeansClustering 163
7.4 Expectation-MaximizationAlgorithm 167
7.5 MixturesofLatentVariableModels 172
7.6 SupervisedLearningafterClustering 173
7.7 SpectralClustering 175
7.8 HierarchicalClustering 176
7.9 ChoosingtheNumberofClusters 178
7.10 Notes 179
7.11 Exercises 180
7.12 References 182
8 NonparametricMethods 185
8.1 Introduction 185
8.2 NonparametricDensityEstimation 186
8.2.1 HistogramEstimator 187
8.2.2 KernelEstimator 188
8.2.3 k-NearestNeighborEstimator 190
8.3 GeneralizationtoMultivariateData 192