Hyperparameter Tuning for Machine and Deep Learning with R: A Practical Guide

327 Pages·2023·4.357 MB·English

by  Eva Bartz, Thomas Bartz-Beielstein, Martin Zaefferer, Olaf Mersmann

Eva Bartz · Thomas Bartz-Beielstein ·
Martin Zaefferer ·
Olaf Mersmann   Editors
Hyperparameter
Tuning
for Machine and
Deep Learning
with R
A Practical Guide
Hyperparameter Tuning for Machine and Deep
Learning with R
· ·
Eva Bartz Thomas Bartz-Beielstein
·
Martin Zaefferer Olaf Mersmann
Editors
Hyperparameter Tuning
for Machine and Deep
Learning with R
A Practical Guide
Editors
EvaBartz ThomasBartz-Beielstein
Bartz&BartzGmbH InstituteforDataScience,Engineering,
Gummersbach,Germany andAnalytics
THKöln
MartinZaefferer Gummersbach,Germany
Bartz&BartzGmbHandwithInstitutefor
DataScience,Engineering,andAnalytics, OlafMersmann
THKöln InstituteforDataScience,Engineering,
Gummersbach,Germany andAnalytics
THKöln
DualeHochschuleBaden-Württemberg
Gummersbach,Germany
Ravensburg
Ravensburg,Germany
ISBN 978-981-19-5169-5 ISBN 978-981-19-5170-1 (eBook)
https://doi.org/10.1007/978-981-19-5170-1
©TheEditor(s)(ifapplicable)andTheAuthor(s)2023.Thisbookisanopenaccesspublication.
OpenAccessThisbookislicensedunderthetermsoftheCreativeCommonsAttribution4.0International
License(http://creativecommons.org/licenses/by/4.0/),whichpermitsuse,sharing,adaptation,distribu-
tionandreproductioninanymediumorformat,aslongasyougiveappropriatecredittotheoriginal
author(s)andthesource,providealinktotheCreativeCommonslicenseandindicateifchangeswere
made.
Theimagesorotherthirdpartymaterialinthisbookareincludedinthebook’sCreativeCommonslicense,
unlessindicatedotherwiseinacreditlinetothematerial.Ifmaterialisnotincludedinthebook’sCreative
Commonslicenseandyourintendeduseisnotpermittedbystatutoryregulationorexceedsthepermitted
use,youwillneedtoobtainpermissiondirectlyfromthecopyrightholder.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Foreword
Hyperparameter tuning? Is this relevant in practice? Is it not rather an academic
gimmick?Thatthelatterisnotthecasehasbeenknownformanyyears.Ontheother
hand,itismostlyunclearwhatexactlythislookslikeinpractice.Whichprocedures
depend on which hyperparameters? How sensitive are the procedures to different
settings of their hyperparameters? And does that in turn depend on which data
constellations are available? How can users develop a good feeling for being on
therighttrackwhentuning?Answerstothesequestionsarenotonlyexpectedwhen
it comes to optimally performing tuning per se, but also when it comes to making
the tuning process transparent, i.e., answering the question why, after all, this and
notthathyperparameterconstellationwaschosen.
Thisbookdeliversanswerstotheabovequestions,someofwhichwerecompiled
as part of a study funded by the Federal Statistical Office of Germany. The
contributed case studies and associated scripts also enable practitioners to repro-
duce the described tuning procedures and apply them themselves. The presented
insights, cross-references, experiences, and recommendations will contribute to a
better understanding of hyperparameter tuning in machine learning and to gain
transparency.
Wiesbaden,Germany FlorianDumpert
March2022
v
Contents
1 Introduction .................................................. 1
EvaBartz
PartI Theory
2 Tuning:Methodology .......................................... 7
ThomasBartz-Beielstein,MartinZaefferer,andOlafMersmann
2.1 IntroductiontoHyperparameterTuning .................... 7
2.2 PerformanceMeasuresforHyperparameterTuning .......... 11
2.2.1 Metrics ......................................... 11
2.2.2 PerformanceMeasures ........................... 12
2.2.3 MeasuresforClassification ........................ 13
2.2.4 MeasuresforRegression .......................... 14
2.3 HyperparameterTuning .................................. 15
2.4 ModelSelectionandAssessment .......................... 19
2.5 TunabilityandComplexity ............................... 20
2.6 TheBasicHPTProcess .................................. 21
2.7 PracticalConsiderations .................................. 23
2.7.1 SomeThoughtsonCrossValidation ................ 23
2.7.2 ReplicabilityandStochasticity ..................... 24
2.7.3 ImplementationinR ............................. 24
3 Models ....................................................... 27
ThomasBartz-BeielsteinandMartinZaefferer
3.1 MethodsandHyperparameters ............................ 27
3.2 k-NearestNeighbor ...................................... 29
3.2.1 Description ..................................... 29
3.2.2 Hyperparametersofk-NearestNeighbor ............ 30
3.3 RegularizedRegression(ElasticNet) ....................... 32
3.3.1 Description ..................................... 32
3.3.2 HyperparametersofElasticNet .................... 33
3.4 DecisionTrees .......................................... 36
vii
viii Contents
3.4.1 Description ..................................... 36
3.4.2 HyperparametersofDecisionTrees ................ 36
3.5 RandomForest ......................................... 40
3.5.1 Description ..................................... 40
3.5.2 HyperparametersofRandomForests ............... 40
3.6 GradientBoosting(xgboost) .............................. 45
3.6.1 Description ..................................... 45
3.6.2 HyperparametersofGradientBoosting ............. 46
3.7 SupportVectorMachines ................................. 53
3.7.1 Description ..................................... 53
3.7.2 HyperparametersoftheSVM ...................... 53
3.8 DeepNeuralNetworks ................................... 58
3.8.1 Description ..................................... 58
3.8.2 HyperparametersofDeepNeuralNetworks ......... 59
3.9 SummaryandDiscussion ................................. 66
4 HyperparameterTuningApproaches ............................ 71
ThomasBartz-BeielsteinandMartinZaefferer
4.1 HyperparameterTuning:ApproachesandGoals ............. 71
4.2 SpecialCase:MonotonousHyperparameters ................ 73
4.3 Model-FreeSearch ...................................... 76
4.3.1 ManualSearch .................................. 76
4.3.2 UndirectedSearch ............................... 77
4.3.3 DirectedSearch ................................. 78
4.4 Model-BasedSearch ..................................... 78
4.5 SequentialParameterOptimizationToolbox ................. 79
4.5.1 spotasanOptimizer ............................. 81
4.5.2 spot’sInitialPhase ............................... 83
4.5.3 TheFunctionspotLoop ......................... 88
4.5.4 EnteringtheMainLoop .......................... 89
4.5.5 FinalSteps ...................................... 100
4.6 Kriging ................................................ 100
4.6.1 TheKrigingModel .............................. 101
4.6.2 KrigingPrediction ............................... 102
4.6.3 ExpectedImprovement ........................... 103
4.6.4 InfillCriteriawithNoisyData ..................... 104
4.6.5 spot’sWorkhorse:Kriging ........................ 105
4.6.6 krigingLikelihood ............................... 111
4.6.7 Predictions ..................................... 113
4.7 ProgramCode .......................................... 114
5 RankingandResultAggregation ............................... 121
ThomasBartz-Beielstein, OlafMersmann,
andSowmyaChandrasekaran
5.1 ComparingAlgorithms ................................... 121
5.2 Ranking ............................................... 122
Contents ix
5.3 RankAggregation ....................................... 127
5.4 ResultAnalysis ......................................... 131
5.5 StatisticalInference ..................................... 132
5.6 Definitions ............................................. 133
5.6.1 HypothesisTesting ............................... 133
5.6.2 Power .......................................... 136
5.6.3 p-Value ........................................ 136
5.6.4 EffectSize ...................................... 137
5.6.5 Sample Size Determination and Power
Calculations .................................... 138
5.6.6 Issues .......................................... 140
5.7 Severity ................................................ 143
5.7.1 Motivation ...................................... 143
5.7.2 Severity:Definition .............................. 144
5.7.3 TwoExamples .................................. 146
5.7.4 Discussionofthe80%Threshold .................. 150
5.7.5 ACommentontheNormalityAssumption .......... 150
5.8 Severity:ApplicationinBenchmarking ..................... 151
5.8.1 ExperimentI:Rosenbrock ........................ 152
5.8.2 ExperimentII:Freudenstein-Roth .................. 156
5.8.3 Experiment III: Powell’s Badly Scaled Test
Function ....................................... 156
5.9 SummaryandDiscussion ................................. 160
PartII Applications
6 HyperparameterTuningandOptimizationApplications .......... 165
ThomasBartz-Beielstein
6.1 SurrogateOptimization .................................. 165
6.2 HyperparameterTuninginMachineandDeepLearning ....... 168
6.3 HPTSoftwareTools ..................................... 171
6.4 SummaryandDiscussion ................................. 172
7 HyperparameterTuninginGermanOfficialStatistics ............ 177
FlorianDumpertandElenaSchmidt
7.1 OfficialStatistics ........................................ 177
7.2 MachineLearninginOfficialStatistics ..................... 180
7.3 ChallengesinTuningfortheFederalStatisticalOffice ........ 180
7.4 DealingwiththeChallenges .............................. 184
8 CaseStudyI:TuningRandomForest(Ranger) .................. 187
ThomasBartz-Beielstein, SowmyaChandrasekaran,
FrederikRehbach,andMartinZaefferer
8.1 Introduction ............................................ 188
8.2 DataDescription ........................................ 190
x Contents
8.2.1 TheCensusDataSet ............................. 190
8.2.2 getDataCensus:GettingtheDatafromOML ..... 194
8.3 ExperimentalSetupandConfigurationoftheRandom
ForestModel ........................................... 197
8.3.1 getMlConfig: Configuration of the ML
Models ......................................... 197
8.3.2 ImplementationDetails:getMlConfig ........... 198
8.4 ObjectiveFunction(ModelPerformance) ................... 203
8.4.1 PerformanceMeasures ........................... 203
8.4.2 HandlingErrors ................................. 204
8.4.3 ImputationofMissingData ....................... 205
8.4.4 getObjf:TheObjectiveFunction ................. 205
8.5 spot:ExperimentalSetupfortheHyperparameterTuner ..... 207
8.6 Tunability .............................................. 210
8.6.1 Progress ........................................ 210
8.6.2 evalParamCensus: Comparing Default
andTunedParametersonTestData ................. 211
8.7 AnalyzingtheRandomForestTuningProcess ............... 213
8.8 Severity:ValidatingtheResults ........................... 217
8.9 SummaryandDiscussion ................................. 217
8.10 ProgramCode .......................................... 218
9 CaseStudyII:TuningofGradientBoosting(xgboost) ............ 221
ThomasBartz-Beielstein, SowmyaChandrasekaran,
andFrederikRehbach
9.1 Introduction ............................................ 221
9.2 DataDescription ........................................ 222
9.3 getMlConfig:ExperimentalSetupandConfiguration
oftheGradientBoostingModel ........................... 222
9.3.1 getMlrTask:ProblemDesignandDefinition
oftheMachineLearningTask ..................... 223
9.3.2 getModelConf Algorithm
Design—HyperparametersoftheModels ........... 223
9.3.3 getMlrResample:TrainingandTestData ........ 225
9.4 ObjectiveFunction(ModelPerformance) ................... 225
9.5 spot:ExperimentalSetupfortheHyperparameterTuner ..... 225
9.6 Tunability .............................................. 226
9.6.1 Progress ........................................ 226
9.6.2 evalParamCensus: Comparing Default
andTunedParametersonTestData ................. 228
9.7 AnalyzingtheGradientBoostingTuningProcess ............ 229
9.8 Severity:ValidatingtheResults ........................... 232
9.9 SummaryandDiscussion ................................. 233
9.10 ProgramCode .......................................... 233
Contents xi
10 CaseStudyIII:TuningofDeepNeuralNetworks ................. 235
ThomasBartz-Beielstein, SowmyaChandrasekaran,
andFrederikRehbach
10.1 Introduction ............................................ 235
10.2 DataDescription ........................................ 238
10.2.1 getDataCensus: Getting the Data
fromOpenML ................................... 238
10.2.2 getGenericTrainValTestData:Split
DatainTrain,Validation,andTestData ............. 238
10.2.3 genericDataPrep:Spec ...................... 239
10.3 Experimental Setup and Configuration of the Deep
LearningModels ........................................ 244
10.3.1 getKerasConf: keras and Tensorflow
Configuration ................................... 244
10.3.2 getModelConf:DLHyperparameters ............ 246
10.3.3 TheNeuralNetwork ............................. 247
10.4 funKerasGeneric:TheObjectiveFunction .............. 251
10.5 spot:ExperimentalSetupfortheHyperparameterTuner ..... 251
10.6 Tunability .............................................. 254
10.6.1 Progress ........................................ 255
10.6.2 evalParamCensus: Comparing Default
andTunedParametersonTestData ................. 257
10.7 AnalysingtheDeepLearningTuningProcess ............... 258
10.8 Severity:ValidatingtheResults ........................... 261
10.9 SummaryandDiscussion ................................. 262
10.10 ProgramCode .......................................... 265
11 CaseStudyIV:TunedReinforcementLearning(inPYTHON) ...... 271
MartinZaeffererandSowmyaChandrasekaran
11.1 Introduction ............................................ 272
11.2 MaterialsandMethods ................................... 272
11.2.1 Software ....................................... 272
11.2.2 TaskEnvironment:InvertedPendulum .............. 273
11.2.3 LearningAlgorithm .............................. 274
11.3 SettinguptheTuningExperiment ......................... 274
11.3.1 File:run.py ..................................... 274
11.3.2 TunedParameters ................................ 275
11.3.3 FurtherConfigurationofSPOT .................... 276
11.3.4 Post-processingandValidatingtheResults .......... 277
11.4 Results ................................................ 278
11.5 Severity:ValidatingtheResults ........................... 279
11.6 SummaryandDiscussion ................................. 281