Machine learning strategies for multi-step-ahead time series forecasting

201 Pages·2014·6 MB·English

Machine learning strategies for
multi-step-ahead
time series forecasting
A thesis submitted for the degree of
Doctor of Philosophy
by
Souhaib Ben Taieb
Départementd’Informatique
UniversitéLibredeBruxelles
Belgium
September2014
Declaration
ThisthesishasbeenwrittenunderthesupervisionofProf. GianlucaBontempi(UniversitéLibrede
Bruxelles,Belgium)andProf. RobJ.Hyndman(MonashUniversity,Australia). Themembersof
theJuryare:
• Prof. GianlucaBontempi(UniversitéLibredeBruxelles,Belgium)
• Prof. YvesDesmet(UniversitéLibredeBruxelles,Belgium)
• Prof. RobJ.Hyndman(MonashUniversity,Australia)
• Prof. MaartenJansen(UniversitéLibredeBruxelles,Belgium)
• Prof. TomLenaerts(UniversitéLibredeBruxelles,Belgium)
• Prof. TimoTeräsvirta(AarhusUniversity,Denmark)
• Prof. MichelVerleysen(UniversitéCatholiquedeLouvain,Belgium)
Iherebydeclarethatthisthesiscontainsnomaterialwhichhasbeenacceptedfortheawardofany
other degree or diploma in any university or equivalent institution, and that, to the best of my
knowledgeandbelief,thisthesiscontainsnomaterialpreviouslypublishedorwrittenbyanother
person,exceptwhereduereferenceismadeinthetextofthethesis.
SouhaibBenTaieb
ii
TomysonHaroneandmyfuturechildren
Itisacapitalmistaketotheorizebeforeonehasdata. Insensiblyonebeginstotwistfactsto
suittheories,insteadoftheoriestosuitfacts. —ArthurConanDoyle,SherlockHolmes.
Thescientistdoesnotstudynaturebecauseitisuseful;hestudiesitbecausehedelightsinit,
andhedelightsinitbecauseitisbeautiful. —HenriPoincaré.
I was like a boy playing on the sea-shore, and diverting myself now and then finding a
smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all
undiscoveredbeforeme. —IsaacNewton.
Iseeapattern,butmyimaginationcannotpicturethemakerofthatpattern. Iseeaclock,
but I cannot envision the clockmaker. The human mind is unable to conceive of the four
dimensions,sohowcanitconceiveofaGod,beforewhomathousandyearsandathousand
dimensionsareasone? —AlbertEinstein.
Abstract
Howmuchelectricityisgoingtobeconsumedforthenext24hours? Whatwillbethetemperature
for the next three days? What will be the number of sales of a certain product for the next few
months? Answeringthesequestionsoftenrequiresforecastingseveralfutureobservationsfroma
givensequenceofhistoricalobservations,calledatimeseries.
Historically,timeseriesforecastinghasbeenmainlystudiedineconometricsandstatistics. Inthe
lasttwodecades,machinelearning,afieldthatisconcernedwiththedevelopmentofalgorithms
that can automatically learn from data, has become one of the most active areas of predictive
modelingresearch. Thissuccessislargelyduetothesuperiorperformanceofmachinelearning
prediction algorithms in many different applications as diverse as natural language processing,
speech recognition and spam detection. However, there has been very little research at the
intersectionoftimeseriesforecastingandmachinelearning.
Thegoalofthisdissertationistonarrowthisgapbyaddressingtheproblemofmulti-step-ahead
timeseriesforecastingfromtheperspectiveofmachinelearning. Tothatend,weproposeaseries
offorecastingstrategiesbasedonmachinelearningalgorithms.
Multi-step-aheadforecastscanbeproducedrecursivelybyiteratingaone-step-aheadmodel,or
directlyusingaspecificmodelforeachhorizon. Asafirstcontribution,weconductanin-depth
studytocomparerecursiveanddirectforecastsgeneratedwithdifferentlearningalgorithmsfor
differentdatageneratingprocesses. Moreprecisely,wedecomposethemulti-stepmeansquared
forecasterrorsintothebiasandvariancecomponents,andanalyzetheirbehaviorovertheforecast
horizon for different time series lengths. The results and observations made in this study then
guideusforthedevelopmentofnewforecastingstrategies.
Inparticular,wefindthatchoosingbetweenrecursiveanddirectforecastsisnotaneasytasksince
it involves a trade-off between bias and estimation variance that depends on many interacting
factors, including the learning model, the underlying data generating process, the time series
length and the forecast horizon. As a second contribution, we develop multi-stage forecasting
strategiesthatdonottreattherecursiveanddirectstrategiesascompetitors,butseektocombine
theirbestproperties. Moreprecisely,themulti-stagestrategiesgeneraterecursivelinearforecasts,
andthenadjusttheseforecastsbymodelingthemulti-stepforecastresidualswithdirectnonlinear
modelsateachhorizon,calledrectificationmodels. Weproposeafirstmulti-stagestrategy,thatwe
calledtherectifystrategy, whichestimatestherectificationmodelsusingthenearestneighbors
model. However,becauserecursivelinearforecastsoftenneedsmalladjustmentswithreal-world
timeseries,wealsoconsiderasecondmulti-stagestrategy,calledthebooststrategy,thatestimates
therectificationmodelsusinggradientboostingalgorithmsthatuseso-calledweaklearners.
Generatingmulti-stepforecastsusingadifferentmodelateachhorizonprovidesalargemodeling
flexibility. However,selectingthesemodelsindependentlycanleadtoirregularitiesintheforecasts
thatcancontributetoincreasetheforecastvariance. Theproblemisexacerbatedwithnonlinear
machinelearningmodelsestimatedfromshorttimeseries. Toaddressthisissue,andasathird
v
Machinelearningstrategiesformulti-step-ahead
timeseriesforecasting
contribution,weintroduceandanalyzemulti-horizonforecastingstrategiesthatexploittheinfor-
mationcontainedinotherhorizonswhenlearningthemodelforeachhorizon. Inparticular,to
selectthelagorderandthehyperparametersofeachmodel, multi-horizonstrategiesminimize
forecasterrorsovermultiplehorizonsratherthanjustthehorizonofinterest.
Wecomparealltheproposedstrategieswithboththerecursiveanddirectstrategies. Wefirstapply
a bias and variance study, then we evaluate the different strategies using real-world time series
fromtwopastforecastingcompetitions. Fortherectifystrategy,inadditiontoavoidingthechoice
betweenrecursiveanddirectforecasts,theresultsdemonstratethatithasbetter,oratleasthas
closeperformanceto, thebestoftherecursiveanddirectforecastsindifferentsettings. Forthe
multi-horizonstrategies,theresultsemphasizethedecreaseinvariancecomparedtosingle-horizon
strategies,especiallywithlinearorweaklynonlineardatageneratingprocesses. Overall,wefound
that the accuracy of multi-step-ahead forecasts based on machine learning algorithms can be
significantlyimprovedifanappropriateforecastingstrategyisusedtoselectthemodelparameters
andtogeneratetheforecasts.
Lastly,asafourthcontribution,wehaveparticipatedintheLoadForecastingtrackoftheGlobal
EnergyForecastingCompetition2012. Thecompetitioninvolvedahierarchicalloadforecasting
problemwherewewererequiredtobackcastandforecasthourlyloadsforaUSutilitywithtwenty
geographicalzones. Ourteam,TinTin,rankedfifthoutof105participatingteams,andwehave
beenawardedanIEEEPower&EnergySocietyaward.
vi
Acknowledgement
Lescinqannéesdemathèsededoctoratontétélesplusproductivestantauniveauprofessionnel
qu’auniveaupersonnel,etcecinotammentgrâceausoutien,àlapatienceetàladisponibilitéd’un
grandnombredepersonnesdemonentouragequejetiensàremercier.
Tout d’abord, ma famille, et tout particulièrement ma mère et mon père, à qui je dois tout (et
plusencore).Jetiensaussiàremerciermesfrèresetsoeurs,quichacunàleurmanièrem’ontaussi
beaucoupdonné.
Durantcescinqdernièresannées,j’aidûsouventm’absenterpourparticiperàplusieursconférences
à l’étranger. Je tiens particulièrement à remercier mon épouse pour son encouragement et sa
patiencelorsdetouscesdéplacements.
Il y a presque deux ans, notre petit Harone s’est joint à nous, et il a très bien choisi sa date de
naissance,enlaretardantd’unesemaine,jusqu’audernierjourdelacompétition“GEFCOM2012"à
laquellejeparticipais.MerciHaronedem’avoirpermisdeterminerlacompétition,maisaussipour
toutelajoieetlabonnehumeurquetum’asprocuréecesdeuxdernièresannées,enparticulier
durantlesmomentslesplusdifficilesdemathèsededoctorat.
J’ai une pensée particulière à tous les membres de ma famille que j’ai eu très peu l’occasion de
rencontrerdurantcesdernièresannées,etquej’aimeraisaussiremercierpourleurencouragement.
Enfin, les moments de joie et de bonheur que j’ai passé avec mes amis m’ont souvent permis
d’oublierlespériodesdifficiles.Jepenseparticulièrementànotregroupe,incluantTarik,Jamel,
Ayoube,AbdelrahmanetIbrahim;maisaussiàAhmed,Abdeslam,GabrieletHadrien.Letour
duMontBlanceffectuéavecGabrieletHadrien,ainsiquelevoyageàMarrakechavecmonfrère
AyoubetmoncousinAhmedresterontdesmomentsmémorables.Merciàtous!
After my family and friends, I would like to thank those who contributed more directly to my
thesis.
My foremost gratitude goes to Gianluca Bontempi for his great advice and support throughout
theentirePhDjourney. IamparticularlygratefultoGianlucaforhiswillingnessandpatienceto
givemecompletefreedomtochoosemyresearchtopics. Hisinsightfulunderstandingofmachine
learninghasallowedmetoprogressalot,andIthankhimforallhetaughtmeaboutthatsubject.
IamdeeplygratefultoRobHyndmanwhojoinedthisthesisasco-supervisor,twoyearsago. Robis
oneofthebestforecastersintheworld,andIhavebeenveryfortunatetoworkwithhim. Iwould
liketoexpressmydeepgratitudetoRobforallthesupportandencouragementheprovidedtome,
especially,hispatienceinansweringallmyquestionsduringverylateorveryearlySkypetalks1
thatwehadeveryweekforthelasttwoyears. Asaresult,ourparticipationtotheGEFCOM2012
competitionledtoexcellentresults.
1DuetothetimezonedifferencebetweenBrusselsandMelbourne.
vii
Machinelearningstrategiesformulti-step-ahead
timeseriesforecasting
IalsothankRobforthegreattimewehadwhiletravelingtoParis,SeoulandRome,discussing
about research and life for many hours. I also deeply appreciate Rob visiting me two times in
Brusselsevenhewasverytiredofallhisprevioustrips.
I would like also to thank Rob for welcoming me at Monash University in Melbourne for five
monthswhereIhadtheopportunitytoworkcloselywithhimandtomeetgreatpeople,including
Georges,Farshid,Anastasios,Julia,Slava,Mehmet,Alexander,Behrooz,Gary,andmanyothers. In
particular,playingsoccerwiththeredandblueteamswasaveryenjoyableexperience.
Morethanaco-supervisor,IconsiderRobasaclosefriendwhoisnotonlyabrilliantresearcher,
butalsoapersonwithanextremelyniceandsincerepersonality.
ResearchisateameffortandIwasfortunateenoughtoworkwithmanycollaborators. Inparticular,
IwouldliketothankAmirAtiyafromCairoUniversitywhoacceptedataveryearlystageofthe
thesistoprovidemeguidanceandsupport. IalsothankAmirfortakingthetimetovisitmein
Brussels,lastyear. GiorgioCoraniinvitedmeatIDSIAinLuganoforoneweek,whichwasavery
excitingexperienceandIamverygratefulforthat.
IthankallmycolleaguesoftheMachineLearningGroupandtheDepartmentofComputerScience
thatcontributedtocreateanenjoyabledailyatmosphere. Manythankstoallofyou: Angélique,
Ahmed,Alessia,Amine,Andrea,Benjamin,Bernard,Boris,Catharina,Claudio,Elisa,Emmanuel,
Gérald,Gianluca,Gilles,Gwënael,Ioannis,Jean,Jean-François,Jean-Sébastien,Joël,Liran,Luciano,
Martin,Maarten,Maryka,Matteo,Miguel,Nikita,Naim,Olivier,Pascaline,Patrick,Stephane,The
AnhHan,Thierry,Tom,Vandy,Véronique,Yann-Aël,Yves,...
I would like to thank all the people that have taken the time to read this thesis and that gave
mefeedback,includingGianluca,Rob,Catharina,Nikita,LiranandSlava. Aspecialthankshas
to be addressed to Catharina who accepted many times to give me feedback on my papers and
applications. In particular, she accepted to read this thesis while she was on holidays. Thanks
Catharina! IamalsoverygratefultoProf. RaymondDevillerswhohasproofreadthepreliminary
versionofthisthesisingreatdetails,eventhoughhewasonholidays.
Finally,myloveforscienceandeducationwasborninlargepartthankstomyexcellentmathematics
teacherinsecondaryschool,DidierPauwels. IwouldliketothankDidierforhisenthusiasmand
thetimehetakestoprovideanexcellenteducationforhisstudents.
Onthisday,19thofAugust2014,mybirthday,IamgratefulforbeingabletodowhatIlovetodo.
FinancialSupport: theworkpresentedinthisthesiswassupportedbytheBelgianNationalFunds
forScientificResearch(FNRS)throughaResearchFellowgrant.
viii
Contents
Declaration ii
Abstract v
Acknowledgement vii
1 Introduction 1
1.1 Timeseriesforecastingandmachinelearning . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Motivationsandaims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.1 Publicationsandconferences . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.2 Researchactivities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.3 Softwaredevelopment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
I Overview 6
2 Background 7
2.1 Learningfromdata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 Differentviewsandtypesoflearning . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.2 Theregressionlearningproblem . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.3 Thecureforoverfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.4 Thelearningprocedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Learningregressionalgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 Linearmodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Penalizedregressionsplines(P-Splines) . . . . . . . . . . . . . . . . . . . . . . 19
2.2.2 Neuralnetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.3 K-Nearestneighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.4 GradientBoosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3 Timeseriesforecasting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.2 Timeseriesdecomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.3 Thestatisticalforecastingperspective . . . . . . . . . . . . . . . . . . . . . . 26
2.3.4 Autoregressivemodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.5 Autoregressivemodelselection . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.3.6 Evaluatingforecastsaccuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3 Anoverviewofstrategiesformulti-step-aheadtimeseriesforecasting 32
3.1 Preamble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2 Multi-stepforecasting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3 Therecursiveanddirectforecastingstrategies . . . . . . . . . . . . . . . . . . . . . 33
ix
Machinelearningstrategiesformulti-step-ahead
timeseriesforecasting
3.4 Recursiveordirectforecasts? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.4.1 Linearmodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4.2 Nonlinearmodels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.5 Alternativeforecastingstrategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.5.1 Improvingrecursiveforecasts . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.5.2 Improvingdirectforecasts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5.3 Hybridforecasts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.6 Timeseriesforecastingwithmachinelearning . . . . . . . . . . . . . . . . . . . . . . 41
3.7 Summaryandconcludingremarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
II Contributions 45
4 Biasandvarianceanalysisformulti-stepforecasting 46
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.2 Meansquaredmulti-stepforecasterrordecomposition . . . . . . . . . . . . . . . . . 47
4.2.1 Furtherdecompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.3.1 Theoreticalanalysisfortwo-stepaheadforecasts . . . . . . . . . . . . . . . 50
4.3.2 MonteCarlosimulationsforh-stepaheadforecasts . . . . . . . . . . . . . . 52
Datageneratingprocesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Biasandvarianceestimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Modelselectionandestimation . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.4 Analysisoftherecursiveanddirectstrategies . . . . . . . . . . . . . . . . . . . . . . 56
4.4.1 ScenarioA:LinearmodelandlinearDGP . . . . . . . . . . . . . . . . . . . . . 60
4.4.2 ScenarioB:LinearmodelandnonlinearDGP . . . . . . . . . . . . . . . . . . . 62
4.4.3 ScenarioC:NonlinearmodelandlinearDGP . . . . . . . . . . . . . . . . . . . 63
4.4.4 ScenarioD:NonlinearmodelandnonlinearDGP . . . . . . . . . . . . . . . . . 67
4.4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.5 Concludingremarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5 Multi-stageforecastingstrategies 76
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.2 Multi-stageforecastingstrategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.3 Biasandvarianceanalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.3.1 ScenarioA:LinearmodelandlinearDGP . . . . . . . . . . . . . . . . . . . . . 83
5.3.2 ScenarioB:LinearmodelandnonlinearDGP . . . . . . . . . . . . . . . . . . . 87
5.3.3 ScenarioC:NonlinearmodelandlinearDGP . . . . . . . . . . . . . . . . . . . 90
5.3.4 ScenarioD:NonlinearmodelandnonlinearDGP . . . . . . . . . . . . . . . . . 91
5.3.5 Averagingstrategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.4 Real-dataexperiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.5 Concludingremarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6 Multi-horizonforecastingstrategies 108
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.2 Relatedwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.3 Themulti-horizonstrategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.3.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.4 Biasandvarianceanalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.5 Real-dataexperiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
x