Approximate Dynamic Programming for Dynamic Vehicle Routing

205 Pages·2017·24 MB·English

by  Marlin Wolf Ulmer

Marlin Wolf Ulmer
Approximate Dynamic
Programming for
Dynamic Vehicle Routing
123
Marlin Wolf Ulmer
Carl-Friedrich-Gauß-Fakultät
Technische UniversitätBraunschweig
Braunschweig
Germany
ISSN 1387-666X
Operations Research/Computer ScienceInterfaces Series
ISBN978-3-319-55510-2 ISBN978-3-319-55511-9 (eBook)
DOI 10.1007/978-3-319-55511-9
LibraryofCongressControlNumber:2017935959
©SpringerInternationalPublishingAG2017
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Foreword
Vehicle routing has received a tremendous attention in recent years. This attention
isdrivenbytoday’scustomerexpectationswithrespecttofastandreliableservice.
Onthemethodologicalsidetheachievementsaredrivenbytheremarkablesuccess
inovercomingthestandardvehicleroutingproblemtowardsformulationsof“rich”
vehicleroutingproblems.However,themoreoperationaldetailsareincorporatedin
the“richness”oftheproblemformulation,thelesslikelythesedetailswillpersistin
animplementationofthestaticoptimizationproblem.Byacceptingthefactthatthe
world is continuously changing, one will also have to accept that a dynamic and
stochastic problem formulation does suggest itself.
Typical sources of uncertainty in vehicle routing are stochastic customer
requests, stochastic demand of customers as well as stochastic service- and travel
times. All these figures may change over time while vehicles are already on the
road. Thus, operational planning has to either incorporate possible stochastic
changes before the implementation of a plan or subsequent decisions have to be
taken while vehicles actually operate. Today’s sensor and communication tech-
niques warrant up-to-date mass data for subsequent decision making. Deferring
decisionstothelatestpossiblepointintime comesalong withthehighestpossible
gain of information, but may lose out on advantages to be achieved by taking the
right decisions early.
Anticipationoffuturesystemstatescanbeseenasakeyfeatureforasuccessful
treatment of dynamic stochastic vehicle routing problems. Thus, in order to take
right decisions early, possible impacts have to be defined. The Markov decision
process is a suitable instrument for modeling state spaces and transitions within
these spaces. Optimization is still present in this modeling approach, but does step
back beyond the view to chains of state transitions forming trajectories from the
knowninitialstatetopossiblefinitestates.Thestochasticsimulationoftrajectories
produces objective function values for possible future states. These values can be
learned offline by means of simulation and can support online decision making
while vehicles are operating on the road.
While already in operation, one may take decisions by relying solely on the
offline information provided. Additionally, one may perform short online
look-ahead simulations in order to adjust as elaborately as possible to the actual
situation faced. Online look-ahead suffers from the relatively small number of
simulationstobecarriedoutatthetimewhenadecisionisdemanded.Thus,offline
aswellasonlineapproacheshavetheirvirtues.Offlineapproachesarerestrictedtoa
coarse grained state representation but depict the global decision space. Online
approaches model the actual decision situation in detail but are restricted to a
narrow scope of possible future outcomes. Compared to the vast number of pub-
lications in the area of static vehicle routing, both approaches have only received
minor attention so far.
The book at hand treats subsequent decision making in routing in novel and
innovativeways.MarlinUlmerintroducesthevehicleroutingproblem,stressesthe
need for a dynamic and stochastic problem formulation and describes sources of
uncertainty for vehicle routing. Of particular importance is the in-depth consider-
ationofapproximatedynamicprogramming.Althoughmuchdetailispresentedfor
theinterestedreader,MarlinUlmerfollowsaclearlineofargumentationpresenting
all this material in terms of the notion of anticipation. Computational experiments
elaborate on a routing problem with stochastic customer requests providing evi-
dence for the usefulness of the approach. This work constitutes a milestone in the
research of dynamic and stochastic vehicle routing.
Braunschweig, Germany Dirk Christian Mattfeld
May 2016
About this Book
This book is the result of manifold discussions with researches from two different
research fields: vehicle routing and stochastic dynamic optimization, namely,
approximatedynamicprogramming(ADP).Bothfieldshavealonghistoryandcan
drawalargebodyofresearchandmethods.Whileinvehicleroutingthedemandfor
stochastic dynamic optimization methods increases, a strong connection to
stochastic dynamic optimization (SDO) and ADP is still missing. The purpose of
thisbookistobuildabridgebetweenthesefieldsenablingthebroadapplicationof
ADP in the field of dynamic vehicle routing.
Thisbookprovidesastraightforwardoverviewforeveryresearcherinterestedin
stochastic dynamic vehicle routing problems (SDVRPs). The book is written for
both the applied researcher looking for suitable solution approaches for particular
problemsaswellasforthetheoreticalresearcherlookingforeffectiveandefficient
methods of stochastic dynamic optimization and approximate dynamic program-
ming. To this end, the book contains two parts. In the first part, the general
methodology required for modeling and approaching SDVRPs is presented. We
presentadaptedandnew,generalanticipatorymethodsofADPtailoredtotheneeds
ofdynamicvehiclerouting.Sincestochasticdynamicoptimizationisoftencomplex
and may not always be intuitive on first glance, we carefully accompany the
theoreticalSDO-methodologywithillustrativeexamplesfromthefieldofSDVRPs.
Thebookcontainsmorethan50explanatoryfigures.Still,wehavebeenmindfulto
maintainasuccinctredthreadthroughthefirstandsecondpartofthebook.Tothis
end, the theoretical methodology is self-contained and the arrangement of the
chaptersallowstoskipexamples,ifdesired. Attheendofeachchapter,webriefly
summarizethemainimplicationsandtheresultingstepstoexpectinthesubsequent
chapter.
The second part of this book then depicts the application of the theory to a
specificSDVRP.Theprocessstartsfromthereal-worldapplication.Wedescribea
SDVRP with stochastic customer requests often addressed in the literature (e.g.,
Bent and Van Hentenryck 2004; Thomas 2007). We show in detail how this
problem can be modeled as a Markov decision process and present several antic-
ipatory solution approaches based on ADP. In an extensive computational study,
we show the advantages of the presented approaches compared to conventional
heuristics. To allow deep insights in the functionality of ADP, we present a com-
prehensive analysis of the ADP-approaches.
Further highlights of this book:
(cid:129) This book gives a comprehensive overview over the real-world applications
demanding for anticipatory dynamic decision making. To this end, we present
many theoretical and practical sources highlighting the importance of decision
support for SDVRPs. We especially identify and motivate same-day delivery,
sharedmobility,healthcare, anddemandresponsivepassengertransportationas
promising future research areas.
(cid:129) We present a comprehensive literature review and analysis regarding SDVRPs.
This review extends current reviews by Pillac et al. (2013); Ritzinger et al.
(2015)andPsaraftisetal.(2015)byananalysisoftheSDVPRswithrespectto
problem characteristics and modeling as well as the degree of anticipation
provided by the applied solution approaches.
(cid:129) Besides showing how SDVRPs can be modeled as Markov Decision Processes
(MDPs), we give an overview how uncertain events can be modeled as
stochastic information. To this end, we analyze the literature and present the
general ways uncertain customer requests, demands, travel times, and services
times aremodeled. Wefurther presentexamples ofMDP-modelsfor thesefour
drivers of uncertainty.
(cid:129) Since the straightforward application of ADP to SDVRPs is challenging, we
adaptgeneralADP-methodstotherequirementsofSDVRPs.Wefurtherpresent
newADP-methods,notonlysuitableforSDVRPs,butformanyproblemswith
acomplexMDP-structure.Inacomprehensivecomputationalevaluation,wenot
onlyshowthesuperiorityofthesemethodscomparedtoconventionalheuristics,
but also present a profound analysis to reveal their advantageous and func-
tionality in detail.
We are positive that this book will function as a foundation in the field of
stochasticdynamicoptimizationforstochasticdynamicvehicleroutingproblems.It
will connect the fields of SDVRP and SDO and will enable researchers to identify
and apply suitable solution approaches leading to high quality anticipation in
dynamic vehicle routing.
Contents
1 Introduction... .... .... ..... .... .... .... .... .... ..... .... 1
1.1 Prescriptive Analytics.... .... .... .... .... .... ..... .... 3
1.2 Scope of This Work..... .... .... .... .... .... ..... .... 4
1.3 Outline of the Following Chapters .. .... .... .... ..... .... 4
1.4 A Recipe for ADP in SDVRPs. .... .... .... .... ..... .... 6
1.4.1 The Application.. .... .... .... .... .... ..... .... 6
1.4.2 The Model. ..... .... .... .... .... .... ..... .... 7
1.4.3 Anticipatory Approaches... .... .... .... ..... .... 8
Part I Dynamic Vehicle Routing
2 Rich Vehicle Routing: Environment. .... .... .... .... ..... .... 15
2.1 Vehicle Routing ... ..... .... .... .... .... .... ..... .... 15
2.2 RVPR: Characteristics and Definition.... .... .... ..... .... 16
2.3 RVRPs in Logistics Management... .... .... .... ..... .... 17
2.4 RVRPs in Hierarchical Decision Making . .... .... ..... .... 18
2.5 Recent Developments of the RVRP-Environment... ..... .... 19
2.5.1 E-Commerce and Globalization.. .... .... ..... .... 19
2.5.2 Urbanization and Demography .. .... .... ..... .... 20
2.5.3 Urban Environment and Municipal Regulations .. .... 21
2.5.4 Technology ..... .... .... .... .... .... ..... .... 22
2.5.5 Data and Forecasting.. .... .... .... .... ..... .... 22
2.6 Implications .. .... ..... .... .... .... .... .... ..... .... 24
3 Rich Vehicle Routing: Applications. .... .... .... .... ..... .... 25
3.1 General RVRP-Entities... .... .... .... .... .... ..... .... 26
3.1.1 Infrastructure.... .... .... .... .... .... ..... .... 26
3.1.2 Vehicles... ..... .... .... .... .... .... ..... .... 26
3.1.3 Customers. ..... .... .... .... .... .... ..... .... 27
3.2 Plans.... .... .... ..... .... .... .... .... .... ..... .... 27
3.3 Objectives.... .... ..... .... .... .... .... .... ..... .... 28
3.3.1 Costs. .... ..... .... .... .... .... .... ..... .... 28
3.3.2 Reliability . ..... .... .... .... .... .... ..... .... 28
3.3.3 Objective Measures... .... .... .... .... ..... .... 29
3.4 Constraints ... .... ..... .... .... .... .... .... ..... .... 29
3.4.1 Time Windows .. .... .... .... .... .... ..... .... 29
3.4.2 Working Hours .. .... .... .... .... .... ..... .... 29
3.4.3 Capacities . ..... .... .... .... .... .... ..... .... 30
3.5 Drivers of Uncertainty ... .... .... .... .... .... ..... .... 30
3.5.1 Travel Times.... .... .... .... .... .... ..... .... 30
3.5.2 Service Times ... .... .... .... .... .... ..... .... 30
3.5.3 Demands.. ..... .... .... .... .... .... ..... .... 31
3.5.4 Requests .. ..... .... .... .... .... .... ..... .... 31
3.6 Classification . .... ..... .... .... .... .... .... ..... .... 31
3.7 Service Vehicles... ..... .... .... .... .... .... ..... .... 32
3.8 Transportation Vehicles .. .... .... .... .... .... ..... .... 34
3.8.1 Passenger Transportation... .... .... .... ..... .... 34
3.8.2 Transportation of Goods ... .... .... .... ..... .... 35
3.9 Implications .. .... ..... .... .... .... .... .... ..... .... 37
3.9.1 Decision Support. .... .... .... .... .... ..... .... 37
3.9.2 Modeling of Planning Situations. .... .... ..... .... 38
3.9.3 Modeling of Uncertainty... .... .... .... ..... .... 38
3.9.4 Modeling of Subsequent Planning.... .... ..... .... 38
3.9.5 Modeling of Applications .. .... .... .... ..... .... 39
3.9.6 Modeling of Anticipation .. .... .... .... ..... .... 39
3.9.7 Anticipatory Methods . .... .... .... .... ..... .... 39
4 Modeling . .... .... .... ..... .... .... .... .... .... ..... .... 41
4.1 Stochastic Dynamic Decision Problem... .... .... ..... .... 41
4.1.1 Dynamic Decision Problems .... .... .... ..... .... 42
4.2 Markov Decision Process. .... .... .... .... .... ..... .... 43
4.2.1 Definition . ..... .... .... .... .... .... ..... .... 43
4.2.2 Decision Policies and Problem Realizations ..... .... 44
4.3 Stochastic Dynamic Vehicle Routing .... .... .... ..... .... 45
4.4 Modeling Planning Situations.. .... .... .... .... ..... .... 46
4.4.1 Decision State... .... .... .... .... .... ..... .... 46
4.4.2 Decision Making. .... .... .... .... .... ..... .... 48
4.5 Modeling Uncertainty.... .... .... .... .... .... ..... .... 49
4.5.1 Deterministic Modeling.... .... .... .... ..... .... 49
4.5.2 Travel Time..... .... .... .... .... .... ..... .... 49
4.5.3 Service Time.... .... .... .... .... .... ..... .... 51
4.5.4 Demands.. ..... .... .... .... .... .... ..... .... 51
4.5.5 Requests .. ..... .... .... .... .... .... ..... .... 52
4.5.6 Stochastic Transitions in SDVRPs.... .... ..... .... 54
4.6 Modeling SDVRPs as MDPs .. .... .... .... .... ..... .... 54
4.6.1 Decision Points.. .... .... .... .... .... ..... .... 55
4.6.2 Travel Times.... .... .... .... .... .... ..... .... 55
4.6.3 Service Times ... .... .... .... .... .... ..... .... 57
4.6.4 Demands.. ..... .... .... .... .... .... ..... .... 57
4.6.5 Requests .. ..... .... .... .... .... .... ..... .... 58
4.7 Vehicle Routing with Recourse Actions.. .... .... ..... .... 59
4.8 Route-Based Markov Decision Process... .... .... ..... .... 59
4.9 Implications .. .... ..... .... .... .... .... .... ..... .... 60
4.9.1 Properties of SDVRP.. .... .... .... .... ..... .... 60
4.9.2 Definition, Reconstruction, and Simulation . ..... .... 60
4.9.3 Anticipation and Prescriptive Analytics.... ..... .... 61
5 Anticipation... .... .... ..... .... .... .... .... .... ..... .... 63
5.1 Definition .... .... ..... .... .... .... .... .... ..... .... 63
5.2 Anticipation in SDVRPs.. .... .... .... .... .... ..... .... 64
5.3 Perfect Anticipation ..... .... .... .... .... .... ..... .... 65
5.3.1 Optimal Policies . .... .... .... .... .... ..... .... 65
5.3.2 Derivation of Optimal Policies .. .... .... ..... .... 66
5.3.3 Limitations ..... .... .... .... .... .... ..... .... 67
5.4 Classification of Anticipation .. .... .... .... .... ..... .... 67
5.4.1 Reactive Versus Non-reactive ... .... .... ..... .... 68
5.4.2 Implicit, Explicit, and Perfect ... .... .... ..... .... 68
5.4.3 Focus of Anticipation: Offline and Online.. ..... .... 68
5.5 Reactive Explicit Anticipation . .... .... .... .... ..... .... 69
6 Anticipatory Solution Approaches.. .... .... .... .... ..... .... 71
6.1 Non-reactive Anticipation. .... .... .... .... .... ..... .... 71
6.1.1 Non-reactive Implicit Anticipation.... .... ..... .... 71
6.1.2 Non-reactive Explicit Anticipation.... .... ..... .... 72
6.2 Reactive Anticipation.... .... .... .... .... .... ..... .... 72
6.2.1 Reactive Implicit Anticipation... .... .... ..... .... 73
6.2.2 Reactive Explicit Anticipation... .... .... ..... .... 74
6.2.3 Approximate Dynamic Programming.. .... ..... .... 74
6.2.4 Reducing the SDP.... .... .... .... .... ..... .... 75
6.2.5 Resulting Approaches . .... .... .... .... ..... .... 76
6.3 Lookahead and Rollout Algorithm .. .... .... .... ..... .... 76
6.3.1 Functionality.... .... .... .... .... .... ..... .... 78
6.3.2 Efficient Computing: Indifference Zone Selection . .... 78
6.4 Value Function Approximation. .... .... .... .... ..... .... 84
6.5 Approximate Value Iteration... .... .... .... .... ..... .... 85
6.5.1 Post-decision State Space Representation .. ..... .... 87
6.5.2 Aggregation..... .... .... .... .... .... ..... .... 88
6.5.3 Partitioning: Lookup Table . .... .... .... ..... .... 88
6.5.4 Efficient Approximation Versus Effective Decision
Making ... ..... .... .... .... .... .... ..... .... 89
6.5.5 Equidistant Lookup Table.. .... .... .... ..... .... 91
6.5.6 Weighted Lookup Table ... .... .... .... ..... .... 91
6.5.7 Dynamic Lookup Table.... .... .... .... ..... .... 92
6.6 Hybrid Reactive Explicit Anticipation ... .... .... ..... .... 96
6.6.1 Motivation. ..... .... .... .... .... .... ..... .... 97
6.6.2 Hybrid Rollout Algorithm.. .... .... .... ..... .... 97
6.6.3 Example: Comparison of Online and Hybrid RAs. .... 98
7 Literature Classification . ..... .... .... .... .... .... ..... .... 103
7.1 Classification . .... ..... .... .... .... .... .... ..... .... 104
7.2 Travel Times . .... ..... .... .... .... .... .... ..... .... 104
7.3 Service Times. .... ..... .... .... .... .... .... ..... .... 106
7.4 Demands. .... .... ..... .... .... .... .... .... ..... .... 106
7.5 Requests. .... .... ..... .... .... .... .... .... ..... .... 107
7.6 Analysis . .... .... ..... .... .... .... .... .... ..... .... 109
7.6.1 Time Distribution .... .... .... .... .... ..... .... 109
7.6.2 Problem... ..... .... .... .... .... .... ..... .... 110
7.6.3 Approaches ..... .... .... .... .... .... ..... .... 112
7.7 Implications .. .... ..... .... .... .... .... .... ..... .... 112
Part II Stochastic Customer Requests
8 Motivation .... .... .... ..... .... .... .... .... .... ..... .... 117
8.1 Application... .... ..... .... .... .... .... .... ..... .... 117
8.2 Replanning and Anticipation... .... .... .... .... ..... .... 119
8.3 Outline .. .... .... ..... .... .... .... .... .... ..... .... 122
9 SDVRP with Stochastic Requests... .... .... .... .... ..... .... 123
9.1 Problem Statement . ..... .... .... .... .... .... ..... .... 123
9.2 Markov Decision Process Formulation... .... .... ..... .... 125
9.3 Literature Review.. ..... .... .... .... .... .... ..... .... 127
10 Solution Algorithms. .... ..... .... .... .... .... .... ..... .... 131
10.1 Routing and Sequencing Decisions.. .... .... .... ..... .... 132
10.1.1 Subset Selection . .... .... .... .... .... ..... .... 132
10.1.2 Cheapest Insertion.... .... .... .... .... ..... .... 133
10.1.3 Improvements ... .... .... .... .... .... ..... .... 134
10.2 Myopic Policy .... ..... .... .... .... .... .... ..... .... 134
10.3 Non-reactive Implicit: Waiting Policies .. .... .... ..... .... 135
10.4 Non-reactive Explicit: Anticipatory Insertion .. .... ..... .... 135
10.5 Non-reactive Explicit: Cost Benefit.. .... .... .... ..... .... 136