Advanced Driver Intention Inference: Theory and Design

249 Pages·2020·8.764 MB·English

by  Yang Xing, Chen Lv, Dongpu Cao

Advanced Driver
Intention Inference
Theory and Design
YANG XING, PHD
Research Fellow
School of Mechanical and Aerospace Engineering
Nanyang Technological University
Singapore
CHEN LV
Assistant Professor
School of Mechanical and Aerospace Engineering
Nanyang Technological University
Singapore
DONGPU CAO
Canada Research Chair in Driver Cognition and Automated Driving
Department of Mechanical and Mechatronics Engineering
University of Waterloo
Canada
]
Elsevier
Radarweg29,POBox211,1000AEAmsterdam,Netherlands
TheBoulevard,LangfordLane,Kidlington,OxfordOX51GB,UnitedKingdom
50HampshireStreet,5thFloor,Cambridge,MA02139,UnitedStates
Copyright(cid:1)2020ElsevierInc.Allrightsreserved.
Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,electronicor
mechanical,includingphotocopying,recording,oranyinformationstorageandretrievalsystem,without
permissioninwritingfromthepublisher.Detailsonhowtoseekpermission,furtherinformationaboutthe
Publisher’spermissionspoliciesandourarrangementswithorganizationssuchastheCopyrightClearance
CenterandtheCopyrightLicensingAgency,canbefoundatourwebsite:www.elsevier.com/permissions.
ThisbookandtheindividualcontributionscontainedinitareprotectedundercopyrightbythePublisher(other
thanasmaybenotedherein).
Notices
Knowledgeandbestpracticeinthisfieldareconstantlychanging.Asnewresearchandexperiencebroadenour
understanding,changesinresearchmethods,professionalpractices,ormedicaltreatmentmaybecome
necessary.
Practitionersandresearchersmustalwaysrelyontheirownexperienceandknowledgeinevaluatingandusing
anyinformation,methods,compounds,orexperimentsdescribedherein.Inusingsuchinformationor
methodstheyshouldbemindfuloftheirownsafetyandthesafetyofothers,includingpartiesforwhomthey
haveaprofessionalresponsibility.
Tothefullestextentofthelaw,neitherthePublishernortheauthors,contributors,oreditors,assumeany
liabilityforanyinjuryand/ordamagetopersonsorpropertyasamatterofproductsliability,negligence
orotherwise,orfromanyuseoroperationofanymethods,products,instructions,orideascontainedinthe
materialherein.
LibraryofCongressCataloging-in-PublicationData
AcatalogrecordforthisbookisavailablefromtheLibraryofCongress
BritishLibraryCataloguing-in-PublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
ISBN:978-0-12-819113-2
ForinformationonallElsevierpublicationsvisitourwebsiteat
https://www.elsevier.com/books-and-journals
Publisher:MatthewDeans
AcquisitionsEditor:CarrieBolger
EditorialProjectManager:GabrielaD.Capille
ProductionProjectManager:SreejithViswanathan
Coverdesigner:AlanStudholme
TypesetbyTNQTechnologies
List of Abbreviations
GP Gaussian Process
ABS AntilockBrakingSystem HMM Hidden MarkovModel
ACC AdaptiveCruiseControl HHMM Hierarchical HiddenMarkovModel
ACP ArtificialSociety,Computational HOG Histogramof OrientedGradients
Experiments,andParallelExecution HRI Human-RobotInteraction
ACT-R AdaptiveControlofThought-Rational HT Hough Transform
ADAS AdvancedDriverAssistanceSystem IDDM Intention-driven DynamicModel
A/D Analog/Digital IMM InteractiveMultiple Model
AIOHMM AutoregressiveInput-Output HMM IMU Inertial MeasurementUnit
ANN ArtificialNeural Network IOHMM Input-Output HiddenMarkovModel
AVM AroundViewMonitoring IPM Inverse PerspectiveMapping
BCI Brain-ComputerInterface JTSM JointTime-Series Modelling
BF BayesianFilter KRR KernelRidgeRegression
BN BayesianNetwork LADAR Laser DetectionandRanging
CAN ControllerAreaNetwork LANA Lane FindinginAnother Domain
CBN CausalBayesian Network LCII Lane ChangeIntentionInference
CDI ComprehensiveDecision Index LDA Lane DepartureAvoidance
CHMM ContinuousHidden MarkovModel LDW Lane DepartureWarning
CLNF Conditional LocalNeuralFields LIDAR Light DetectionandRanging
CLR ConstantLearningRate LKA Lane KeepingAssistance
CPS Cyber-PhysicalSpace LOO Leave-One-Out
CPSS Cyber-Physical-Social Space LSV Lane SamplingandVoting
CNN ConvolutionalNeural Network LSTM LongShort-TermMemory
CSI ChannelStateInformation MIC Maximal Information Coefficient
DBSCAN Density-based Spatial Clustering of MHMM ModifiedHidden MarkovModel
ApplicationwithNoise MLR Multisteps LearningRate
DBN DynamicBayesianNetwork NGSIM Next-Generation Simulation
DII DriverIntentionInference NFIR NonlinearFiniteImpulseResponse
DWT Discrete WaveletTransform NN Neural Networks
ED EdgeDistribution OLS Orthogonal LeastSquares
EEG Electroencephalograph OOB Out-of-Bag
ECG Electrocardiogram PCA Principal ComponentAnalysis
EMG Electromyography PRM Revolutions PerMinute
EOG Electrooculography RANSAC RandomSampleConsensus
ERRC ErrorReduction Ratio-Causality RF RandomForest
EV Electric Vehicle RGB Red-Green-Blue
FFNN Feedforward NeuralNetwork RGB-D Red-Green-Blue Depth
FIR FieldImpedance Equalizer RMSE RootMeanSquareError
FPR False-Positive Rate RNN Recurrent NeuralNetwork
GA GeneticAlgorithm ROC ReceiverOperating Characteristic
GAN GenerativeAdversarialNetworks ROI RegionofInterest
GMM GaussianMixture Model RVM RelevanceVectorMachine
GNSS GlobalNavigation SatelliteSystem SA Situation Awareness
GOLD GenericObstacleandLaneDetection SAE Society ofAutomobileEngineers
GPS GlobalPositioning System SBL Sparse BayesianLearning
v
vi LISTOFABBREVIATIONS
SF SteerableFilter TS TimeSliced
SCR SkinConductance Response TPR True-Positive Rate
SWA SideWarning Assistance TTI TimetoIntersection
SVM SupportVectorMachine TTC TimetoCollision
SVR SupportVectorRegression TTCCP TimetoCriticalProbability
TDV Traffic-Driver-Vehicle V2V Vehicle-to-Vehicle
THW TimeHeadway WHO World HealthOrganization
Abstract
Longitudinal and lateral control of the vehicle on the inference.Chapters3and4containthetechniquesfor
highwayarehighlyinteractivetasksforhumandrivers. traffic context perception that focus on sensor integra-
The intelligent vehicles and the advanced driver- tion, sensor fusion, and road perception. A review of
assistancesystems(ADAS)needtohaveproperaware- lanedetectiontechniquesanditsintegrationwithapar-
nessofthetrafficcontextaswellasthedrivertomake alleldrivingframeworkisproposed.Next,anovelinte-
an appropriate assistant to the driver. The ADAS also grated lane detection system is designed. Chapters 5
needtounderstandthepotentialdriverintentcorrectly and6discussthedriverbehaviorissues,whichprovide
since it shares the control authority with the human the driver behavior monitoring system for normal
driver.Thisbookprovidesresearchonthedriverinten- driving and secondary tasks detection. The first part is
tion inference, particular focus on the two typical based on the conventional feature selection method,
vehicle control maneuvers, namely, lane change ma- while the second part introduces an end-to-end deep
neuver and braking maneuver on highway scenarios. learning framework. Understanding the driver status
A primarymotivation of thisbook is topropose algo- and behaviors is the preliminary task for driver inten-
rithms that can accurately model the driver intention tioninference.Thedesignandanalysisofdriverbraking
inferenceprocess.Driver’sintentionwillberecognized and lane change intention inference systems are pro-
based on the machine learning methods due to its posed in Chapters 7 and 8. Machine-learning models
goodreasoningandgeneralizingcharacteristics.Sensors and time-series deep-learning models are used to esti-
inthevehicleareusedtocapturecontexttrafficinforma- mate the longitudinal and lateral driver intention.
tion, vehicle dynamics, and driver behavior Finally, discussions and conclusions are made in
information. Chapter9.
This book is organized in sections and chapters,
whereeachchapteriscorrespondingtoaspecifictopic.
KEYWORDS
Chapter1introducesthemotivation,humanintention
background, and general methodological framework ADAS,Computervision,Driverbehaviors,Driverinten-
usedinthisbook.Chapter2includestheliteraturesur- tion inference, Intelligent vehicles,Automated driving,
veyandthestate-of-the-artanalysisofdriverintention Machinelearning.
vii
CHAPTER 1
Introduction
Worldwidetrafficdepartmentshavereportedthatmore ADAS techniques such as lane departure avoidance,
than 1.2 million traffic-related injuries happen each lane keeping assistance, and side warning assistance
year. Among these traffic accidents, more than 80% can help the driver to make the right decision and
werecausedbyhumanerrors[1].TheWorldHealthOr- reducetheworkload.
ganization (WHO) reported that traffic accidents each It is predicted that the shipment of ADAS in the
year cost around V518 billion worldwide and on future has great potential and can generate a huge
average, 1%e2% of the world gross domestic product amountofcommercialbenefitbasedonmanyautomo-
[2,3]. In the past, in-vehicle passive safety systems tivemarketanalyzers.OneexampleisshowninFig.1.2
suchasairbagsandseatbeltshaveplayedasignificant accordingtothepredictionofGrandViewResearch,Inc.
role in the protection of drivers and passengers. These ADAS products will show a significant increase in the
technologies have saved millions of lives. However, next5years.ThereforetheutilizationofADASproducts
they are not designed to prevent accidents from willbecomemoreaccessibletothepublic,althoughthis
happening but just try to minimize the injuries after canbringaseriesofproblems.Forexample,thefinan-
the accidents happen[4].Therefore recent efforts have cialcostwillincrease.Also,asmostoftheautomotive
been devoted to the development of safer and intelli- companies are developing their ADAS products, safety
gentsystemstowardthepreventionofaccidents.These insurance and product quality can be different from
systems are known as the Advanced Driver Assistance each other. Once the drivers are getting familiar with
Systems(ADAS). theseproducts,theywillheavilyrelyonthesesystems.
ADAS is a series of fast-developing techniques that AveryfamousexampleistheTeslacarcrashesthatare
aredesignedforimprovingdriversafetyandincreasing caused bytheirautopilotADASproducts, asshownin
thedrivingexperience[5].ADASreliesonamultimodal Fig.1.3.TheautopilotproductsofTeslaareoneofthe
sensor fusion technique to integrate multiple sensors mostsuccessfulcommercialdriverassistanceandsemi-
suchaslightdetectionandranging(lidar),radar,cam- automated driving assistance systeminthe world. The
era,andGPSintoaholisticsystem.Thesensors’work- product is set of intelligent computing, perception,
ing ranges are shown in Fig. 1.1. Most of the current and control units, which can significantly increase
FIG.1.1 Distribution of Advanced DriverAssistanceSystems in an advancedvehicle(deepscale.ai/adas.
html).Lidar,lightdetectionandranging.
AdvancedDriverIntentionInference.https://doi.org/10.1016/B978-0-12-819113-2.00001-4
Copyright©2020ElsevierInc.Allrightsreserved. 1
2 AdvancedDriverIntentionInference
drivingsafetyissues.However,evensuchasmartsystem factor, the driver. Vehicles are working in a three-
can be reported for car crashes worldwide. One of the dimensional environment with continuous driver-
mostcommonreasonsforacrashisthedriverovertrust- vehicle-roadinteractions.Driversarethemostessential
ingtheautopilotwhenthesystemisactivated,whichis partofthissystem,whocontrolthevehiclebasedonthe
aprobleminthefuture. surroundingtrafficcontextperception.Thereforeallow-
ThereasonswhyADAScannotbe100%trustedare ing ADAS to understand driver behaviors and follow
multifold. Currently, most of the reasons are due to driver’s intention is of importance to driver safety,
immaturetechniques;however,adeeperreasonisthat vehicledrivability,andtrafficefficiency.
thedriverandtheautomationlackmutualunderstand- Humandriverintentioninferenceisanidealwayto
ing.TheinputsofcurrentADASaremainlybasedonly allowADAStoobtaintheabilityofreasoning.Therea-
onthevehicledynamicstatesandtrafficcontextinfor- sons for developing driver intention inference tech-
mation. Most of the systems ignore the most critical nique are multifold: first of all, the most important
FIGURE1.2 AdvancedDriverAssistanceSystemsmarketprediction(GrandViewResearch,Inc.https://
www.grandviewresearch.com).ACC,adaptivecruisecontrol;AEB,automaticemergencybraking;AFL,
adaptivefrontlight;BSD,blindspotdetection;LDWS,lanedeparturewarningsystems;TPMS,tirepressure
monitoringsystem.
FIGURE1.3 ATeslacarhascrashedintoaparkedpolicecarinCalifornia,USA.(https://www.bbc.com/
news/technology-44300952).
CHAPTER1 Introduction 3
and significant motivation is to improve driver and can act as the guidance to design an automatic
vehicle safety. Accordingly, two different driving sce- decision-makingsystem.
narios require inferring the driver’s intention. The first Moreover,intermsofthelevel3automatedvehicle
one is to better assess the risk in the future based on (according to the SAE International standard on the
the driver’s interesting region. The second one is to classification of automated vehicles), accurate driver
avoidmakingdecisionsthatareoppositetothedriver’s intentionpredictionenablesasmootherandsafertran-
intent. For the first case, there is evidence that a large sition between the driver and the autonomous
number of accidents are caused by human error or controller[11,12].Whenthelevel3automatedvehicles
misbehavior, including cognitive (47%), judgment are operating in an autonomous condition, all the
(40%), and operational errors (13%) [6]. Therefore drivingmaneuversarehandledbythevehicle.However,
monitoring and correcting driver intention and once the vehicle cannot deal with an emergent situa-
behavior seem to be crucial in the effectiveness of a tion, it has to give the driving authority to the driver.
future ADAS. Meanwhile, the increasing use of in- This process is known as disengagement [13]. In such
vehicledevicesandinformationsystemstendtodistract a case, the vehicle can assess the takeover ability of
thedriverfromthedrivingtask.Forthedesignoffuture thedriveraccordingtothecontinuouslydetectedinten-
ADAS, it is therefore beneficial to integrate intended tion.Ifthedriverisfocusingonthedrivingtaskatthat
driverbehaviorsfromtheearlydesignstages[7,8]. moment andhas an explicit intention, the vehicle can
ADASusuallyautomaticallyinterveneinthevehicle warn the driver and pass the driving authority to the
dynamics and share the control authority. To ensure driver.Thiswillmakesurethetransitionbetweendriver
cooperation, it is crucial that ADAS is aware of driver andcontrollerisassmoothaspossible.However,ifthe
intentionanddoesnotoperateagainstthedriver’swill- driver is believed to be unable to handle the driving
ingness.Forexample,incomplextrafficconditionssuch task, the autonomous driving unit should help the
asintersectionandroundabout,itiscrucialnottointer- driver gain situation awareness as soon as possible
ruptthedrivermakingdecisions,especiallynottosus- andtakeemergencyactionimmediately.
pend the driver with misleading instructions. This Anotheressentialreasontodevelopthedriverinten-
makes it reasonable or even necessary for ADAS to tioninferencesystemisitwillcontributetothedevel-
have the ability to accurately understand the driver’s opment of automated vehicles. As shown in Fig. 1.4,
driving intention. On the other hand, intention infor- each level of understanding about the driver can be
mation enables for more accurate prediction of future mapped into the corresponding intelligent level of an
trajectories, which would be beneficial to risk assess- autonomousvehicle.Comprehensiveresearchoneach
ment systems [9,10]. Driver intention inference will layerwillcontributetothedevelopmentoftherelative
benefit the construction of the driver model, which layer in the autonomous vehicle. Driver intention
FIGURE1.4 Theevolutionfromthecurrentvehicletofutureautonomousvehicle.
4 AdvancedDriverIntentionInference
recognition is a relative higher-level understanding of driver behavior analysis is of importance to infer
humandriversandrelatedtothedecision-makinglayer driverintention.
of autonomous vehicles. Modeling driver intention 4. Driver lane change intention inference algorithms:
mechanismsiscriticaltotheconstructionofautomated Based on the specific traffic context and driver be-
decision-making algorithms. Human drivers are the haviors, the next task is to infer driver intention
teacher of automated drivers. The automated drivers properly. The algorithms for intention inference
can learn when and how to make the decision based should have the ability to capture the long-term
on the driver’s intention knowledge. Once the human dependency between the temporal sequences.
driverbecomesthepassengerintheautomateddriving Moreover,theintentioninferencealgorithmsshould
vehicles, it will be easier for the passengers to accept predicttheintentionasearlyaspossible.
that the automated driving systems is such systems The driver lane change intention platform requires
that can remember the driving habit of the passenger. the integration of software and hardware systems.
Therefore a good study about when and how drivers Driverintentioninferencehastotakethetrafficcontext,
generate their intentions will benefit the design of the driver behaviors, or dynamic vehicle information into
decision-makingmoduleforintelligentvehicles.Based consideration,whichwillfusemultimodalsignalsand
on such a design method, the vehicles will be more mining the long-term dependency between different
similar to human drivers, which will make it much signals based on machine learning methods. In terms
easier for humans to accept these highly intelligent of the hardware system, the sensors, included in this
vehicles. book, contain RGB and RGB-D cameras and vehicle
As discussed earlier, teaching ADAS to understand navigation system. Besides, all the sensors are tested
driver intention is essential as well as challenging to and mounted on a real vehicle in this case to collect
enhance the safety of the driver-vehicle-traffic close- naturalistic data. Specifically, the traffic context such
loop system. To focus more, this book will target two aslanepositionsandfrontvehiclepositionwillbepro-
ofthemostpopulardrivingscenariosinbothlongitudi- cessedwithimage-processingmethods.Onewebcam-
nalandlateraldirections,namely,thebrakingandlane eraismountedinsidethecabinet.Thedriverbehavior
changemaneuvers.Forexample,duringastandardlane dynamics will be evaluated within a steady and dy-
changemaneuver,thedriverisexpectedtoperformase- namic vehicle. The RGB-D camera (Microsoft Kinect
ries of behaviors (e.g., mirror checking and turn the V2.0)willbeusedforthesteadyvehicle,whileanother
steeringwheel).Thedriver’slanechangeintentioncan web camerawill beusedto recordthe driverbehavior
be inferred in an early stage by recognizing driver be- during the highway driving task. These signals are
haviors and traffic context situations. A driver lane recorded with one laptop for further processing and
change intention system facing next-generation ADAS analyzing. The algorithms used in this project are
isdevelopedinthisstudy.Basedonthis,fourmainob- mainly focused on machine learning methods, which
jectivesaredetermined: include supervised learning, unsupervised learning,
1. Driver intention process analysis: To predict driver anddeeplearningmodels.Allthealgorithmsarewrit-
lane change intention, it is vital to understand the teninMATLABandCþþ.
human intention mechanism, such as how Thedriver’sintentioninferencetaskdescribedinthis
theintentionisgeneratedandwhatisthestimuliof bookreliesonmachinelearningalgorithmstoworkin
the intention. The nature behind driver intention realtime.Thereasonsforusingmachinelearningcanbe
isthefirstquestionthatneedstobeanswered. multifold. First, the real-time traffic context and driver
2. Trafficcontextperception:Thedriverisinthemiddle behaviordatacanbehighdimensionalandoflargevol-
ofthetraffic-driver-vehicleloop.Trafficcontextisthe ume, and very few mathematic models can deal with
inputtothedriverperceptionsystem,whichmakes such data. However, machine learning algorithms are
itactasthestimuliofthedriver’sintention.There- useful for high-dimensional multimodal data process-
fore understanding the current traffic situation will ing.Second,theutilizationofamachinelearningalgo-
benefittheintentioninferencesystem. rithm enables learning the long-term dependency
3. Understanding driver behaviors: Driver behaviors, between driver behaviors and traffic context, which
such as mirror checking, are the most important significantly increases the inference accuracy for the
clues before the driver makes a lane change. The lane change intention. Finally, it is hard to find the
driverhastoperformaseriesofchecking actionto intention generation and inference pattern based on
haveafullunderstandingofthesurroundingcontext observationandmodeling.Themachinelearningalgo-
beforehe/shedecidestochangethelane.Therefore rithms provide an efficient way to learn knowledge