Deep Spatiotemporal Models for Robust Proprioceptive Terrain Classification

20 Pages·2017·47.26 MB·English

.
Deep Spatiotemporal Models for
.
.
Robust Proprioceptive Terrain
.
Classification .
.
.
Abhinav Valada and Wolfram Burgard
Abstract
Terrain classification is a critical component of any autonomous mobile robot system operating in unknown real-
world environments. Over the years, several proprioceptive terrain classification techniques have been introduced
to increase robustness or act as a fallback for traditional vision based approaches. However, they lack widespread
adaptationduetovariousfactorsthatincludeinadequateaccuracy,robustnessandslowrun-times.Inthispaper,we
use vehicle-terrain interaction sounds as a proprioceptive modality and propose a deep Long-Short Term Memory
(LSTM) based recurrent model that captures both the spatial and temporal dynamics of such a problem, thereby
overcomingthesepastlimitations.OurmodelconsistsofanewConvolutionNeuralNetwork(CNN)architecturethat
learns deep spatial features, complemented with LSTM units that learn complex temporal dynamics. Experiments
ontwoextensivedatasetscollectedwithdifferentmicrophonesonvariousindoorandoutdoorterrainsdemonstrate
state-of-the-artperformancecomparedtoexistingtechniques.Weadditionallyevaluatetheperformanceinadverse
acoustic conditions with high ambient noise and propose a noise-aware training scheme that enables learning of
moregeneralizablemodelsthatareessentialforrobustreal-worlddeployments.
Keywords
Convolutionalneuralnetworks,Recurrentneuralnetworks,Long-shorttermmemorynetworks,Terrainclassification,
Acoustics,Temporalclassification
1. Introduction vehicles body (Brooks and Iagnemma, 2005), vehicle-
terrain interaction sounds (Christe and Kottege, 2016;
As the transition of robots from mere lab equipment to Libby and Stentz, 2012; Ozkul et al., 2013; Valada
autonomous machines that tackle complex problems in etal.,2015)androughnessestimationusingaccelerometers
unknown environments occurs, the perceptual challenges (Eriksson et al., 2008; Weiss et al., 2006). Each of these
that they face increases exponentially. Mobile robots in approacheshavetheirownbenefitsanddrawbacks:optical
particular should have the capability to distinguish the sensors perform remarkably well in the presence of good
terrainthattheytraverseonandaccordinglyoptimizetheir illumination but are drastically affected by changes in
navigation strategy. Failing to adapt the trafficability can lighting conditions. Classification with RGB images is
lead to disastrous situations. For instance, a robot that can usuallyachievedusingcolorandtexturefeaturesextracted
travelwithacertainmaximumspeedoncarpet,cannotuse fromthescene(Sungetal.,2010).Theuseoftexture-based
the same speed to traverse on sand, if not it would lead to image descriptors such as local ternary patterns have also
entrenchment.Moreover,thepredominatelyadoptedvision beenexplored.Insuchapproaches,featuresextractedusing
based approaches have a high probability of failure due to
the similar visual appearance of these two terrains. Often,
there are also situations where terrains are covered with
DepartmentofComputerScience,UniversityofFreiburg,Germany
leaves or water which can be very challenging for vision- http://deepterrain.cs.uni-freiburg.de
basedclassifiers.
Correspondingauthor:
Over the years, this has motivated researchers to Abhinav Valada, Albert-Ludwigs-Universita¨t Freiburg, Technische
explore alternate modalities such as lidars (Suger et al., Fakulta¨t, Autonome Intelligente Systeme, Georges-Ko¨hler-Allee 079,
79110FreiburgimBreisgau,Germany.
2015; Thrun et al., 2006), vibrations induced on the
Email:[email protected]
2 ValadaandBurgard
local ternary patterns are used on sequences of mutated
images and classified using Recurrent Neural Network
(RNN) configurations (Otte et al., 2015). Lidars on the
other hand, have extensively been used for traversability
analysis. However, they are not suitable for fine-grained
terrain classification where two or more terrains may have
thesamedegreeofcoarseness.Nonetheless,theyhavebeen
shown to benefit from being able to learn from partially
labelleddataandusingsemi-supervisedlearningtechniques
(Santamaria-Navarro et al., 2015; Suger et al., 2015).
Commonly, features used include statistics on remission
values,roughnessandslope.Astherearebenefitsinusing Figure1. ThePioneerP3-DXplatformthatweuseinour
experiments,showingtheshotgunmicrophonewiththe
both images and lidar data, approaches have also been
shockmountmountedclosetothewheel.
explored to learn classifiers from both modalities. Here,
featuresfromimagesincludecolor,texture,andgeometric
features and from lidar data typically surface normals,
curvature,groundheight,pointfeaturehistograms,linearity
hardware setups. Sounds from such interactions have
and planarity are used (Namin et al., 2014; Posner et al.,
distinct audio signatures and can be even used for
2008).CNN-basedapproachesforbothnear-rangeandfar-
fine-grained terrain classification such as distinguishing
range terrain classification using stereo and RGB images
between grass and mowed-grass (Valada et al., 2015).
have been demonstrated during the Defense Advanced
Unlike speech or natural domestic sounds, vehicle-terrain
Research Projects Agency (DARPA) Learning Applied to
interactionsoundsareveryunstructuredinnatureasseveral
Ground Robots (LAGR) program (Hadsell et al., 2009;
dynamic environmental factors contribute to the signal.
Muller et al., 2013). These approaches achieved state of
Our previous work demonstrated that features learned
the art performance by combining both supervised and
using Deep Convolutional Neural Networks (DCNNs)
unsupervisedlearningusingdeephierarchicalnetworks.In
significantly outperforms classification using shallow,
contrast,acousticsbasedapproachesperformwellinallthe
manually-engineered feature sets. Our network combines
above scenarios but are easily affected by environmental
three temporal pooling methods to achieve the time-series
noise. Research using acoustics and other proprioceptive
representationlearning.
techniquesarediscussedinSection2.
In this work, we extend our previous approach by
The benefit of exploring these alternate modalities is
incorporating a deep LSTM recurrent framework in
that the disturbances that affect optical or active sensors
order to further exploit the temporal dynamics in our
do not affect proprioceptive sensors such as microphones,
problem. Recurrent neural models form compositional
hence enabling us to use them in a complementary
representationsinthetimedomain,similartoconvolutional
classifier or fuse them with other modalities to increase
modelsformingcompositionalrepresentationsinthespatial
robustness. These alternate approaches have failed to gain
domain.LSTMbasedrecurrentneuralnetworkmodelshave
popularity beyond research. This can be attributed to
recentlyachievedimpressiveresultsforsequentiallearning
the following: (1) manually designing feature sets that
taskssuchaslanguagetranslation(Sutskeveretal.,2014),
perform well in every real-world condition is tedious and
speech recognition (Graves and Jaitly, 2014) and image
impractical; (2) existing techniques have slow run times
captioning (Donahue et al., 2015). The structure of our
makingthemunusableforreal-timeapplications;(3)most
proposednetworkleveragesspatio-temporalinformationin
techniquesrequirespecifichardwaresetupsthataredifficult
eachclipandintransitioningclipsaswell,wherewedefine
to replicate; and (4) approaches often lack reliability in
a clip as the time window that we take for classification.
real-worldscenarios.
Weshowthatourend-to-endtrainedrecurrentneuralmodel
In this paper, we propose a novel multiclass terrain learns improved feature representations compared to our
classification approach that overcomes these impediments DCNNonbothourdatasets.
by using only the sound from vehicle-terrain interaction DCNNs learn highly discriminative features from
to classify both indoor and outdoor terrains. We gathered training data, however, a caveat of this attribute is that
two long-scale vehicle-terrain interaction datasets: one by the training data cannot encompass every possible real-
equipping our mobile robot with a shotgun microphone world scenario. For acoustics-based terrain classification,
and the other using a mobile phone microphone, as this primarily relates to the ability of the model to
shown in Figure 1. The rationale behind this was adapt to ambient noise in the environment. Noise filtering
to test the generalizability of the model to different techniques can be applied if the training data can
ValadaandBurgard 3
include every possible noise disturbance, but this is 2. Related Work
highly impractical. Therefore, the model has to learn the
distribution of common noises so that it can generalize Terrain classification using proprioceptive modalities has
effectively to different situations. Noise sensitivity is not been explored in the same depth as vision based
strongly correlated to the capability to operate in different approaches, yet there is a sizable amount of work in
real-worldenvironments.Moreover,therearetwodifferent this area. The most researched proprioceptive terrain
typesofnoisesourcesinourapplication;ambientnoiseand classificationtechniqueisusingaccelerometerdata(Ojeda
thenoisefromthemotorsoftherobot.Asthemicrophone et al., 2006; Trautmann and Ray, 2011; Weiss et al.,
ismountedclosetothetiresoftherobot,itisinevitablethat 2006). This is achieved by extracting features such as
thecapturedsignalsalsoincludethenoisefromthemotors powerspectraldensity,discretefouriertransformandother
of the robot. Therefore this is already accounted for in the statistical measures from the vibrations induced on the
dataset. In order to make our model generalize effectively vehicles body. Such approaches demonstrate a substantial
toreal-worldenvironmentswithvaryingtypesandamounts amount of false positives for finer terrain classes such
of ambient noise, we present a noise-aware training as asphalt and carpet. However, accuracies as high as
scheme. Our training scheme randomly injects ambient 91.8% has been reported for a seven class problem using
noisesignalsfromtheDiverseEnvironmentsMultichannel Support Vector Machines (SVMs) (Weiss et al., 2006). In
Acoustic Noise Database (DEMAND) (Thiemann et al., a similar work, accelerometer data from a mobile sensor
2013) at different Signal-to-Noise (SNR) ratios. We then networksystemwasusedtodetectpotholesandotherroad
demonstrate that our recurrent model trained using our anomalies(Erikssonetal.,2008).Handengineeredfeatures
noise-aware scheme shows improved robustness in real- wereusedandthesystemachievedanaverageaccuracyof
worldconditions. 90%inreal-worldexperiments.
Thereisabodyofworkinterrainclassificationtailored
Specifically, the following are the main contributions of
to legged robots. Unlike inwheeled robots, proprioceptive
thispaper:
terrain classification can enable safe foothold placement
(1) Weproposeanewdeepspatio-temporalarchitecture which is critical to ensure the stability of such legged
for learning complex dynamics in proprioceptive systems. In one of the initial works (Hoepflinger et al.,
signals. 2010), the authors extracted features from ground contact
(2) We optimize various hyperparameters of the model forces and joint motor current measurements to train a
andshowtheirinfluenceontheperformance. multiclass AdaBoost classifier. Although they did not test
(3) We demonstrate that our approach is hardware the efficacy of their approach on real-world terrains, they
independent and the performance of our model on demonstrated the classifiers capability to identify different
datafromaninexpensivehardwareisonparwithdata coarsenessandcurvaturesofsurfaces,asafirststeptowards
fromahigh-qualitydevice. real-world proprioceptive terrain classification. In another
(4) We extensively evaluate the utility of our proposed approach (Best et al., 2013), the authors demonstrate the
Global Statistical Pooling (GSP) on various deep ability to classify four different outdoor terrains using
spatio-temporalmodelconfigurations. positionmeasurementsfromlegservosofahexapodrobot.
(5) We quantify our models performance in seven Theyextracta600-dimensionalfeaturevectorconsistingof
different environments having adverse acoustic gait-phase domain, as well as frequency domain features.
conditions. Their approach utilizes a 2.7s window of data and a SVM
(6) We introduce a noise aware training scheme that istrainedtoclassifytheterrains.
substantially increases the generalizability of the
Vibration data from contact microphones have also
modeltoreal-worldscenarios.
been successfully utilized for terrain classification. The
(7) We present thorough empirical evaluations on over
vibrations captured are similar to accelerometers than air
6hofaudiodata.
microphones that we use in this work, as they pick up
Rest of this paper is organized as follows. In only structure-borne sound and minimal environmental
Section 2, we discuss the work that has been previously noise.Contactmicrophoneshavepreviouslybeenmounted
done using acoustic and other proprioceptive sensors on analog rover’s wheel frame to capture the induced
for terrain classification. We then detail our recurrent vibrations and classify the terrain (Brooks and Iagnemma,
LSTM approach, the network architecture and training 2005). They extract log-scale power spectral density
in Section 3. In Section 4, we provide a brief overview features and train a pairwise classifier. For a three class
our data collection methodology. Finally we present problem, they achieve an average accuracy of 74% on
our experimental evaluation in Section 5, followed by a wheel-terrain testbed and 85.3% on a rover. They
conclusionsanddiscussioninSection6. also demonstrate a self-supervised classification approach
4 ValadaandBurgard
whereavisualclassifieristrainedusinglabelsprovidedby they do not address the robustness of their models to
avibration-basedclassifier(BrooksandIagnemma,2007). ambient noise, which is one of the most critical properties
The use of vehicle-terrain interaction sounds for terrain for real-world adaptation. In the following section, we
classification has been the most sparsely explored among describe our DCNN architecture, the methodology using
all the proprioceptive modalities. Typically, combinations which we incorporate temporal recurrence into our model
of state-of-the-art handcrafted audio features are used andournoise-awaretrainingscheme.
with traditional machine learning algorithms. Recently,
a multiclass sound-based terrain classification system
3. Technical Approach
for a mobile robot was presented, that utilizes features
including spectral coefficients, moments and various other One of the main objectives of our work is to develop an
spectral and temporal characteristics (Libby and Stentz, end-to-endtrainablerecurrentmodeltailoredforclassifying
2012). Their SVM based classifier achieves an average unstructured vehicle-terrain interaction sounds, with as
accuracy of 78% over three terrain classes and three minimal preprocessing as possible. Even though learning
hazardous vehicle-terrain interaction classes. They also directly from raw waveforms has been demonstrated
showed that smoothing over larger temporal window of for speech processing, recent work has shown that
about 2s yields an improved accuracy of 92%. In another the performance of such approaches is lower and the
approach (Ojeda et al., 2006), a suite of sensors including computational cost is higher, when compared to learning
microphones, accelerometers, motor current and voltage fromsimpletransformationssuchasspectrograms(Graves
sensors,infrared,ultrasonicsandencoderswereusedalong and Jaitly, 2014; Khunarsal et al., 2013). Therefore we
withafeedforwardneuralnetworkclassifier.Howevertheir choose to train our models on top of this minimal
microphone based approach, only achieved an average preprocessing.Thissignalprocessingcanalsobecomputed
accuracyof60.3%forafiveclassproblem.Theyconcluded in the input layer of the network. The features used for
thattheoverallperformancewaspoorandsuchanapproach classification are learned by the network from this input
waspromisingonlyforclassessuchasgrass. representation.Figure2depictsourpreprocessingpipeline.
Microphones have also been used for terrain classifi-
cation with legged robots. In one such approach, terrain 3.1. Spectrogram Transformation and
is classified from the steps taken by a hexapod robot Augmentation
(Ozkuletal.,2013).Theyusezero-crossingrate,frequency
Unlike speech decoding or handwriting recognition
bandcoefficientsanddelta-featuresalongwithafunctional
tasks that require specific alignment between the audio
trees classifier. They also experiment with different noise
and transcription sequences, our application does not
elimination techniques to remove motor/gear-head noise,
necessitate a specific target for each frame. We only need
but concluded that the performance was worse after the
toclassifyasetofframes/clipswithoutrequiringanyprior
noise elimination. Their seven class classifier achieved an
alignment in the sequences. Therefore, we use individual
average accuracy of 90%. More recently, a SVM based
clips as a new sample for classification. We first split the
system that uses statistics of spectral and band features,
raw audio signal into short clips of t and t of overlap
w ov
was used to classify terrain from the sounds produced
between the clips. We then experimentally identify the
during the locomotion of hexapod (Christe and Kottege,
shortest clip length, overlap and the number of windows
2016).Theirclassifierwastrainedonadatasetthatcontains
taken by the LSTM units that best enables our model to
5minworthofdatafromeachterraintype.Theirapproach
learnthediscriminativeinformationfromthetimevarying
uses 1s windows for classification and operates at 1Hz.
vehicle-terraininteractionsignal.
They report an average accuracy of 89.08% for seven
We extract the Short-Time Fourier Transform (STFT)
terrain classes. They also investigated the use of spectral
based spectrogram for each clip in the sequence. We first
subtraction to eliminate the servo noise. They report an
block each audio clip into M samples with 50% overlap
improvement in the sensitivity from 92.9% to 95.1% after
between the frames. Let x[n] be the recorded raw audio
spectralsubtraction.
signal with duration of N samples, f be the sampling
f s
In all of the previous works, manually handcrafted frequency,S(i,j)bethespectrogramrepresentationofthe
features were extracted after specialized preprocessing 1-D audio signal and f(k)=kf /N . By applying STFT
s f
steps. The approaches were evaluated on comparatively onthelengthM windowedframeofthesignal,weget
limited data and not in varying real-world environments.
Most importantly, they do not model the temporally N(cid:88)f−1 (cid:18) 2πk (cid:19)
discriminative information in proprioceptive data which X(i,j)= x[n]w[n−j]exp −p n ,
could potentially improve the classification performance, p=0 Nf (1)
increaserobustnessandreducefalsedetections.Inaddition, p=0,...,N −1
f
ValadaandBurgard 5
Waveform
time FFT and
y
c Normalization
n
e
u
q
e Spectrogram
r
f
Overlapping
Windows
Augmentation
Figure2. Overviewofourpreprocessingpipeline.Rawaudiowaveformofthevehicle-terraininteractionistransformedinto
itsspectrogramrepresentationandthenaseriesofaugmentationsareappliedtoforgethelearnedfeaturerepresentation
invarianttocertainsignalperturbations.
We use a Hamming window function w[n] to compensate in training of very deep networks. We experimented
for the Gibbs effect while computing STFT by smoothing with several signal augmentation strategies and created
the discontinuities at the beginning and end of the audio additional samples by applying a set of augmentations
signal,i.e., A on the audio signal in the frequency domain. Using
t
(cid:18) (cid:19) two dimensional affine transform and warping, random
n
w[n]=0.54−0.46cos 2π , offsets in time and frequency were applied to perform
M −1 (2)
shiftingofthespectrogram.Furthermore,wecreatedmore
n=0,...,M −1 samples using time stretching, modulating the tempo,
random equalization, and by varying the volume gain. We
Wethencomputethelogofthepowerspectrumas
also experimented with frequency and time normalization
Slog(i,j)=20log10(|X(i,j)|) (3) withaslidingwindowandlocalcontrastnormalization.
WechoseN as2048samples,thereforethespectrogram
f 3.2. Network Architecture
contains 1024 Fourier coefficients. Experiments from our
previous work (Valada et al., 2015) revealed that most of Recurrent Neural Network (RNN) models learn complex
thespectralenergyisconcentratedbelow512coefficients, temporal dynamics from a sequence by mapping inputs
therefore we only use the lower 512 coefficients to reduce x=(x1,...,XT), to a sequence of hidden states h=
the computational complexity and runtime. The noise and (h1,...,hT), and hidden states to an output sequence
intensity levels vary a fair amount in the entire dataset y =(y1,...,yT).Thisisachievedbyiteratingthroughthe
as we collected data in several real-world environments. followingequationsfromt=1toT,as
Factors such as environmental noise at different times of
the day, variations in weather conditions and variations
h =g(W x +W h +b ) (5)
in the servo noise at different speeds, contribute to this. t ih t hh t−1 h
Therefore, we normalize the spectrograms by dividing yt =Whoht+bo (6)
by the maximum amplitude. We then compute the mean
where g is is the hidden layer activation function such
spectrum over the entire dataset and subtract it from the
as an element-wise application of sigmoid non-linearity,
normalizedspectrogram.Wecomputethisas
W and b terms denote the weight matrices and bias
S(i,j)=S (i,j)/maxS (i,j) (4) vectors, with subscripts i,h, ando denoting input, hidden
log log
i,j andoutputrespectively.
Data augmentation has been shown to improve the As discussed extensively in previous works (Donahue
performance of several deep learning tasks and also help et al., 2015; Graves and Jaitly, 2014; Hochreiter and
6 ValadaandBurgard
conv-10 fc11 fc12 LSTM
Input conv-1
conv-4
conv-7
Global
cccp-2cccp-3 cccp-5 cccp-6 cccp-8 cccp-9 MP256 Max
class
pooling
MP Global
MP MP 512 256 L2 norm
pooling 9
128
512
MP256 GAvlogb al 4096 4096 2048
pooling
Figure3. Overviewofourterrainclassificationpipeline.Wefirsttransformtherawaudiosignaloftheterraininteractioninto
itsspectrogramrepresentationandthenfeeditintoaDCNNforfeaturelearningandclassification.TheLSTMisunrolledin
timeandMPreferstomaxpooling.
Schmidhuber, 1997), traditional RNNs often suffer from the same weight, thereby providing invariance to small
vanishing and exploding gradient problems that occur phaseshiftsinthesignal.
from propagating gradients through several layers of the DCNNs that are designed to operate on images for
RNN. The larger the length of temporal input, the harder various perception tasks, specifically preserve the spatial
it is to train the RNN. The long short-term memory information of features learned. However, our application
(LSTM) architecture (Hochreiter and Schmidhuber, 1997) does not benefit from this, as we are only interested in
was proposed as a solution to this problem and to identifying the presence or absence of features, rather
enable exploitation of long-term temporal dynamics from than localizing the features in the frame. To this end, we
a sequence. LSTMs incorporate memory units containing introduce a new Global Statistical Pooling (GSP) scheme
several gates that regulate the flow of information in and that applies different pooling mechanisms across entire
outofthecells. input feature maps and combines them to gather statistics
of features across a particular dimension. Our architecture
WefirstuseourDCNNarchitecture(Valadaetal.,2015),
utilizes this GSP by incorporating three different global
toextractfeaturesfromthespectrogramofeachclipinthe
pooling layers after cccp-9 to compute the statistics of
sequenceintime.Thespectrogramsinourtrainingsetareof
the features across time. We use an inner product layer to
theformS ={s ,...,s }withs ∈RN.Eachofthemare
1 T i combinetheoutputsofmaxpooling,L2normpoolingand
ofsizev×wandnumberofchannelsd(d=1inourcase).
averagepooling.Inourpreviouswork(Valadaetal.,2015),
Ournetwork,showninFigure3,consistsofsixconvolution
we investigated various combinations of different global
layers, six Cascaded Cross Channel Parametric Pooling
pooling layers and found that for other configurations,
(CCCP) layers (Lin et al., 2013), three Fully-Connected
the accuracy dropped over 3%. As our new proposed
(FC) layers, an LSTM layer and softmax layer. All the
architecture utilizes LSTM units to model the temporal
convolutionlayersareonedimensionalwithakernelsizeof
relationships,weexploretheutilityofthisglobalstatistics
threeandconvolvealongthetemporaldimension.Weusea
poolinginSection5.WeusetheXavierweightfiller(Glorot
fixed convolutional stride of one. CCCP layers that follow
and Bengio, 2010) to initialize the convolution and CCCP
the first, second and third convolution layers are used to
layers by drawing from a zero mean uniform distribution
enhancediscriminabilityforlocalpatcheswithinreceptive
from[−a,a]andthevarianceasafunctionofinputneurons
fields. CCCP layers effectively employ 1×1 convolutions (cid:112)
n , where a= 3/n . The effects of initializing with
overthefeaturemapsandthefilterslearnedareabetternon- in in
justaGaussianfillerarediscussedinSection5.
linear function approximator. A max-pooling layer with a
Rectified linear units (ReLUs) significantly help in
kernel of two follows the second, fourth and sixth CCCP
overcomingthevanishinggradientproblem.WeuseReLUs
layers. Max-pooling adds some invariance by only taking
f(x)=max(0,x),aftertheconvolutionlayersanddropout
the high activations from adjacent hidden units that share
regularization (Hinton et al., 2012) on the inner product
ValadaandBurgard 7
computation for minibatch using a momentum coefficient
αas0<α<1.Theupdaterulecanthenbewrittenas
Input σ Output σ
Gate Gate ∂E
Memory ∆w (t)=α∆w (t−1)−(cid:15) (14)
Cell ij ij ∂w (t)
φ × φ × ij
ht(=zt)
We found the model to be extremely sensitive to the
Input
Modulation × learningparametersused.Therefore,weusetheSpearmint
Gate
Bayesian optimization library (Snoek et al., 2012) to tune
xt σ Forget thehyperparameters.Wefirstoptimizedthelearningpolicy
ht-1 Gate andtheinitiallearningrateoverfixed,inverse,stepandpoly
policies.Weobtainedthebestperformanceusinganinitial
Figure4. DepictionofaLongShort-termMemoryCell
learning rate of λ =0.01 and the poly policy as λ =
(Graves,2013;ZarembaandSutskever,2014). 0 n
λ ×(1−N/N )c, where N is the iteration number,
0 max
N isthemaximumnumberofiterations,andcispower.
max
Wetunedthenumberofoutputsintheinnerproductlayers
andtheLSTMlayerastheywerehighlycorrelatedandhad
layers.WethenstacktheLSTMlayerafterfc12,followed
abigimpactontheperformanceofthemodel.Weincreased
byanotherinnerproductlayerandasoftmax layer.Ifx is
t the number of parameters in the inner product layers in
theinputtotheLSTMlayerattimet,theactivationscanbe
order to make the LSTM model converge. This parameter
formulatedas
tuningisfurtherdiscussedinSection5.
i =σ(W x +W h +b ) (7)
t xi t hi t−1 i
3.3. Noise Aware Training
f =σ(W x +W h +b ) (8)
t xf t hf t−1 f
The basic principle of our noise aware training scheme
o =σ(W x +W h +b ) (9)
t xo t ho t−1 o
is to randomly inject common environmental noises into
g =φ(W x +W h +b ) (10)
t xc t hc t−1 c the training data so that the signal features are learned
ct =ft(cid:12)ct−1+it(cid:12)gt (11) along with noise patterns, and hence accounted for during
h =o (cid:12)φ(c ) (12) inference.Asourhierarchicalmodelencompassesaspatial
t t t
and a temporal component, the lower spatial layers learn
where σ(x)=(1+e−x)−1 is the sigmoid nonlinearity featuresthatdescribeboththepuresignalandnoisesignal,
and φ(x)= ex−e−x =2σ(2x) is the hyperbolic tangent whereas the higher layers learn to distinguish between
ex+e−x
nonlinearity. h ∈RN is the hidden unit, g ∈RN is the them.Thetemporalcomponentenablesthemodeltolearn
t t
input modulation gate, c ∈RN is the memory cell, i ∈ theevolutionofboththesignals.Duetothespatio-temporal
t t
RN is the input gate, f ∈RN is the forget gate, and depth, the network is capable of learning heterogeneous
t
o ∈RN is the output gate. W and b are the weight signal patterns and at what stage to de-emphasize the
t
matrixandbiaswithsubscriptsi,f,h,c,andorepresenting noise while making decisions. Noise-conditioned decision
input, forget, hidden, cell and output gates respectively. (cid:12) boundariesareleanedaswetrainthenetworkwithsignals
representselement-wisemultiplication.Thehiddenstateh corrupted with noise at different Signal-to-Noise Ratios
t
models the terrain that the robot is traversing on at time t. (SNRs). The noise-aware training scheme also regularizes
Theoutputofthememorycellchangesovertimebasedon the network. It helps improve the classification of pure
thepaststatesandthecurrentstateofthecell.Therefore,the signalsastheeasilydegradedpartsofthesignalareblurred
hidden stateis formed basedon the short-termmemory of by noise which forces the network to learn the more
thepastclip.Attimestept,thepredicteddistributionP(y ) dominantfeatures,henceavoidingover-fitting.
t
canbecomputedbytakingthesoftmax overtheoutputsof During the training stage, we randomly select
thesequencefromtheLSTMunitsz ,i.e., a noise following a multinomial distribution
t
Mult(µ ,µ ,...,µ ), where n is the types of noises
1 2 n
and µ is sampled from a Dirichlet distribution as
i
exp(W z +b )
P(yt =c)= (cid:80) exp(Wzc t,zc +c b ) (13) (µ1,µ2,...,µn) ∼ Dir(α1,α2,...,αn), where αi is
zc t,c(cid:48) c set to control the base distribution of the noise types.
c(cid:48)∈C
Furthermore, the SNR of the noised sample follows a
We train the entire model end-to-end using minibatch Gaussian distribution N(µ ,σ ), where µ and
SNR SNR SNR
Stochastic Gradient Decent (SGD) with a batch size of σ are the mean and variance. We randomly select a
SNR
256. We optimize the SGD by smoothing the gradient startpointsonthenoisesignalandscaleitaccordingtoa
8 ValadaandBurgard
SNR before adding it to the pure vehicle-terrain clip. The Kottege,2016;LibbyandStentz,2012;Ozkuletal.,2013).
noise samples are concatenated if the noise clip length is Inordertoevaluatetheperformanceofourproposedmodel,
smaller than the vehicle-terrain clip length. We use the we extracted several of these traditional baseline audio
Praat framework (Boersma and Weenink, 2013) for the features. We explored the use of both time and frequency
noisecorruption. domain features. For wheeled mobile robots, previously,
We use 18 classes of noise recordings from the Diverse GinnaandShapefeatureshavebeendemonstratedtoyield
Environments Multichannel Acoustic Noise Database the best results (Libby and Stentz, 2012). Ginna features
(DEMAND).Wecategorizethenoiseintosevenclassesfor (Giannakopoulos et al., 2006) is a six dimensional feature
in-depthexperimentsasfollows: vector consisting of zero crossing rate (ZCR), short time
energy (STE), spectral centroid (SC), spectral rolloff (SR)
• White: White noise has a very wide band and it is
and spectral flux (SF). While, shape features (Wellman
one of the most common noise sources. It has a
et al., 1997) is four dimensional feature vector consisting
very similar effect to that of various physical and
of spectral centroid, standard deviations, skewness and
environmentaldisturbancesincludingwindandwater
kurtosis.
sources.
ZCR can be defined as the rate at which the signal
• Domestic:Thiscategoryincludesnoisesfromliving
changes from positive to negative. ZCR is one of the
rooms, kitchens and washing rooms. As our terrain
most traditional audio feature used for speech and music
classes also contain indoor terrains, it is critical to
classification. STE can be defined as the energy in short
trainthemodelwithcommonindoorambientnoises.
segment of the signal. It can be given as the square of the
• Nature: The nature category contains outdoor noise
amplitude of the signal. Spectral centroid also referred to
samplesfromasportsfield,arivercreekwithflowing
as the median of the power spectrum, indicates where the
waterandacitypark.
centerofmassofthespectrumis.Theycanbecalculatedas
• Office: This category contains recordings from an
office with people working on computers, a hallway
N−1
1 (cid:88)
where peoplepass byoccasionally.and frominterior ZCR= |sgn[y(m)]−sgn[y(i−1)]|
2N
publicspaces.
m=0
• Public:Thepubliccategorycontainsrecordingsfrom
interior public spaces such a bus station, a cafeteria (15)
withpeopleandatauniversityrestaurant. (cid:40)
1,x(n)≥0
• Street:Thestreetcategorycontainsnoiserecordings where, sgn[x(m)]= (16)
−1,x(n)<0
from outdoor inner-city public roads and pedestrian
traffic. It contains a busy traffic intersection, a town N−1
(cid:88)
squarewithtouristsandacafeatapublicspace. STE= X((m).w(n−m))2 (17)
• Transportation:Thiscategorycontainsrecordingsof m=0
vehiclenoisessuchascars,subwaysandbuses. N(cid:80)−1
f(m)X(m)
As vehicle-terrain sounds are already unstructured in SC= m=0 (18)
N−1
nature, corrupting it with various heterogeneous noises (cid:80) X(m)
mightleadtothemodelnotconverging.Wefirstinvestigate m=0
theeffectofthesedisturbancesonmodelstrainedforaudio-
where, X(n) is a sample of the power spectrum at bin
basedterrainclassification.Wethentrainournetworkwith
n, f(n) is the center frequency of the bin n, w(n)
afixedSNRonalltheaboveambientnoisecategoriesand
is the windowing function, and N is the size of the
compareitwithtrainingoncorruptedsampleswithvarying
window. Spectral flux indicates the change in the power
SNRs.Theweightsandbiasesforthenoise-awaretraining
spectrum between successive windows. Spectral rolloff is
areinitializedbycopyingthemfromourpuretrainedmodel
thefrequencybelowthe95thpercentileofthepowerinthe
asdescribedintheprevioussection.Weusealearningrate
spectrum.Theycanbegivenas
1/10th of the initial rate used for training the network.
Resultsfromtheseexperimentsarediscussedinsection5.4.
(cid:115)
N−1
(cid:80) (X(m,n)−X(m,n−1))2
3.4. Baseline Feature Extraction
m=1
SF= (19)
There is a wide spectrum of audio features developed for N −1
various audio recognition and detection tasks. In previous K fmax
(cid:88) (cid:88)
works, researchers have explored the utilization of these SR=K,where X(m)=0.95 X(m) (20)
features for sound-based terrain classification (Christe and m=0 m=0
ValadaandBurgard 9
(a)Asphalt (b)MowedGrass (c)GrassMed-High
(d)Paving (e)Cobble (f)Offroad
(g)Wood (h)Linoleum (i)Carpet
Figure5. Terrainclassesandanexampleoftheircorrespondingspectrogramfora2,000msclip(colorizedspectrograms
areonlyshownforbettervisualization,spectrogramsusedfortrainingareonechannel).
where fmax isthe maximumfrequency atbin m. Spectral MFCCsareoneofthemostwidelyusedfeaturesforaudio
skewness measures the symmetry of the distribution of classification and Chroma features are strongly related to
the spectral magnitude around the mean. Spectral kurtosis the harmonic progression of the audio signal. We use
measures the similarity of the distribution of the spectral a combination of 12-bin MFCCs and 12-bin Chroma
magnitude to that of a Gaussian distribution. They can be features for our comparisons. Timbral features containing
definedby means and variances of spectral centroid, spectral rolloff,
spectral flux, zero crossing rate, low energy, and first 5
1 N(cid:88)−1(cid:18)X(m)−µ(cid:19)3 MFCCs, have shown impressive results for music genre
SS= (21)
S σ classification(TzanetakisandCook,2002).Wealsousethis
m=0
19dimensionalfeaturevectorforourbaselinecomparison.
SK= 1 N(cid:88)−1(cid:18)X(m)−µ(cid:19)4−3 (22) Forourfinalbaselinecomparison,weuseacombinationof
S σ cepstral features containing 13-bin MFCCs, Line Spectral
m=0
Pair (LSP) and Linear Prediction Cepstral Coefficients
N−1
1 (cid:88)
whereµ= X(m) (23) (LPCCs)(Verma,2008).
S
m=0
(cid:118)
(cid:117) N−1 4. Data Collection and Labelling
(cid:117)1 (cid:88)
σ =(cid:116) (X(m)−µ)2 (24)
S To the best of our knowledge, audio-based terrain
m=0
classification using mobile robots has been focused on
where, S is half the size of the window S =N/2, µ outdoor terrains. Whereas, there has been some work on
is the mean across the windows, and σ is the standard both indoor and outdoor audio-based terrain classification
deviation across the windows. A combination of Mel- using legged robots. Our objective is to create a model
frequency Cepstral Coefficients (MFCCs) and Chroma thatcanclassifyawidevarietyofbothindoorandoutdoor
features have been demonstrated to yield state of the art terrains. Some indoor and outdoor terrains can have very
performance for music classification tasks (Ellis, 2007). similar visual features (Figure 5(a), 5(h), 5(i)) and hence
10 ValadaandBurgard
Table1. Classificationaccuracyofseveralbaselinefeatureextractionapproachesonourdatasetforacliplengthof300ms.Our
recurrentneuralmodelperforms8.83%betterthanthebestbaselineapproach.
Features SVMLinear SVMRBF k-NN
Ginna 44.87±0.70 37.51±0.74 57.26±0.60
Spectral 84.48±0.36 78.65±0.45 76.02±0.43
Ginna&Shape 85.50±0.34 80.37±0.55 78.17±0.37
MFCC&Chroma 88.95±0.21 88.55±0.20 88.43±0.15
Trimbral 89.07±0.12 86.74±0.25 84.82±0.54
Cepstral 89.93±0.21 78.93±0.62 88.63±0.06
DCNN(ours) 97.52±0.016
DCNNwithLSTM(ours) 98.76±0.009
pose a challenge for classifying using the vision based noisy disturbances, we refer to uncommon temporary
counterparts. We collected over 15h of vehicle-terrain environmental disturbances such as a nearby car or train
interactiondatafromsixdifferentoutdoorterrainsandthree passing.Thedataisthensplittotrainandtestsets,ensuring
differentindoorterrains. that the classes have approximately the same number of
Our approach is platform independent, as the network samplestopreventanybiastowardsaspecificclass.Itwas
learns features from the given training data. For our also ensured that the training and validation sets do not
experiments, we use the P3-DX mobile robot platform to containclipsfromthesamelocation.
collect data as it has a small footprint and feeble motor
noise. We equipped the P3-DX with rugged wheels that 5. Experimental Results
enabled us to collect data both indoors and structured as
wellasunstructuredoutdoorsenvironments.Oneofthebig WeusetheCaffe(Jiaetal.,2014)deeplearningframework
challenges for audio classification is to deal with ambient for our implementations and the LSTM described in
and environmental noise. Training the network with data (Zaremba and Sutskever, 2014) but a version of the
that includes interference from nearby sound sources can implementationfasterthan(Donahueetal.,2015).Allour
drastically affect the performance. In order to prevent models are trained end-to-end and the experiments were
such biases in training data, we equip the robot with a runonasystemcontainingaNVIDIATITANXGPUwith
shotgun microphone to record vehicle-terrain interaction cuDNNacceleration.Theresultsfromourexperimentsare
sounds. A shotgun microphone has a supercardioid polar describedinthefollowingsections.
pickup pattern, which helps in rejecting off-axis ambient
sounds.Specifically,wechosetheRhodeVideoMicProand 5.1. Baseline Comparisons
mounteditnearthewheelofthemobilerobotasshownin
In this section, we empirically evaluate our base DCNN
Figure 1. The microphone has an integrated shock mount
model and our LSTM model with several baseline audio
thatpreventsthepickupofanyunwantedvibrationscaused
features discussed in Section 4.3. For all the baseline
duringthetraversal.
experiments that we present in Table 1, we choose a
We collected data in several different locations to have fixed clip length of 300ms. We compare with SVMs
enough variability and to enable our model to generalize and k-Nearest Neighbours (kNNs) classifiers, which have
effectively to real-world environments. Therefore even shown the best performance in the work by (Libby and
signalsinthesameclasshavevaryingspectralandtemporal Stentz, 2012). SVMs perform well in high dimensional
characteristics.Therobotsspeedwasvariedfrom0.1ms−1 spaces and kNNs perform well when there are irregular
to 1.0ms−1 during the data collection runs. The data was decision boundaries. As a preprocessing step for the
recorded in the lossless 16-bit WAV format at 44.1kHz baseline features, we normalize the data to have zero
to avoid any recording artifacts. There was no software mean. We use the one-vs-all voting scheme with SVMs
level boost added during the recordings as we found it to to handle multiple classes and experimented with Linear
amplify the ambient noise substantially, instead we used andRadialBasisFunction(RBF)asdecisionfunctions.We
a 20dB hardware level boost. The data was manually useinversedistanceweightingforkNNsandoptimizedthe
labelledusingthelivetagsandtimestampsmadeduringthe hyperparametersforbothclassifiersbyagrid-searchusing
recordings.Awaveformanalyzertoolwasusedtocropout cross-validation. For the baseline classifiers, we used the
significant noise disturbances that had substantially higher implementations in scikit-learn and LibSVM. The results
amplitudes than the vehicle-terrain interaction signals. By fromthiscomparisonareshowedinTable1.