Computing Egomotion with Local Loop Closures for Egocentric Videos

10.4 MB·

by  Suvam Patra

Computing Egomotion with Local Loop Closures for Egocentric Videos
SuvamPatra∗ HimanshuAggarwal∗ † HimaniArora∗ † SubhashisBanerjee
IITDelhi QualcommInc. ColumbiaUniversity IITDelhi
ChetanArora
IIITDelhi
7
1
0 Abstract
2

n Finding the camera pose is an important step in many
a egocentricvideoapplications. Ithasbeenwidelyreported
J
that, state of the art SLAM algorithms fail on egocentric
7
1 videos[1,2,3,4].Inthispaper,weproposearobustmethod 08
V] fcoerntcraicmveirdaeopso.seInestainmaetgioocne,ndteriscigvnieddeos,pethceificcaamllyerfaorvieegwos- sem semarf 0
the same scene point multiple times as the wearer’s head a
C sweepsbackandforth.Weusethisspecificmotionprofileto rf 000 sem
s. performshortloopclosuresalignedwithwearer’sfootsteps. 21 arf 0
c Foregocentricvideos,depthestimationisusuallynoisy. In 06
2
[ an important departure, we use 2D computations for rota-

1 tionaveragingwhichdonotrelyupondepthestimates. The
v two modification results in much more stable algorithm as Figure1.Wepresentanegomotionestimationtechniquetargeted
3 isevidentfromourexperimentsonvariousegocentricvideo foregocentricvideos. Thefigureontherightshowsatrajectory
4 datasetsfordifferentegocentricapplications.Theproposed computed from LSD-SLAM [5] on a sequence from [2]. The
7 algorithm resolves a long standing problem in egocentric different color segments show breaks in the trajectory computa-
4
tioneveryfew100frames. Ontheotherhandouralgorithmruns
0 visionandunlocksnewusagescenariosforfutureapplica-
successfullyevenafter12000frames(ontheleft). Theproposed
. tions.
1 methodresolvesalongstandingprobleminegocentricvision.
0
7
1 1.Introduction
: forunderstandingwearer’sactivities[8,9,10,11,12],gaze
v
Ability to record hands-free videos led to initial popu- fixation [3], temporal segmentation [3], video summariza-
i
X larityofwearablecameraslikeGoogleglass[6]andGoPro tion [13, 14, 15] etc. In many of these applications, com-
r [7]. Thesewearablecamerasaretypicallyharnessedtothe puting the pose of the camera and hence the position and
a
wearer’shead,andprovideafirstperson(wearer’s)viewof gazeofthewearerisimportant.
thesurroundings.Thecapturedvideosarethereforealsore-
For a long time, the egocentric community has strug-
ferredtoasegocentricvideos. Withtheimprovementinde-
gledwiththeproblemofegomotionestimation[1,2,3,4].
signandusabilityofegocentriccameras,suchcamerasare
Thoughvisualodometryalgorithms[5,16,17,18,19]have
nowbeingusedforlifelogging,lawenforcement,assistive
beenquitesuccessfulforhandheldorvehiclemountedcam-
visionaswellassmarthomeapplications.
eras, the relatively ‘wild’ nature of egocentric videos: un-
Owing to their unique first person perspective and
restrained camera motion, wide variety of indoor and out-
always-onnature,egocentricvideosarerichincontentusu-
doorscenesaswellasmovingobjects, makesitextremely
ally unseen from point and shoot videos. This has gener-
challengingfortheegomotionestimationalgorithms. Head
atedlotsofinterestamongthecomputervisioncommunity.
rotation of the wearer causes the scene to change rapidly
Computer vision researchers are mining egocentric videos
making any kind of feature tracking in egocentric video a
∗EqualContribution non-trivial task. Any error in tracking causes the feature
†ResearchassistantatIIITDelhiduringthisproject based visual odometry methods to lose putative matches,
andultimatelyleadtotheirfailure. Evenwhenthetracking eachsuchproblemthestate-of-the-artegomotionesti-
is not lost, sharp rotations introduce inaccuracies in pose mationhasbeenreportedtofail[1,2,3,4]. However
estimates,and,intheabsenceofanyrecoverymechanism, theproposedmethodworksflawlesslyonallofthese,
theestimationsdrifttopointsofinstabilitycreatingbreaks indicatingtherobustnessandefficacyofthetechnique.
in the trajectory. The typical motion profile of linear for-
ward velocity due to wearer walking/driving in egocentric Fig. 3givesaschematicrepresentationoftheproposed
videos, kills any standard loop closure opportunity relying approach.
onsimilarcamerapose.
2.Relatedwork
Analternateapproachistoemploythemorerobustdense
orsemi-denseSLAM1 [5,16,17]techniqueswhichutilize
Visual odometry algorithms can be classified into two
agreaterportionoftheimage. Eventhoughthesemethods
broadcategories:stereoandmonocular.Stereobasedvisual
perform better than their sparse counterparts, they are still
odometryalgorithmscanusuallygiveinformationaboutthe
unabletohandlethewilddynamicsofegocentricvideos.
absolutescaleoftheworld. However,whenthedistanceto
Itistobenotedthatshakyhandheldvideosdonothave
the scene is much larger compared to the stereo baseline,
motion profiles similar to egocentric videos. For example,
thesemethodsgivenorealadvantageandhence,monocular
evenafastmovinghand-heldcameratypicallydonothave
visual odometry algorithms are typically used. Monocular
dominant3Drotationandusuallyscansascenepointmul-
methods employ only a single camera and suffer from the
tipletimesgivingstandardloopclosureopportunities.
problemofscaleambiguity. Thescalemayberesolvedby
In this paper we present a technique for camera ego-
the integration of sensors such as IMU or with scene cali-
motionestimationtargetedespeciallyforegocentricvideos.
brationobjects.
Weusethesetocorrectposeinaccuraciesusingrotationav-
Furtherclassificationofthesealgorithmscanbedoneon
eraging [20] and Gauss-Newton re-initialization. The spe-
thebasisofthetechniquesusedforposeestimation,namely
cificcontributionsofthepaperareasfollows:
featurebasedmethods,densemethodsandhybridmethods
thatemployacombinationofboth. Feature-basedmethods
1. We present a fast and robust egomotion estimation
mainlyconsistoftwosteps[18,21].Inthefirststep,impor-
methodforegocentricvideos.
tantfeaturesareextractedfromtheimagesandmatched. In
2. We suggest a novel local loop closure technique thesecondstep,thecameraposesareestimatedusingonly
aligned with the wearer’s head motion for egocentric these sparse feature points. The reduction in the number
videos. We exploit the typical to-and-fro sweeps in ofpointsusedintheestimationprocessgreatlyreducesthe
egocentriccameramotiontodetectlocalloopclosures. computationalloadtherebyincreasingthespeed. However,
While in all previous work for handheld videos, loop alargepartofusefulinformationinthesceneisignored.
closures are typically carried out at the end, the pro- Dense methods use the entire image and not just few
posedlocalloopclosuresalignedwiththeheadsweep selected features [19]. The camera poses are estimated as
ofthewearerenablefixingofestimationerrorslocally thesetofparameterswhichminimizetheimagedifference
andefficiently. Thisnotonlymakesthealgorithmsta- over all pixels in the image. To increase the accuracy of
ble but also improves the accuracy significantly. The estimation,semi-densemethodsareusuallyadopted,which
suggestedimprovementisspecifictoegocentricvideos performphotometricerrorminimizationonlyinregionsof
only. sufficientgradient[5,17].
The work closest to ours is LSD-SLAM [5] which
3. While standard loop closure techniques typically fix does dense tracking directly on sim(3), to explicitly de-
bothrotationandtranslation,themethodologyfailsin tect scale-drifts. LSD-SLAM builds upon [17], to con-
thepresenceofnoisydepthestimatescommoninego- tinuously estimate a semi-dense inverse depth map for the
centric setting. We suggest to fix only rotation using currentframe, whichinturnisusedtotrackthemotionof
2D techniques which is both faster and more stable. thecamerausingdenseimagealignment. Givenaninverse
We show the improvement in computed trajectory by depthmap,bothmethodsestimatethecameramotionusing
fixingrotationalone. non-linear minimization in combination with a coarse-to-
finescheme,asoriginallyproposedin[16]. Theminimiza-
4. WeexperimentonmultipleproblemssuchasEgoSam- tionisdoneusingweightedGauss-Newtonoptimizationon
pling[1],Hyperlapse[2],TemporalSegmentation[3], Lie-Manifolds.Ourmethodalsousesasimilaroptimization
Activity Recognition [4] and Gaze Fixation [3]. For techniquefortheinitialcameraposeestimation.
As discussed in detail in [22], loop closures are de-
1Because of their overlap, we will be using SLAM/Odometry tech-
tected using three major approaches in literature - map-
niquesinterchangeablyinourdiscussions,thoughourfocusisonlyonthe
egomotionestimationandnotontheassociated3Dmap. to-map, image-to-image and image-to-map. L. Clemente
et al. [23] use a map-to-map approach where they find mateasξ ←ξ+∆ξ,where∆ξiscomputedas:
correspondences between common features in two sub-
maps.M.CumminsandP.Newman[24]usevisualfeatures argmin(cid:88)w [I(cid:48)(τ(x;ξ+∆ξ))−I(x)]2 =
i
for image-to-image loop-closures. Matching is performed ∆ξ
x
basedonpresenceorabsenceofthesefeaturesfromavisual (cid:88) ∂τ
vocabulary.B.Williamsetal.[25]useanimage-to-mapap- wi[I(cid:48)(τ(x;ξ))+∇I(cid:48)∂ξ∆ξ−I(x)]2 (1)
proach and find loop-closure using re-localization of cam- x (cid:124) (cid:123)(cid:122) (cid:125)
erabyestimatingtheposerelativetomapcorrespondences. FirstorderTaylorapproximation
Usingtheassumptionthataerialvideoviewsaroughlypla- (cid:104) (cid:105)
nar ground and homographies can be used to register the where,∇I(cid:48) istheimagegradient ∂I(cid:48) ∂I(cid:48) and ∂τ isthe
∂x ∂y ∂ξ
frames, Leotta et al. [26] have proposed a homography- Jacobianofthewarpingfunction.
guidedloopclosurealgorithmtoaddressthelong-termloop The pose of each image frame I(cid:48) is calculated with re-
closure problem. In this paper, we use an image-to-image specttoakeyframeI. Theposesthusobtainedareconcate-
approachforthepurposeofdetectingloopclosures. natedtoestimatetheposewithrespecttothefirstkeyframe
(assumed to be the world origin with ξ = 0). Since only
3.Background
thedepthcorrespondingtothereferenceframeIisrequired
in estimating the warping function, only a keyframe is as-
The6DOFcameraposeofanimageI(cid:48) w.r.tareference
sociatedwithadepthmap. Thekeyframesareswitchedat
image I refers to the rotation and translation of its camera
regularintervalstomaintainsufficientoverlapbetweenthe
centre. The camera pose is represented as an element of a
frames.
LieGroupbya4×4matrixg∈SE(3)whichcomprisesof
a3×3rotationmatrixR∈SO(3)anda3×1translation
3.2.DepthMapEstimation
(cid:20) (cid:21)
R t
vectort: g = . InLieAlgebra,thecameraposeis
0 1 The previous section assumed that the depth at a pixel
representedbya6elementvectorξ ∈se(3): ξ =(cid:2)w v(cid:3), Z(x) is known apriori. We jointly estimate both the cam-
where, w and v represent the rotation and translation re- eraposeandthescenedepthbycomputingstereomatches
spectively. along epipolar lines at different values of depth. The SSD
We estimate the camera pose by aligning an image I(cid:48) error is calculated at each of these pixels and the one with
withareferenceimageI whichminimizesthephotometric minimum error is chosen to be the correct match. Since
errorbetweenthetwoimageframes.Followingthenotation initially both the depth and camera pose is unknown, the
given in [16], the problem can be reduced to one of mini- algorithmisbootstrappedwitharandomdepthmaphaving
mizing the sum (cid:80) r , where, the residual r , at a pixel highweightonlyatthosepixelswheresufficientgradientis
i∈I i i
i is given as: r = I(cid:48)(τ(ξ,x ))−I(x ). Here ξ is the es- available. Afterafewkeyframepropagation,thedepthmap
i i i
timated camera pose of I(cid:48) w.r.t I and τ : x → x(cid:48) is the convergestoareasonableapproximation.
warpingfunctionmappingapixelinI toapixelinI(cid:48).
3.3.RotationAveraging
Thewarpingfunctioncanbeconstructedasfollows. Us-
ing the inverse projection function, π, the 3D world point Wheneverthegazedirectionreturnstoaneutralposition
correspondingtoapixelxisgivenas: X=π−1(x,Z(x)), following a deviation to the left or to the right, we detect
where Z(x) is the depth at a pixel x. When X is viewed localloopclosuresusingaprocedureoutlinedlaterinSec-
in the coordinate frame of the second camera, it is trans- tion4.1.Theloopclosuresprovidemultipleredundantrota-
formedbytheposeg = exp(ξ)ofI(cid:48) w.r.tI togetthe3D tionestimatesfromthecurrentframetopastframes-both
worldpointX(cid:48) givenbyX(cid:48) = T(ξ,X) = RX+t. When through pair-wise estimates, and direct rotation estimates
X(cid:48) is projected on the second camera, we get the warped with a few archived key frames. The multiple redundant
pixelcoordinatesx(cid:48) =π(X(cid:48)). rotationestimatesalongdifferentpathsallowustoaverage
outtherotationstoobtainconsistentrotationsalongallsub-
3.1.PoseEstimation
paths. WedenotebyCthesetofcamerasinawindowse-
The camera pose ξ can be computed as the one mini- lectedafterloop-closuredetection.
mizing the residual: argminξ(cid:80)xiwi(ri(ξ))2. The resid- Letusassumethattherelativerotationbetweencamerai
ual is calculated over all pixels xi having valid depth esti- andcameraj bedenotedbyRij andtheabsoluterotations
mates. The weight wi at a pixel indicates the confidence be denoted by {R1,··· ,RN} (where N is the number of
measure of the estimated depth. The objective function is cameras in the current window). Then the following rela-
non-linear (since τ is non-linear in ξ) and can be solved tionshipshouldhold:
usingGauss-Newtonoptimization. Weassumethatthecur-
rent estimate of ξ is known and iteratively refine the esti- R R−1 =R (2)
j i ij
Camera Pose & Depth  Short Loop Closure Detection Camera Pose & Depth Map 
Map Estimation on Sliding Window Refinement
Rotation Refinement
New Image
KL divergence on  Shortlisted 
KFs’ KFs Rotation Averaging (RA)
New KF ? 
image  (having 
(based on rapid pose change 
histograms similar 
detection)
histograms) Refined rotations + stable 
Yes No saved depth maps 
+ initial translations
•Establish  •Track new 
Filter loop closure KFs 
relationship with  frame on 
•Use view‐vector difference 
old KF current KF using  Translation & Depth Map Refinement
Gauss‐Newton  between initial and loop 
•Transfer and  on SE(3) using 4  closing KFs Gauss‐Newton on 
refine depth map  pyramid levels
highest pyramid level
on new KF
Initialisewith: 
•Refine depth 
map and pose  No •Refined rotations 
(Gotonext window)
of current KF  •Stable depth map 
Loop closure detected? •Initial translations
Yes
Initial camera poses and depth  Save depth map of stable KF & calculate  Refined depth map + camera poses 
map on current sliding window extra poses on redundant paths for RA for current window
Gotonext window
Figure2.Ourframework.(KFheredenoteskeyframes)
Weestimatetheabsoluterotations{R ,··· ,R }bymini- poseestimation. Inegocentricsetting,wherethemotionof
1 N
mizingthesumoferrorsbetweentheestimatedpairwisero- thewearerislinearforward,ausermaynotrevisitapartic-
tationsR andtherelativerotationscalculatedasR R−1 ularscenepoint,whichmakesgloballoopclosureimpossi-
ij j i
fromequation2as: ble. Furthermore,giventheusualnoisymeasurementsdue
{R ,··· ,R }= argmin (cid:88) d2(cid:0)R R−1,R (cid:1) towildnatureofegocentricvideos,thecomputedtrajecto-
1 N j i ij riestendtodriftquicklyunlessfixedbyloopclosures. We
{R1,···,RN}(i,j)∈C note that, a wearer’s head typically scans the scene left to
(3)
right and back during the course of natural walking. The
where
1 camera looks at same scene multiple times, thus forming
d(R1,R2)= √2||log(R2R−11)||F (4) a series of short local visual loop closures. We take ad-
vantageofthisphenomenontoimprovetheaccuracyofthe
which is the intrinsic bi-variate distance measure defined
estimatedcameraposes.
on the manifold of 3D rotations or the Special Orthogonal
We maintain an archive of the past keyframes, which
groupSO(3)and||.|| istheFrobeniusnorm. Weusethe
F is continuously updated whenever a new keyframe is ini-
robustfunctionandthetechniquedescribedin[20]tosolve
tialized. An incoming frame I(cid:48) is not only mapped to
theoptimizationproblemdefinedinEquation3.
the current keyframe I but also to the previous keyframes
4.ProposedAlgorithm It−1,It−2··· for establishing redundant paths. Initially,
when a new frame is received, an attempt is made to find
Our overall framework is depicted in Figure 2. In what amatchwithanarchivedkeyframe. Amatchissaidtoex-
followswedescribesomeofthekeysteps. ist when the KL Divergence [27], computed between the
histogramsoftwoqueryimages,issmallandtheestimated
4.1.Shortlocalloopclosures
camera view vectors point to the same direction. This ac-
For a typical hand held video, global loop closure is an tuallyputsaconstraintonthecurrentframethatitislook-
importantsteptofixtheaccumulatederrorsoverindividual ing in the same direction or view as of the keyframe un-
(a) (b) (c) (d)
Figure3.Ourapproachtocorrectionofposesusingshortloopclosuresandrotationaveraging. (a)showsatypicaltrajectorywhereeach
cameraisconnectedtothenext,andduetolargerotationtwocameras(indicatedinred)arewronglyestimated.Thedottedarrowsrepresent
intermediateframes(b)onlykeyframestakepartinloopclosures(c)showsshortloopclosuredetectionusingKLdivergenceandextra
matches(dottededges)arefoundtoclosetheloops(d)showsthecorrectedtrajectory(green)afterrunningrotationaveragingandone
iterationofGauss-Newtonforcorrectingcamerapositions.Imagesarebestviewedincolour.
incomingframes. Giventhetrackingchallengesandnoisy
3Destimatesfromanegocentricvideo,themainsourceof
error in egocentric camera pose estimation is the reliance
onthesenoisy3Destimates. Wedoarotationstabilization
using 2D, image based, pairwise rotation estimates, which
isnoteffectedbysuchnoisy3destimates.
4.3.Gauss-Newtonre-initialization
Although rotation averaging gives robust estimates of
camera rotations, it does not alter the camera translations.
However,thenewrotationestimatesarelikelytochangethe
Figure 4. Effect of rotation averaging and Gauss-Newton re-
cameracenterestimates. Thenewcameracentersarethen
initializationisshownontheGeorgiaTech.SocialInteractiondata
re-estimated by running a second level of Gauss-Newton
set[28].Trajectoryinblueisbeforerefinementwithredovalsde-
optimizationusingthenewlyaveragedrotationsandprevi-
pictingbreaksinthetrajectoryduetolargerotations(somesample
ously calculated translations as initial estimates. A single
imagesarealsoshown),whichgetscorrectedafterrefinementby
iterationatthehighestresolutionoftheimage-pyramiden-
ourmethodinpink. Thetrajectoryisdepictedforasequenceof
500framesonwhichwedetectedmultiplebatchesofshortlocal sures that the rotation estimates remain nearly unchanged
loopclosureswithinatimeframeof5-10seconds.Imagesarebest andonlyminorreadjustmentsarebroughtaboutinthetrans-
viewedincolour. lations, thereby keeping the final pose estimates restricted
in the local neighborhood/minima. This can be achieved
at minimal computational cost. Moreover, this approach
der consideration. This constitutes a local loop closure.
is more suitable than simply fixing rotation and updating
In our experiments, we select a key frame for every 10
thetranslationestimateswithout3Drefinement,whichmay
frames. To keep the loop closures local, only the neigh-
maketheestimatesincompatibleanderroneous.
bouring keyframes are searched up to a fixed time interval
Fig.4showsaresultfromtheproposeddepictingadvan-
T. Additionally the loop closures are detected on sliding
tageofshortloopclosurefollowedbyrotationaveraging.
windowsforimposingasmoothnessconstraintoverthetra-
jectory.
5.Experiments
4.2.Rotationaveragingbasedposerefinement
Robust and accurate camera pose estimation can effec-
Short local loop closures give rise to a number of re- tively be used in numerous egocentric applications. How-
dundant paths between the keyframes. These additional ever,state-of-the-artvisualSLAMsolutionssuchas[5]fail
constraintsarethenusedtorefinetheinitialposeestimate. tofindaccuratetrajectoriesforegocentricvideos. Thishas
Sharprotationsoveralongerperiodcausetherotationesti- led to various researchers adapting alternate, sub-optimal
matestodrift. Thisdriftiscorrectedbyrotationaveraging approachesinegocentricvideobasedproblems[1,3,2,4].
(seesection3.3formoredetails).Rotationaveragingisper- Inthissection,wefocusonseveraloftheseproblems. We
formedinsmallsizedwindowstoensureenoughflexibility run the proposed method on the datasets associated with
forerrorstobeaveragedout. Thiscontinuouscycleofpose eachoftheseproblems.However,insteadofbypassingego-
estimationfollowedbyrotationaveragingoverashortloop motionestimationashasbeendoneintheoriginalsolutions
closure ensures that consistent and accurate estimations of to these problems, we use the egomotion computed from
the prior camera pose are used to initialize the next set of ouralgorithmforthetasksathand. Weshowthatforallthe
Figure5.Comparisonoftrajectoriesobtainedforegocentricvideos. Ineachoftheseimagesthetrajectoriesobtainedbyourmethodare
markedinblue,whilethoseattainedbyLSD-SLAMaremarkedinred. Duetothelackofanystandardizeddatasetswithgroundtruth
posesforegocentricvideos,theaccuracyofthetrajectoriesisdemonstratedbyoverlayingthemonGPSdata(pink). Itshouldbenoted
thattheLSD-SLAMtrajectoriesshownherearetheonesobtainedjustbeforethealgorithmcrashed. Thefrequentcrossedtrajectoriesin
LSD-SLAMresultsisduetonoisy3Destimatesmaintainedbythealgorithmwhichoftenleadstofalsematches
problems that we tested our approach on, the usage of our
methodresultsineithersimilarorimprovedperformance.
We have implemented our version of LSD-SLAM in
C++, including short loop closures and the other heuris-
ticsmentionedinthelastsection. Alltheexperimentshave
beencarriedoutonaregulardesktopwithCorei72.3GHz
processor (containing 4 cores) and 16 GB RAM, running Figure6.Weshowtheegomotioncomputedusingourtechnique
Ubuntu 14.04. Unoptimized implementation of the pro- onavideousedin[4,3].Theleftandrightfiguresshowthecom-
posedalgorithmsrunsat2to3framespersecondonsucha putedXandZtranslationsrespectively. Theredcurveintheplot
showstheframesclassifiedasstationaryandthebluecurveindi-
machine. Wewillreleasethesourcecodeofourimplemen-
catestransitframes. Blackcurveindicatesunusedframesinours
tationaftertheacceptanceofthepaper.
aswellastheoriginalpaper
Theproposedalgorithmrequiresintrinsicparametersof
the camera for egomotion estimation. For the sequences
capturedbyususingGoProHero3+camera,wehavecal-
ibratedthecameraandusedtheestimatedparameters. For
thesequencestakenfromothersources,wedonothavethe
informationaboutthemakeandtheversionofthecameras
used.Thereforeforallthesevideos,weusedthesamesetof
calibrationvaluesasestimatedbyusforourGoProHero3+.
Figure7.Weuseouralgorithmtocomputeegomotioninavideo
Theapproximatevaluesappeartoworkreasonablywellin
from[4]todetect‘static’and‘stair-climbing’activities. Leftand
ourexperiments, indicatingtherobustnessoftheproposed
right images show X and Y translation components of the com-
algorithmagainstcameracalibrationanddistortionparam-
putedegomotionrespectively.Theredcurveindicateframesclas-
eters. We would like to mention that we also tried auto-
sified as stationary, blue indicate stair-climbing and black show
calibration. However,owingtonoisyandinaccuratedepth framesnotusedintheoriginalaswellasourclassification.
estimates,thecalibrationestimationneverconverges. Most
oftheSLAMtechniquesthatuseself-calibration(like[29])
do so in controlled environments. In the absence of exact ableinegocentricvideosduetoto-and-froheadmovement.
calibration, we have used standard parameters for GoPro Alsomostofthemrelyonthenoisy3Destimatesforloop
camerasasmentionedearlier. closures. Wewouldliketonotethatboththecontributions
Intheegocentricvideos, rotationsformthemajorcom- (loopclosureandrotationaveragingusing2Destimates)are
ponent of motion because of quick head movements (in essentialfortherobustcameraposeestimation. Inourex-
comparison to typical slow forward translations). Rota- periments, LSD SLAM with loop closures over short win-
tion estimation accuracy is therefore a major requirement dowsalsofailsduetotheusageofnoisy3Destimates.
insuchegomotionestimationproblems. Weexperimented Sincewedonothaveegocentricvideoswithgroundtruth
with rotation computations in both quaternion and Lie- poses(asdescribedearlier,shakyhandheldisnotequivalent
algebraic frameworks. We found the Lie-algebra based to egocentric), we have relied more on macro level accu-
computationstobemoresuitableascomparedtothequater- racy cues such as overlaying the computed paths on maps
nion approach which often led to instabilities. The global (see Fig. 5). We also synchronise the trajectories with the
loopclosuretechniquesusedincurrent stateoftheart, in- videos to manually verify if the significant events such as
cludingLSD,ORBandDTSLAM,donotexplicitlymake wearerturningarecapturedcorrectly. Itmaybenotedthat
use of the large number of redundant paths that are avail- the observed instability of the LSD SLAM on the videos
showninFig. 5cannotbeattributedtocalibrationparame-
tersasthesewerethesequencescapturedbyusandexactly
computedcalibrationparametershavebeenusedforthisex-
periment.
5.1.TemporalSegmentation[3]
Poleg et al. [3] have suggested a technique for tempo-
rally segmenting egocentric videos by labeling each frame
as one of seven activities. The set of activities chosen by
themformsapartitionimplyingthatatanypointoftime,a
wearermustbeinoneofthesevenactivitystates. Manyof
theseactivitiessuchaswalking,sittingetc.couldhavebeen
classified by simple egomotion computation. However,
theyhavereportedthefailureofposecomputationontheir
videosandhavesuggestedopticflowfeaturesasanalterna-
Figure8.Comparingproposedapproachwithnaive10×fastfor-
tive. Totesttheefficacyofouralgorithm,wepickedaran-
warding and Egosampling [1] on a publicly available video [2].
dom video sequence from their dataset, Huji Yair 8 part1.
The first row shows output from uniform sampling. The second
LSD-SLAM failsonthissequenceinconsonancewiththe
andthirdrowsshowoutputsfromEgoSamplingandproposedap-
observation by Poleg et al. However our algorithm works
proach. Focusing on the location of tree and the pedestrian re-
flawlessly. We classify frames into stationary and transit vealsthatbothEgoSamplingaswellasproposedapproachachieve
by computing inter-frame X and Z translations2 and accu- equivalent stabilization which is much better compared to naive
mulatingthemoverwindowsof10frames. ASVMclassi- uniformsampling.
fieristhentrainedoverthesefeatures. Weobtainanaccu-
racyof97%forthetwoclassclassificationproblemagainst
the reported accuracy of 98% with the original flow based
method. Figure6showstheresult.
5.2.ActivityClassification[4]
In an another paper of theirs, Poleg et al. [4] have ex-
Figure9.Wearer’sgazefixationiseasytodetectinanegocentric
tended the optical flow based technique presented in their
videobylookingattheconstancyofcameralookatpoint. How-
originalpaper[3]. Here,insteadoftemporalsegmentation, ever,Polegetal. reportedfailureofegomotioncomputationand
theyshowhowtotrainacompactCNNforclassifyinglong suggestedaflowbasedtechnique. Weusethecomputedegomo-
termactivitiesofthewearerbasedonthesuggestedsparse tionfromourtechniqueontheirvideosandsuccessfullydetectthe
opticalflow.Whileopticalflow,canbecomputedreliably,it gazefixationinstances. Thefiguresshowsomefixationsdetected
isanambiguouscuefortheconsideredegocentricactivities. byourmethod.
Cameraegomotionisamuchbetterindicator(ifpossibleto
compute reliably) and can improve the quality of predic-
tion. However,asintheiroriginalwork,Polegetal. report
thefailureofSLAMtechniques. Toshowtherobustnessof
ouralgorithm,weusedouralgorithmtocomputeegomotion
on one of their videos, “2013 American Lung Association
Fight For Air stair climb”. We then used the computed X Figure 10. We tested the proposed method on long shaky se-
quencesgiveninHyperlapse[2].Thetrajectoryshownisforvideo
and Y translation components of egomotion to distinguish
sequencegl02.mp4inHyperlapsedataset.
betweentwoclasses,stationaryvsstairclimbing.Similarto
theapproachdescribedinsection5.1,weaccumulatethese
componentsoverwindowsof10framesthereaftertraining for solving range of problems in variety of video capture
aSVMclassifieroverit. WeachievedanF1-scoreof0.99 settings. The selection of test sequence/classes is only in-
for both the classes whereas [4] achieved an F1-score of dicativeandrandom.
0.74and0.61forstairclimbingandstaticrespectively. Fig-
5.3.Egosampling[1]
ure7showstheresults. Itmaybenotedthatthepurposeof
theexperimentsistoshowtheusefulnessofourtechnique
Egocentric videos tend to be long and shaky making
2Inthe3Dworldcoordinates, variationsindepth, heightandleft& themhardtowatch. Fastforwardingisanaturalsolutionto
rightdirectionsaremarkedonZ,YandXaxisrespectively. quicklybrowsesuchvideos. However,fastforwardbyuni-
(a) (b) (c) (d)
Figure11.Ouralgorithmfailsintheabovecases. In(a)thewearerispartofmultipleenvironmentsand(b)suffersfromextremeblur. In
(c)thesceneisformedofnonLambertiansurfacescausinginaccuraciesintheposeestimates.(d)hasnegligibleillumination
formsamplingmayaccentuatetheshakealreadypresentin 5.5.Hyperlapse[2]
egocentricvideos. EgoSamplingbyPolegetal. isanadap-
Hyperlapsegivesatechniquetosmoothlyfast-forwarda
tive frame sampling technique which models frame sam-
longhand-heldoregocentricvideowithrapidcameramo-
pling and video stabilization as a joint optimization prob-
tion[2]. Thealgorithmusesastructurefrommotiontech-
lem. The problem is posed as one of finding a shortest
nique [30] to compute camera trajectory as well as the 3D
path in a graph where each node represents a frame in the
map. SincetheSfMalgorithmdonotscalewellthevideos
video and the edge between frames t and k represents the
weredividedinbatchesof1400frames.Wetestedouralgo-
perceived stability if frame k is placed immediately after
rithmaswellasLSD-SLAM onavideofromtheirdataset.
frametintheoutput. Amongstmanycosts,oneofthecosts
Figure 10 shows the result. While LSD-SLAM fails, our
suggested by Poleg et al. is the ‘shakiness’ cost which bi-
methodworksevenonlongsequencesof10,000frames.
ases the selection in favor of forward or similarly looking
frames. TheyhavereportedfailureofSLAMandhaveused 5.6.FailureCases
focusofexpansioncalculatedfromopticalflowforthepur-
Ourexperimentsindicatethattheproposedalgorithmis
pose.WeconsiderthesameframeworkasEgoSampling[1]
much more robust compared to state of the art algorithms
butuseourcomputedegomotiontocomputetheshakiness
like LSD-SLAM . However, we could identify multiple
cost. Figure8showsthatusingourmethodonecanachieve
cases where the work still needs improvement. Figure 11
similarimprovementinstabilizationasEgoSamplingcom-
showssomefailurecases.
paredtouniformsampling,butinamoreprincipledmanner.
6.Conclusion
5.4.Gazefixation[3]
Egomotionisanimportantinformationinmanyegocen-
Insteadofcontinuousrotationsofthecameraduetothe
tric applications. However, its use has been restricted un-
constant head movement of the wearer, egocentric videos
til now because of failure of state-of-the-art visual SLAM
sometime exhibit anomalies when this natural head rota-
techniquesonsuchvideosinthewild. Weexploitthetyp-
tion temporarily ceases. The anomalies usually indicate
ical motion profile in an egocentric video to perform local
wearer’sgazefixation. Polegetal. havegivenaflowbased
loopclosuresbasedonrealignmentofwearer’sgazedirec-
techniquetodetectsuchgazefixations[3].Weuseourtech-
tions. This allows us to fix the camera pose locally and
nique to find the egomotion and estimate the camera view
efficiently. Notingthatcameramotionisdominant3Drota-
vector,correspondingtothedirectionofthewearer’sgaze.
tion,weusetwosteploopclosuredoingrotationaveraging
Gaze fixation can then be detected by looking at the con-
first followed by translation fixation. Our experiments in-
stancy of the camera look at point. As in the original pa-
dicate that exploiting these properties of egocentric videos
per, we use a moving window of 5 seconds to detect gaze
leadstoarobustcameraposeestimationalgorithm. Ourex-
fixation. Our hypothesis is based on identifying a point of
periments on many egocentric applications where egomo-
intersection of the view vectors for the frames within this
tion has been reported to fail indicates that the proposed
movingwindow. Weusetheproposedapproachtoidentify
algorithm can be used successfully. This resolves a long
gazefixationpointsintwovideosfromthe“HUJIEgoSeg
standingprobleminegocentricvisionunlockingtheuseof
dataset(Huji Chetan 1andHuji Yair 2)”. Thegazeanno-
egomotioninmanyegocentricapplications.
tationwasnotavailableinthedatasetandwemanuallyan-
notated the frames. From the 17 gaze fixation points, our
References
methodsuccessfullydetected16fixationswith3falseposi-
tives,whichiscomparabletowhathasbeenreportedinthe [1] Y. Poleg, T. Halperin, C. Arora, and S. Peleg,
originalwork. “EgoSampling: Fast-forward and stereo for egocen-
tric videos,” in Proceedings of the IEEE Conference [13] Z.LuandK.Grauman, “Story-drivensummarization
onComputerVisionandPatternRecognition(CVPR), for egocentric video,” in Proceedings of the IEEE
2015,pp.4768–4776. Conference on Computer Vision and Pattern Recog-
nition(CVPR),2013,pp.2714–2721.
[2] J. Kopf, M. F. Cohen, and R. Szeliski, “First-person
hyper-lapse videos,” ACM Transactions on Graphics [14] K. M. Kitani, T. Okabe, Y. Sato, and A. Sugimoto,
(TOG),vol.33,no.4,pp.78:1–78:10,2014. “Fastunsupervisedego-actionlearningforfirst-person
sportsvideos,”inProceedingsoftheIEEEConference
[3] Y.Poleg,C.Arora,andS.Peleg,“Temporalsegmenta-
onComputerVisionandPatternRecognition(CVPR),
tionofegocentricvideos,”inProceedingsoftheIEEE
2011,pp.3241–3248.
ConferenceonComputerVisionandPatternRecogni-
tion(CVPR),2014,pp.2537–2544. [15] Y. J. Lee, J. Ghosh, and K. Grauman, “Discover-
ingimportantpeopleandobjectsforegocentricvideo
[4] Y. Poleg, A. Ephrat, S. Peleg, and C. Arora, “Com- summarization,” in Proceedings of the IEEE Confer-
pact CNN for indexing egocentric videos,” in Pro- ence on Computer Vision and Pattern Recognition
ceedings of the IEEE Winter Conference on Applica- (CVPR),2012,pp.1346–1353.
tionsofComputerVision(WACV),2016.
[16] C. Kerl, J. Sturm, and D. Cremers, “Robust odome-
[5] J. Engel, T. Schops, and D. Cremers, “LSD-SLAM: tryestimationforRGB-Dcameras.”inProceedingsof
Large-Scale Direct Monocular SLAM,” in Proceed- the IEEE International Conference on Robotics and
ingsoftheEuropeanConferenceonComputerVision Automation(ICRA),2013,pp.3748–3754.
(ECCV),2014,pp.834–849.
[17] J. Engel, J. Sturm, and D. Cremers, “Semi-dense vi-
[6] Google, “Glass,” https://www.google.com/glass/start. sual odometry for a monocular camera,” in Proceed-
[Online]. Available: https://www.google.com/glass/ ings of the IEEE International Conference on Com-
start puterVision(ICCV),2013,pp.1449–1456.
[7] GoPro, “Hero3,” https://gopro.com/. [Online]. Avail- [18] G. Klein and D. Murray, “Parallel tracking and map-
able: https://gopro.com/ pingonacameraphone,”inProceedingsoftheIEEE
International Symposium on Mixed and Augmented
[8] T. Starner, J. Weaver, and A. Pentland, “Real-Time
Reality(ISMAR),2009,pp.83–86.
American Sign Language Recognition Using Desk
and Wearable Computer Based Video,” IEEE Trans- [19] R.A.Newcombe,S.J.Lovegrove,andA.J.Davison,
actionsonPatternAnalysisandMachineIntelligence “DTAM:Densetrackingandmappinginreal-time,”in
(PAMI),vol.20,no.12,pp.1371–1375,1998. ProceedingsoftheIEEEInternationalConferenceon
ComputerVision(ICCV),2011,pp.2320–2327.
[9] H. Pirsiavash and D. Ramanan, “Detecting activities
of daily living in first-person camera views,” in Pro- [20] A. Chatterjee and V. M. Govindu, “Efficient and ro-
ceedingsoftheIEEEConferenceonComputerVision bustlarge-scalerotationaveraging.”inProceedingsof
and Pattern Recognition (CVPR), 2012, pp. 2847– the IEEE International Conference on Computer Vi-
2854. sion(ICCV),2013,pp.521–528.
[10] A.Fathi,A.Farhadi,andJ.M.Rehg,“Understanding [21] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardo´s,
egocentric activities,” in Proceedings ofthe IEEEIn- “ORB-SLAM: a versatile and accurate monocular
ternational Conference on Computer Vision (ICCV), SLAM system,” IEEE Transactions on Robotics,
2011,pp.407–414. vol.31,no.5,pp.1147–1163,2015.
[11] S.SundaramandW.Mayol-Cuevas,“Highlevelactiv- [22] B. Williams, M. Cummins, J. Neira, P. Newman,
ityrecognitionusinglowresolutionwearablevision,” I. Reid, and J. Tardo´s, “A comparison of loop clos-
inProceedingsoftheIEEEComputerVisionandPat- ing techniques in monocular SLAM,” Robotics and
ternRecognitionWorkshops(CVPRW),2009,pp.25– AutonomousSystems,vol.57,no.12,pp.1188–1197,
32. 2009.
[12] M. S. Ryoo and L. Matthies, “First-person activity [23] L. Clemente, A. Davison, I. Reid, J. Neira, and J. D.
recognition: Whataretheydoingtome?” inProceed- Tardo´s,“Mappinglargeloopswithasinglehand-held
ingsoftheIEEEConferenceonComputerVisionand camera,”inProceedingsofRobotics:ScienceandSys-
PatternRecognition(CVPR),2013,pp.2730–2737. temsConference,2007.
[24] M. Cummins and P. Newman, “FAB-MAP: Proba-
bilisticLocalizationandMappingintheSpaceofAp-
pearance,” The International Journal of Robotics Re-
search,vol.27,no.6,pp.647–665,2008.
[25] B. Williams, M. Cummins, J. Neira, P. Newman,
I. Reid, and J. D. Tardos, “An image-to-map loop
closingmethodformonocularSLAM,”inProceeding
of the International Conference on Intelligent Robots
andSystems,2008,pp.2053–2059.
[26] M. J. Leotta, E. Smith, M. Dawkins, and P. Tunison,
“Opensourcestructure-from-motionforaerialvideo,”
in 2016 IEEE Winter Conference on Applications of
ComputerVision(WACV),2016,pp.1–9.
[27] P.J.Moreno,P.Ho,andN.Vasconcelos,“AKullback-
Leibler divergence based kernel for svm classifica-
tion in multimedia applications.” in Proceedings of
Advances in Neural Information Processing Systems
(NIPS),2004,pp.1385–1392.
[28] A.Fathi,J.K.Hodgins,andJ.M.Rehg,“Socialinter-
actions: Afirst-personperspective,”inProceedingsof
theIEEEConferenceonComputerVisionandPattern
Recognition(CVPR),2012,pp.1226–1233.
[29] H. Ha, S. Im, J. Park, H.-G. Jeon, and I. S. Kweon,
“High-quality depth from uncalibrated small motion
clip,” in Proceedings of IEEE Conference on Com-
puterVisionandPatternRecognition(CVPR),2016.
[30] C. Wu, “Towards linear-time incremental structure
from motion,” in Proceedings of the International
Conferenceon3DVision(3DV),2013,pp.127–134.