Visual Analysis of Articulated Motion - Robotics Research Group

171 Pages·2006·8.63 MB·English

DPHIL THESIS
V A
ISUAL NALYSIS OF
A M
RTICULATED OTION
P A. T
HILIP RESADERN
October12,2006
ROBOTICS RESEARCH GROUP
DEPARTMENT OF ENGINEERING SCIENCE
UNIVERSITY OF OXFORD
ThisthesisissubmittedtotheDepartmentofEngineeringScience,
UniversityofOxford,forthedegreeofDoctorofPhilosophy. Thisthesis
isentirelymyownworkand,exceptwhereotherwiseindicated,describes
myownresearch.
ForMumandDad
PhilTresadern DoctorofPhilosophy
ExeterCollege HillaryTerm,2002
V A
ISUAL NALYSIS OF
A M
RTICULATED OTION
Abstract
The ability of machines to recognise and interpret human action and gesture from
standardvideofootagehaswide-rangingapplicationsforcontrol,analysisandsecurity.
However,inmanyscenariostheuseofcommercialmotioncapturesystemsisundesir-
able or infeasible (e.g. intelligent surveillance). In particular, commercial systems are
restricted by their dependence on markers and the use of multiple cameras that must
besynchronizedandcalibratedbyhand. Itistheaimofthisthesistodevelopmethods
that relax these constraints in order to bring inexpensive, off-the-shelf motion capture
severalstepsclosertoareality.
In doing so, we demonstrate that image projections of important anatomical land-
marks on the body (specifically, joint centre projections) can be recovered automat-
ically from image data. One approach exploits geometric methods developed in the
field of Structure From Motion (SFM), whereby point features on the surface of an
articulated body impose constraints on the hidden joint locations, even for a single
view. An alternative approach explores Machine Learning to employ context-specific
knowledge about the problem in the form of a corpus of training data. In this case,
joint locations are recovered from similar exemplars in the training set via searching,
samplingorregression.
Havingrecoveredsuchpointsofinterestinanimagesequence,wedemonstratethat
theycanbeusedtosynchronizeandcalibrateapairofcameras,ratherthanemploying
complex engineering solutions. We present a robust algorithm for synchronizing two
sequences, of unknown and different frame rates, to sub-frame accuracy. Following
synchronization, we recover affine structure using standard methods. The recovered
affine structure is then upgraded to a Euclidean co-ordinate frame via a novel self-
calibration procedure that is shown to be several times more efficient than existing
methodswithoutsacrificingaccuracy.
Throughout the thesis, methods are quantitatively evaluated on synthetic data for a
groundtruthcomparisonandqualitativelydemonstratedonrealexamples.
ii
Acknowledgements
Manythanksgofirsttomysupervisor,Dr. IanReid,forhisenthusiasticsupportduring
thegoodtimesandendlesspatienceduringthebad. Papersalwayssoundedbetterafter
his comments and suggestions, ideas came thick and fast, and he was always there to
steermeawayfromthemoretorturouspathsahead.
Thanks also go to all members of the Active Vision and Visual Geometry groups
at Oxford. They are a source of inspiration, enthusiasm and assistance whenever re-
quired. Joint thanks must also go to the staff of the Royal Oak, Woodstock Rd, for
theirgoodserviceduringtheweeklypost-reading-grouplabbanter.
My time in Oxford would have been a much less pleasant experience had it not
been for the good people I socialized with during my stay. In particular, thanks to
Adrian and Nick for the numerous hours spent down the pub patiently listening to my
griping about the PhD, only to return the favour and reminding me I wasn’t alone in
myfrustration. ThanksalsotoabsentfriendsEmilyandDiane-wemissyou.
Special thanks must go to Joanne for being such a loving companion during an
otherwisedifficultyear.
Thanks also go to friends from outside of the dreaming spires – Ste, Andy, Matt,
Tim, Chris, Rebecca, Melissa, Charlie, Gill etc. etc. Whenever Oxford felt a little too
small for comfort, they were there to remind me that there is another world outside,
too.
Finally, of course, ∞ thanks go to my parents for their love and support, both emo-
tional and financial. Their appreciation of the education system that their country has
to offer and the encouragement of their children to make the most of it got myself,
NickandSimonwherewearetoday. Thanks,folks–I’mdeadproud.
And to anyone I’ve forgotten to mention - thanks and apologies. I’m sure I’ll re-
memberyoulaterandfeelsorrythatIeverforgotinthefirstplace.
iii
Contents
1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.3 Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 CommercialMotionCapture . . . . . . . . . . . . . . . . . . . . . . 6
1.3.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 MarkerlessMotionCapture . . . . . . . . . . . . . . . . . . . . . . . 10
1.4.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 ThesisContributions . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Relatedwork 13
2.1 HumanMotionCapture . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.1 Trackingpeoplefromthetopdown . . . . . . . . . . . . . . 13
2.1.2 Trackingpeoplefromthebottomup . . . . . . . . . . . . . . 21
2.1.3 Importancesampling . . . . . . . . . . . . . . . . . . . . . . 23
2.2 StructureFromMotion . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.1 RankconstraintsandtheFactorizationMethod . . . . . . . . 25
2.2.2 ExtensionstotheFactorizationMethod . . . . . . . . . . . . 27
3 Recovering3DJointLocationsI:StructureFromMotion 29
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.1.1 Relatedwork . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2 MultibodyFactorization . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2.1 Universaljoint: DOF = 2,3 . . . . . . . . . . . . . . . . . 33
rot
3.2.2 Hingejoint: DOF = 1 . . . . . . . . . . . . . . . . . . . . 36
rot
3.2.3 Prismaticjoint: DOF = 0 . . . . . . . . . . . . . . . . . . 37
rot
3.3 Multibodycalibration . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.1 Universaljoint . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.2 Hingejoint . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.3.3 Prismaticjoint . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.4 Estimatingsystemparameters . . . . . . . . . . . . . . . . . . . . . 40
iv
3.4.1 Lengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.4.2 Angles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5 Robustsegmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.6.1 Jointanglerecoverywithrespecttonoise . . . . . . . . . . . 42
3.6.2 Linklengthrecoverywithrespecttonoise . . . . . . . . . . . 43
3.7 Realexamples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.7.1 Universaljoint . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.7.2 Hingejoint . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.7.3 Detectingdependentmotions . . . . . . . . . . . . . . . . . . 46
3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.8.1 Futurework . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4 Recovering3DJointLocationsII:MachineLearning 49
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.1.1 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.2 SearchingandSampling . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.1 LinearSearch . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.2 TreeSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.2.3 TreeSampling . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.3 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.3.1 LinearRegression . . . . . . . . . . . . . . . . . . . . . . . 55
4.3.2 KernelRegression . . . . . . . . . . . . . . . . . . . . . . . 57
4.3.3 NeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . 59
4.3.4 MixtureModels . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.4 ParticleFiltering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.4.1 Hybridprior . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.4.2 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.5.1 Data-DrivenPoseEstimation . . . . . . . . . . . . . . . . . . 62
4.5.2 Particlefiltering . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.6 RealExamples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.6.1 Starjumpssequence . . . . . . . . . . . . . . . . . . . . . . . 67
4.6.2 Squatssequence . . . . . . . . . . . . . . . . . . . . . . . . 68
4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.7.1 Futurework . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5 VideoSynchronization 71
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.1.1 Relatedwork . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.2 Generalizedrankconstraints . . . . . . . . . . . . . . . . . . . . . . 75
5.2.1 Homographymodel . . . . . . . . . . . . . . . . . . . . . . 75
v
5.2.2 Perspectivemodel . . . . . . . . . . . . . . . . . . . . . . . 76
5.2.3 Affinemodel . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.2.4 Factorizationapproach . . . . . . . . . . . . . . . . . . . . . 79
5.3 Rank-basedsynchronization . . . . . . . . . . . . . . . . . . . . . . 80
5.4 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.5.1 Monkeysequence . . . . . . . . . . . . . . . . . . . . . . . . 85
5.6 Realexamples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.6.1 Runningsequence . . . . . . . . . . . . . . . . . . . . . . . 90
5.6.2 Handstandsequence . . . . . . . . . . . . . . . . . . . . . . 90
5.6.3 Jugglingsequence . . . . . . . . . . . . . . . . . . . . . . . 93
5.6.4 ‘Pins’sequence . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.7.1 Futurework . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
6 Self-CalibratedStereofromHumanMotion 98
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.1.1 Relatedwork . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.2 Self-Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.2.1 Motionconstraints . . . . . . . . . . . . . . . . . . . . . . . 101
6.2.2 Structuralconstraints . . . . . . . . . . . . . . . . . . . . . . 102
6.3 Baselinemethod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.3.1 Recoveryoflocalstructure . . . . . . . . . . . . . . . . . . . 104
6.3.2 Recoveryofglobalstructure . . . . . . . . . . . . . . . . . . 104
6.4 Proposedmethod . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.4.1 Minimalparameterization . . . . . . . . . . . . . . . . . . . 106
6.4.2 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.5 Bundleadjustment . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.6 Practicalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.7.1 Runningsequence . . . . . . . . . . . . . . . . . . . . . . . 110
6.8 Realexamples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.8.1 Runningsequence . . . . . . . . . . . . . . . . . . . . . . . 115
6.8.2 Handstandsequence . . . . . . . . . . . . . . . . . . . . . . 116
6.8.3 Jugglingsequence . . . . . . . . . . . . . . . . . . . . . . . 117
6.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.9.1 Futurework . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7 Conclusion 121
7.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.2 Futurework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
vi
A AnEmpiricalComparisonofShapeDescriptors 143
A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
A.1.1 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 144
A.1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 145
A.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
A.2.1 Datasetgeneration . . . . . . . . . . . . . . . . . . . . . . . 145
A.2.2 Evaluationmethod . . . . . . . . . . . . . . . . . . . . . . . 146
A.3 Shaperepresentation . . . . . . . . . . . . . . . . . . . . . . . . . . 148
A.3.1 Lineartransformations . . . . . . . . . . . . . . . . . . . . . 148
A.3.2 Humoments . . . . . . . . . . . . . . . . . . . . . . . . . . 153
A.3.3 Lipschitzembeddings . . . . . . . . . . . . . . . . . . . . . 154
A.3.4 HistogramofShapeContexts . . . . . . . . . . . . . . . . . 156
A.4 Finalcomparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
A.4.1 Cleandata . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
A.4.2 Noisydata . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
A.4.3 Occludeddata . . . . . . . . . . . . . . . . . . . . . . . . . . 161
A.4.4 Realdata . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
A.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
A.5.1 Futurework . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
vii
Chapter 1
Introduction
The ability to interpret actions and “body language” is arguably the ability
that has enabled humans to form complex social structures and become the
dominant species on the planet. This thesis focuses on a computational solu-
tiontothisproblem,knownasHumanMotionCapture(HMC),wherewewish
to recover the human body pose in each frame of an image sequence. In this
firstchapter,weintroduceHMCinthewidercontextofMachineVisionbefore
outlining its applications, commercial (i.e. markered) solutions and limita-
tions. Wethendiscussmarkerlesssystemsthatexistinresearchenvironments,
theproblemstheyovercomeandtheproblemsyettobesolved.
1.1 Background
Human beings absorb much of their information regarding the real world via visual
input. This visual input is essential for day-to-day tasks such as searching for food,
detecting and avoiding hazards, and navigating within our environment. The aim of
Machine Vision is to replicate this faculty using cameras and computers, rather than
the eyes and brain, to receive and process the data, thus bestowing the same abilities
onmobilerobotsandintelligentcomputersystemsofthefuture.
Since the mapping from the 3D world to a 2D image incurs significant informa-
tionloss(i.e.depth),weimposeconstraints,typicallyencodedasassumptionsorrules
learned from experience, to rule out spurious or inconsistent interpretations of com-
plex scenes. Indeed, these assumptions are sufficiently strong that they may induce
1
1. INTRODUCTION
Figure1.1: TwotwinsinanAmesroom.
an incorrect interpretation of the scene geometry, as demonstrated by optical illusions
suchastheAmesroom(Figure1.1).
This thesis focusses on constraints that apply to images of articulated objects. We
defineanarticulatedobjectasanystructurethatispiecewiserigidbutdeformsaccord-
ing to a finite number of degrees of freedom. Since a rigid body has 6 degrees of
freedom (corresponding to translation and orientation in 3D), a collection of N rigid
bodies will in general have 6N degrees of freedom. However, articulation between
objects reduces the number of degrees of freedom such that the structure can be com-
pletelydeterminedby< 6N parameters.
Articulated objects are of considerable interest to us since they are abundant in our
environment, ranging from furniture fittings and mechanical linkages to biological or-
ganisms,includingthehumanbodyitself. Itisourhighlydevelopedabilitytointerpret
imagesofsuchdynamicstructuresthathaveenabledhumanstointeractandcommuni-
2