A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms

8.5 MB·

by  Abdolrahim Kadkhodamohammadi

A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms
AbdolrahimKadkhodamohammadi1,AfshinGangi1,2,MicheldeMathelin1,NicolasPadoy1
1ICube,UniversityofStrasbourg,CNRS,IHUStrasbourg,France
2RadiologyDepartment,UniversityHospitalofStrasbourg,France
{kadkhodamohammad, gangi, demathelin, npadoy}@unistra.fr
7
1
Abstract
0
2
Many approaches have been proposed for human pose
n
estimationinsingleandmulti-viewRGBimages. However,
a
J some environments, such as the operating room, are still

5 verychallengingforstate-of-the-artRGBmethods. Inthis
2 paper, we propose an approach for multi-view 3D human
pose estimation from RGB-D images and demonstrate the
]
V benefits of using the additional depth channel for pose re-
finementbeyonditsuseforthegenerationofimprovedfea-
C
tures. Theproposedmethodpermitsthejointdetectionand Figure 1. Synchronized pairs of color and depth images from a
.
s estimationoftheposeswithoutknowingapriorithenumber novelmulti-viewoperatingroomdataset.Theimagesarerecorded
c ofpersonspresentinthescene. Weevaluatethisapproach duringlivesurgeriesusingathree-viewRGB-Dcamerasystem.
[
onanovelmulti-viewRGB-Ddatasetacquiredduringlive

1
surgeriesandannotatedwithgroundtruth3Dposes.
v We believe that the drop in performance is caused by
2 the inherent visual challenges present in such an environ-
7
ment. Figure 1 shows an operating room during real surg-
3 1.Introduction
eries: one can notice that the visual appearances of many
7
0 Recovering the configuration of human body parts, differentsurfacesareverysimilar, whichmakesitdifficult
. whichisalsoreferredtoasHumanPoseEstimation(HPE), todistinguishpersonsfromthebackground.Peoplearealso
1
0 can benefit a wide variety of applications such as video wearing loose and textureless gowns that make it hard to
7 surveillance and behavior analysis [16] and human com- discriminatebodyparts.Inaddition,thecamerapositioning
1 puterinteractions[20].Vision-basedHPEhasbeenactively possibilities are very limited due to the objects and equip-
v: researched over the years and promising results have been mentthatneedtobefrequentlydisplacedintheroom,such
i reported on various challenging datasets recorded in com- asceilingmountedarticulatedarms,screensandtherespira-
X
monindoorandoutdoorscenes[28,41,33,3,40,19]. torytower. Furthermore,performingasurgeryrequiresthe
r Even though state-of-the-art models such as [19, 41] collaborationofmultiplepeople,whichincreasestheriskof
a
achieve impressive results on standard computer vision occlusions.
datasets, our experiments show that they do not necessar- In[23],wehaveproposedanapproachbasedonpictorial
ily generalize well to special environments like operating structures for clinician detection and pose estimation. The
rooms(ORs). ThequantitativeresultspresentedinSection approachreliesonapairofcoloranddepthimagescaptured
3 on data recorded during real surgeries show that there is from a single viewpoint using a low-cost RGB-D sensor
stillalargemarginforimprovement. Theseresultsarealso similartotheMicrosoftKinectcamera[30]. Thatworkhas
inagreementwith[23],whoreportedthattheKinectskele- demonstrated that the combination of color and depth in-
ton tracker [30], which has been successfully used in the formationalongwiththeuseofdepthinformationtomodel
gameindustry,doesnotgeneralizewelltotheORenviron- 3Dconstraintsbetweenneighboringbodypartsgreatlyim-
ment.Themainissuesareincorrectbackgroundsubtraction proves the pose estimation results. Following the findings
andthemixingofthebodypartsbelongingtodifferentper- of[23],weproposeinthispapertouseamulti-viewsystem
sons. Our supplementary video presents qualitatively how basedonRGB-Dcameras. Thesystemusesthreecameras
theaforementionedapproachesperformonourORdata. in order to capture the environment from complementary
views and to reduce the risk of occlusions in such a clut- specific environment like the operating room, a priori in-
tered environment. We show that the advantages of using formation about the room should be leveraged to identify
depth maps in such a multi-view approach go beyond the spuriouscandidates. Therefore, wealsoproposeamethod
meregenerationofimprovedappearancefeatures. tolearnaprioronthe3Dbodykinematicandroomlayout
In a multi-view RGB system, correspondences across constraints. This prior, based on random forest, is used to
viewsaretraditionallyestablishedbyrelyingonappearance recognizeandremoveskeletonswithunlikely3Dshapesor
similarityandtriangulation[6,14,2,7],whichisunreliable positions. Relyingdirectlyonhighlevel3Dskeletoninfor-
in OR environments containing many surfaces that are vi- mationenablesthemodeltobetterexploretheaprioriin-
sually similar. Instead, the depth data enables us to back- formationoftheORandtobuildastrongerpriorcompared
project points to 3D and is not affected by the visual ap- totraditionalposepriorsthatarebasedonthedisplacement
pearance of the surfaces in the scene [23]. It also enables orvisualsimilarityamongparts[40,19,23].
us to back-project points that are only visible in one view, Thispaperinvestigatesmulti-personmulti-viewposees-
whileinmulti-viewRGBsystems,pointsshouldbevisible timation using RGB-D data and makes the following con-
inatleasttwoviews. tributions. First, we extend the 3D pictorial structures of
Current multi-view human pose estimation approaches [23] to use a ConvNet body part detector on RGB-D im-
have been proposed either for single-person scenarios [14, ages. Second,weproposearandomforestbasedmethodto
8,18,2]orformulti-personscenariosinwhichthenumber automaticallylearnapriortoincorporateaprioriinforma-
of persons is known in advance [25, 6, 7]. The approach tion about the environment. Third, we introduce a novel
proposedinthisworkmakesnoassumptionaboutthenum- multi-view energy formulation to estimate 3D body con-
berofpersonsinthescene. Tothisend, ourapproachfirst figurations by leveraging depth data and reasoning across
processeseachviewseparatelytodetectputativeskeletons. all views. Finally, we have evaluated the approach on a
Next,aprioriinformationabouttheenvironment,modeled novel multi-view OR dataset generated from several days
usingrandomforests,isappliedtofilterspuriousskeletons. ofrecordingsduringlivesurgeries.
Theresultingskeletonsarethenmergedacrossviews1. Fi-
Related work. Approaches using RGB-D data for human
nally, a novel energy function is optimized to incorporate
pose estimation are mostly based on background subtrac-
evidence across views and update initial part positions di-
tion [4, 30], which is a very challenging task in the OR.
rectlyin3D.
An exception is [9], which relies on random forests and
Our single-view RGB-D pose estimation approach ex-
color to cluster image pixels into body parts and then es-
tends 3D Pictorial Structures (PS) [23] by incorporating
timate body poses. However, the performance of the sys-
ConvolutionalNeuralNetworks(ConvNets)forthepartde-
tem degrades dramatically due to occlusions and clutter,
tection [19]. ConvNets have recently enjoyed a great suc-
whichmisleadthepixelclassifierandthecolor-basedclus-
cess in solving many vision-based tasks including human
teringalgorithm. Themulti-viewRGB-Dsystemsthathave
pose estimation [36, 34, 29, 19, 27]. They are capable of
beenproposedalsorelyonbackgroundsubtractionandusu-
learningstrongdetectorsthatcanincorporateawideimage
allyaddresssimplescenariosinlaboratorysetups[39]. For
context through deep network architectures with large re-
this reason, we focus in the following description on part-
ceptivefields[38]. Mutualspatialconstraintsamongbody
based approaches (using RGB images) that do not rely on
partsarehowevernotexplicitlymodeled,eventhoughthey
background subtraction, since they are at the core of our
areessentialtoguaranteejointconsistencyinthepredicted
method.Suchapproachesrepresentthehumanbodyasaset
body configuration, especially in multi-person and clut-
of body parts and model body kinematic constraints using
teredenvironmentssuchasORs. Therefore,weuseadeep
adeformationmodel. Thepictorialstructuresframeworkis
ConvNet-based part detector constructed for RGB-D data
the main part-based approach and has driven much of the
inconjunctionwitha3Dpairwisedependencymodeltoen-
progressinthefieldsinceitsintroductionin[13].
force body kinematic constraints directly in 3D. This is in
In single-view pictorial structures, exact inference was
contrast to current methods that rely on 2D displacement
madetractablebytheseminalworkof[12],whichhasbeen
[40, 21, 35] or visual similarities [19] among body joints.
extendedindifferentwaysbyeitherconstructingastronger
Enforcingbodykinematicconstraintsin3Discrucialtore-
bodypartdetector[10,41]orbyimprovingthedeformation
liablyestimatebodypartconfigurationsofdifferentindivid-
model[28,23]. Mostrecently,withtheavailabilityoflarge
ualswhoareclosetoeachotherintheprojected2Dimage
trainingsetsandhighcomputationalpower,state-of-the-art
andarevisuallysimilar.
resultsareobtainedbyusingdeepconvolutionalneuralnet-
Incorrect detections and occlusions can however result
worksasbodypartdetectors[34,19,26].
in spurious skeleton candidates in each view that can mis-
Single-viewmultiplehumanposeestimationisoftenper-
leadthemulti-viewmergingalgorithm. Wearguethatina
formedintwosteps: persondetectionfollowedbyposees-
1Weassumethattheextrinsicparametersofthecamerasareknown. timation[15,32]. But,whenpeopleareincloseproximity
to each other, e.g. in the operating room, a body part can recordingsduringlivesurgeries,whichismorechallenging
beassignedtomorethanonepersonduetoweakbodypart dueto: (1)ahighvariationinnumberofpersonsperframe
detections or occlusions. [19], which is built on [27], has and(2)thepresenceofmanymovableobjectsinthescene,
proposedamulti-personposeestimationapproachbasedon whichmakesitdifficulttocomputetheforeground.
integerlinearprogramingtojointlydetectpeopleandesti-
mate their body part configurations. In this approach, part 2.Method
detection is performed using deep ConvNets and interpart
Westartthissectionbyrecapitulatingtheclinicianpose
pairwiseconstraintsareenforcedbasedon2Ddisplacement
estimatorof[23]andthenpresentthedifferentcomponents
andappearancesimilaritybetweenbodyparts. But,image-
thatleadtoourmulti-viewRGB-Dapproach.
basedpairwiseconstraintsarenotverydiscriminativeinop-
erating rooms since people are wearing textureless gowns 2.1.Single-viewbodyposeestimator
withsimilarcolors(seeFigure1). Instead, [23]usesmore
In [23], we have presented a pictorial structures model
discriminative3Dpairwiseconstraints. Wefollowasimilar
to estimate body configurations on RGB-D images. This
formulation to [23] using 3D pairwise constraints, but in-
model represents the body as a set of n joints and learns
cludeadeepConvNet-baseddetectorinourmodelinstead
multiple mixtures of parts to be robust to appearance
ofasupportvectormachinewithengineeredvisualfeatures.
changes. Themodelusestenbodyjointstoindicateupper-
Most work on multi-view human pose estimation fo-
body poses, since lower body parts are often occluded in
cusesonsingle-personscenariosincontrolledenvironments
operating rooms. A body configuration is specified by a
to reduce the ambiguity of data association by relying
pair(l,t),wherel ={l ...l }indicatesthe2Dpositionsof
on background subtraction or exemplar-based approaches 1 n
thebodyjointsandt belongstoasetofmpossiblemixture
[14,18,31]. Inordertoberobusttoclutteredbackground, i
typest={t ...t }foreachbodyjoint.Theposeestimation
part-basedapproachesareusedtogenerateparthypotheses 1 n
isdefinedasanenergyminimizationoveratree-structured
per view and then estimate the body configuration in 2D
graph G = (V,E), whose nodes are the body joints and
[1,2]orin3D[8].
whose edges indicate dependencies between joints. The
Recently, a PS approach has been proposed to estimate body joint dependencies are defined following the human
theposesofmultiplepersonsinamulti-viewsetup[5]. The bodyskeleton. Givenapairofalignedcoloranddepthim-
bodyposeestimationisperformedin3Dbyrelyingon2D agesdenotedbyI andD,respectively,thescoreassociated
view appearance cues and on multi-view cues computed withabodyconfiguration(l,t)isdefinedas:
through triangulation. This approach has been extended
in [6] to also include temporal cues. However, all these S(I,D,l,t)=(cid:88)φ(I,D,l )+ (cid:88) wti,tj.ψ(D,l ,l )
i ij i j
methodsrequirepriorknowledgeofthenumberofpersons i∈V ij∈E
present and have been evaluated on scenarios recorded in +(cid:88)bti + (cid:88) bti,tj,
controlled laboratory environments that include people in i ij
i∈V ij∈E
upright poses only. In contrast, our method detects the
(1)
number of persons and does not rely on triangulation that
mightnotalwaysbereliableincomplexandclutteredenvi- wherethefirsttermistheappearancemodel,whichisalso
ronments. Moreover,wehaveevaluatedourapproachona referred to as the part detector, and the second term is the
datasetrecordedduringrealsurgeries,whichcontainsmany deformationmodelthatenforcespairwisedependenciesbe-
visual challenges and where people exhibit a much wider tween body joints. The last two terms are part type com-
rangeofarticulationscomparedtothoseusedintheafore- patibility score functions, where bti captures the score of
i
mentionedworks. assigningaparticularmixturetypetopartiandbti,tj isthe
ij
A multi-view clinician pose estimation approach has scoreassociatedwiththeco-occurrenceofaparticularpair
been proposed in [7]. Background subtraction and track- ofparttypes. Thecompatibilityscorefunctionsandwti,tj
ij
ing over the entire sequence are used to find the trajec- arethemodelparameters. Theseparametersarelearnedus-
tories of the persons and to localize them using bounding ingastructuredsupportvectormachineformulation.
boxes. Then,theapproachpresentedin[5]andaConvNet- Thepartdetectorassignsaconfidencescoreforplacing
based RGB part detector are used to estimate the pose of the body joint i at image location l . [23] relies on hand-
i
eachindividualgiventheboundingboxes. Toevaluatethis crafted features, namely Histogram of Oriented Gradients
approach, two sequences containing a constant number of (HOG)andHistogramofDepthDifferences(HDD).Inthis
individuals have been recorded during two medical proce- work,wecomputethepartdetectionscoresusingtheDeep
dures simulated by actors. In contrast, our approach re- ConvNetmodelpresentedinSection2.1.1.
liesonamulti-viewsetofimagesfromasingletime-step. The deformation model is parametrized by w and
ij
Moreover,ourdatasethasbeengeneratedfromfourdaysof ψ(D,l ,l ). The weights w encode the deformations
i j ij
between pairs of joints. The relative displacement of useallpositivesamplesandkeepatmostthreetimesmore
joint i w.r.t. joint j is captured by ψ(D,l ,l ) = negativesamples.Thenetworkistrainedwithcrossentropy
i j
[|d |,dc,dc2,dr,dr2]T, where |d | is the absolute 3D lossandstochasticgradientdescentfor50kiterations. The
3D 3D
Euclidean distance between the joints and (dc,dr) are initiallearningrateissetto5×10−5fortheadaptedlayers
the relative displacements along columns and rows of the and5×10−6fortherest. Thisyieldsthebestresultsinour
image. The 3D joint positions are computed by back- experiments. In[19],thenetworkistrainedforthreetasks:
projecting 2D joints into 3D using the depth image. This bodypartdetection,locationrefinement,whichistherela-
term enables the model to incorporate more reliable 3D tiverowandcolumndisplacementfromascoremaplocation
bodypartlengthsforassemblingbodyjoints,whichisim- tothegroundtruth,andregressiontootherparts. However,
portant in order to discriminate between detections on the trainingforthelasttwotasksdidnotyieldanyperformance
surfaceofapersonanddetectionsonthebackground. improvement during our experiments. We therefore only
Estimating the body configuration in this model trainforthebodypartdetectiontask.
corresponds to finding the optimal body joint posi-
2.2.Randomforestsbasedprior
tions and mixture types given color and depth images.
This process, called inference, computes (l∗,t∗) =
Todesignarobustmethod,webelievethatitisessential
argmax S(I,D,l,t). It is performed in 3D using an ef-
l,t toincludepriorsspecifictotheenvironment. Eventhough
ficient algorithm that makes exact inference tractable. For
a general body kinematic prior is included in the pose es-
moreinformation,wereferthereaderto[23].
timation model through pairwise constraints, it cannot be
guaranteed that these constraints are always properly en-
forced due to the high complexity of the pose estimation
2.1.1 ConvNet-basedRGB-Dbodypartdetector
modelthatpredictshumanposesdirectlyfromimagepixel
Motivated by the great success of deep convolutional neu- values. Inaddition,thisprioronlycapturesbodykinematic
ral networks in recent years [29, 17, 34, 19, 26], we pro- constraints and does not incorporate a priori information
posetouseRGB-DbodypartdetectorsbasedondeepCon- abouttheenvironment.InanenvironmentliketheOR,con-
vNets in order to automatically learn feature representa- straintssuchaspossiblehumanposesandpossiblelocations
tions instead of relying on engineered feature representa- can also be used to improve the reliability of the method.
tions such as HOG or HDD. To this end, we build on the Such constraints cannot be easily handcrafted. Further-
very deep residual network [17], which has recently been more, including them in the pose estimation model would
used for part detection and shown promising results [19]. need higher-order dependency terms. Adding such terms
Thebodypartdetectionisformulatedasamulti-labelclas- wouldincreasethenumberofmodelparametersand,more
sification problem, where a set of n scores is generated at importantly,dramaticallyincreasethecomplexityofthein-
each image location to denote the probability of part pres- ference algorithm. We therefore propose to automatically
ence. The scores are obtained by using sigmoid activation learntheprior,whichweformulateasabinaryclassification
functions on the output neurons. We adapt the network to problemthattakesaskeletonestimatedbythesingle-view
learn body part detectors for pairs of color and depth im- detector as input and outputs whether this skeleton corre-
ages. Wechangetheinputlayertoacceptfourdimensional spondstoaspuriousdetectionornot.
data(i.e.threecolorchannelsanddepthchannel). Wealso We base our approach on Random Forests (RF), which
change the res3d pose layer to generate part score maps areanensembleofdecisiontreesconsistingoftwotypesof
for ten upper-body parts instead of the fourteen full body nodes: split and leaf nodes. In each split node, a decision
parts. During pose estimation, we use the ConvNet-based functionforwardssamplestooneofthebranchesuntilthey
bodypartdetectortopredictconfidencescoresforallparts finallyreachaleafnodecontainingapredictionfunction.In
at every image locations. Hereafter, we refer to this HPE ourcase,weuseRFwithbinarytreesandthemeanoverall
modelasDeep3DPS. predictionstoaggregatethevotesacrossalltrees. Thetrees
Fine-tuning.Weinitializethenetworkfromthepre-trained arelearnedautomaticallygivenalabeledtrainingset,which
model of [19], which is trained on the MPII Human Pose we construct using the skeletons estimated by our single-
dataset. We fine-tune the network on the single-view clin- view pose estimator on a set of images for which ground-
ician pose dataset from [23] that consists of 1451 RGB-D truth is available. The detected skeletons are compared to
frames including 1991 persons using the Caffe framework the ground truth using the probability of correct keypoints
[22].Wescaletheimagesdownto85%anduseabatchsize (PCK) metric, which is commonly used for evaluation in
oftwo. Similarlyto[19],wegeneratetargettrainingscore multiple-person pose estimation [41, 23, 27]. We label a
mapsforallbodyjointsbyassigningthepositivelabel1for detectedskeletonaspositiveifthehead,neck,andleftand
allimagelocationswithin15pixelstothegroundtruthlo- rightshouldersarecorrectlylocalizedaccordingtoPCK.
cationandnegativelabel0otherwise. Duringtraining, we ForRFtraining,weproposetocombinevariousfeatures
computed from the 3D skeletons, which are all expressed 2.3.2 Multi-viewRGB-DOptimization
inthecommonroomreferenceframe. Thereferencecoor-
Weformulateourmulti-viewRGB-Dapproachasanenergy
dinatesystemischosenw.r.t. theoperatingtableindefault
minimizationoverthesamegraphGasinSection2.1and
position,whichmakesthepriorgeneralizabletootherORs.
definetheenergyfunctionE(∆)overthegraphas:
Thisenablesourpriortoencodetwotypesofinformation:
room layout and possible clinician poses. Certain parts of
(cid:88)(cid:16) (cid:17)
the room, such as the floor or the ceiling, are for instance E(∆)= λ1.Φconf(δi)+λ2.Φdepth(δi)
notexpectedtohavecliniciansorcertainbodyparts. Thus, i∈V (2)
(cid:88)
asfirstsetoffeatures,weusethepositionsofthe3Dbody + Ψ (δ ,δ ),
i,j i j
partstoenabletheRFtobuildaninternalrepresentationof
(i,j)∈E
their spatial occupancy probability. To capture the set of
possible human poses in the OR, we include a second set whereλ1 andλ2 areweightingcoefficients,∆={δ1...δn}
offeatures, namelytherelative3D displacementsbetween is a set of displacement labels for all body parts, δi ∈ R3
all pairs of body joints. The prior also serves to verify 3D is a 3D displacement offset for part i, Φ(.) are the unary
partlengthsandexcludeincorrectskeletonsthatmayoccur potentials and Ψi,j(δi,δj) is a pairwise dependency term
due to weak detections and foreground/background confu- enforcingbodyphysicalconstraints.
sions. As third feature, we include the detection score of The first term in (2) incorporates part detection confi-
theindividualskeletontoincorporatedetectionconfidence. dencescorescomputedbytheConvNetpartdetector.Given
Toenableourpriortobetterencodehigh-levelinformation, thelistofallviewsviews,wedefine:
weusetheRFmethodinamulti-layerscheme,referredto (cid:88) (cid:16) (cid:17)
as auto-context in the machine learning literature [37]. A Φconf(δi)= conf proj(P(δi),v) , (3)
multi-layermodelislearned,wherethefirstRFlayeriscon- v∈views
structed using only the three aforementioned types of fea-
where P(δ ) is the 3D position of part i displaced by an
i
tures,whiletheotherlayersuseanotherextrafeaturethatis
offset δ and proj(p ,v) projects the 3D point p . In
i 3D 3D
the classification confidence generated by the previous RF
order to provide a smooth cost function, we compute the
layer.
distancetransformsofthedeepConvNetscoremapsusing
2.3.Multi-viewhumanposeestimation thegeneralizeddistancetransformalgorithm[12]. Wefind
thatthistransformationisnecessarytoavoidlocalminima.
2.3.1 Multi-viewfusion
conf(p )∈[0..1]isthevalueofthedistancetransformof
2D
Theobjectiveofthemulti-viewfusionistocombinethe3D thescoremapofpartiatlocationp2D. Thesecondtermis
skeletons across all views. For a given frame, defined as definedas:
a set of RGB-D images recorded from all cameras at the (cid:88) (cid:12) (cid:16) (cid:17) (cid:12)
Φdepth(δ )= (cid:12)D proj(P(δ ),v) −Z(P(δ ),v)(cid:12),
same time step, detections from all views are first put in a i (cid:12) i i (cid:12)
set. The two closest skeletons that do not originate from v∈views
(4)
the same view are then merged. This procedure is iterated
untilnopairofmergingcandidatesisleftintheset,where
where D(p ) is the depth value at image location p ,
2D 2D
theconditionformergingtwoskeletonsisthatthedistance
Z(p ,v)isthezvalueofthe3Dpointp inthecoordi-
3D 3D
between their heads and the distance between their necks
nate system of the view and |.| is the absolute value oper-
arebothsmallerthanaconstantT . Sincetheleft/rightside
s ator. To reduce the effect of the noise present in the depth
labels of the individual detections are not always reliable,
image, we smooth the depth image with a median filter of
to ensure a consistent merging of the 3D joints we use the
size 7×7px. This term quantifies the distancebetween the
3Dpositionsoftheshoulderstofindthecorrectassociation
displaced 3D joint and the surfaces captured by the depth
betweenthetwoskeletons. Finally,forallskeletonsresult-
cameras. Therefore, it can help to avoid placing parts in
ingfromamergingstep,theleftandrightsidelabelsareset
ghost3Dlocationsthatdonotcorrespondtoanysurfacein
based on a majority vote among the supporting skeletons.
the scene. These two unary terms incorporate multi-view
If a merged skeleton originates from only two supporting
cues, wheretheRGB-DConvNetisusedtoincludeimage
skeletons, whichdonot agreeonthesidelabel, we setthe
evidence and depth is used to integrate a reprojection cost
sideaccordingtotheskeletonwithhighestconfidence.
acrossallviews.
Asaresult,weobtainasetofinitial3Dskeletonsgener-
The pairwise term is used to enforce kinematic con-
atedfromskeletonscomingfromoneormoreviews. Then,
straints, namely body part lengths between pairs of joints.
a new multi-view energy function, presented next, is used
LetΨ bedefinedas:
i,j
to drive the body parts towards their optimal 3D locations
byjointlyoptimizingoverallviews. Ψ (δ ,δ )=|(cid:107)P(δ )−P(δ )(cid:107)−µ |, (5)
i,j i j i j i,j
Setting Head Shld Elbow Wrist Hip Avg
Deep3DPS(DeeperNet) 89.6 56.5 50.6 54.3 42.9 58.8
Deep3DPS(RGB) 93.7 74.9 69.6 71.8 66.6 75.3
Deep3DPS(Depth) 91.0 75.0 69.1 68.0 63.2 73.2
Deep3DPS(RGBD) 93.4 77.0 71.5 73.7 69.1 76.9
+Auxiliarytasks 91.4 72.1 64.9 68.4 63.5 72.1
3DPS(IHOG+HDD)[23] 90.8 74.2 62.2 63.4 57.5 69.6
Insafutdinovetal.[19]2 91.1 53.7 47.5 50.1 38.4 56.2
YangandRamanan[41]3 30.4 35.2 19.6 24.3 16.7 25.2
Table1.Poseestimationresultsofseveralsingle-viewapproaches
Figure2.Multi-viewexamplesillustratingtheresultsoftheRF- usingPCKmetric.
basedprior. Acceptedskeletonsareshowninorangeandrejected
skeletonsinpurple.
distributedacrossthedataset. Allclinicianswhohavemore
than 50% of their upper-body parts visible in at least one
where (cid:107).(cid:107) is the L -norm and µ is the average distance
2 i,j viewhavebeenannotatedintheseframes. Theannotations
betweenjointsiandj,i.e.averagepartlength. Theaverage
areperformedusingatoolthatdisplaysa3Dpointcloudre-
part lengths are computed over the entire training dataset.
constructedfromallthreeviewsaswellasthecorrespond-
Notethatsincethebodypartlengthsarerelativelyconstant
ingindividual2Dimages. Thistoolallowstheusertomove
in3D,itisherenotneededtolearnperson-specificaverage
the body joints either in the 2D views or in the 3D point
partlengths.
cloud. Whenever a joint is moved in 2D, the 3D position
Inference. Inordertorecover3Dbodypartconfigurations,
ofthecorrespondingjointintheaverage3Dskeletonisup-
we need to perform inference in 3D. This problem corre-
datedusingthedepthmapandthenreprojectedbacktoall
sponds to optimizing the energy function in Eq. (2). Note
views. Thus,theannotatorcanverifythecorrectnessofthe
thatusingtheoptimizationalgorithmof[23]wouldrequire
annotatedskeletonsusingboth3Dvisualizationand2Dre-
toconstructa3Dstatespacethatincludesall2Dpositions
projectionsacrosstheviews.
back-projected to 3D (amounting to the number of views
In order to have a fair comparison with our method in
multipliedbythesizeoftheimages)augmentedwithextra
[23], the single-view dataset of [23] is used for training
nodes for each back-projected node to account for occlu-
all3DPSsingle-viewposeestimationmodelsandforfine-
sions. Such a large state space would degenerate the per-
tuningthenetwork. Forallmodels,evaluationisperformed
formanceandslowdowntheinference. Similarly,theinfer-
onthenewmulti-viewdataset. Asthemulti-viewdatasetis
enceapproachfrom[8]wouldlimitustousesimplebinary
usedforrandomforesttraining,toevaluatethemodel,a4-
pairwise terms. Instead, we perform discrete optimization
foldleave-one-outcross-validationisperformed,whereone
usingthefast-PDalgorithm[24],whichcaststheoptimiza-
foldsisusedfortestingandtherestfortraining. Theevalu-
tionprobleminanintegerprogrammingframeworkandex-
ationreportstheaverageresultsofthecross-validation.
ploitssolutionsformbothprimalanddualproblemsforeffi-
ciency. Toperformtheoptimization,wedefineasetofdis- 3.1.Single-viewposeestimation
cretedisplacementlabelsLforeachbodyjointbysampling
Table 1 reports the performance of different models on
densely from a cube centered at the initial joint position.
the multi-view dataset using the PCK metric [41, 23, 19].
Thesamplingfunctionisparametrizedby(k,s),wherekis
All 3DPS models have been trained on the single-view
thenumberofsamplesalongeach3Ddirectionandsisthe
dataset used in [23]. The 3DPS method using the pre-
step size between the samples. We perform the optimiza-
trained network of [19] as body part detector, referred to
tioniterativelybystartingwithacoarselabelsetthatcovers
as DeeperNet, achieves a better performance compared to
a large 3D space. At the end of each iteration, we update
the full DeeperCut approach from [19] that estimates the
thepartpositionsbasedonthedisplacementlabelsandthen
bodyposesviaajointoptimizationacrossallpeople. These
generateafinerlabelsetforthenextiteration.
results indicate that in an environment with many visu-
ally similar surfaces, a 3D deformation model, even with
3.Experimentalresults
tree-structured graph, is more reliable than a fully con-
Datasets. We have generated a novel multi-view RGB-D nected deformation model which relies on appearance and
dataset,illustratedinFigure1,byrecordingallactivitiesin 2D displacement constraints. Fine-tuning the network on
anoperatingroomforfourdays. Forquantitativeanalysis,
2Weusethemodelthatwasmadepubliclyavailablebytheauthors.
the3Dupperbodyposesof1378clinicianshavebeenman- 3ThemodelistrainedontheBuffydataset[11]usingthepublicimple-
ually annotated in 741 multi-view frames that are evenly mentationoftheapproach.
87 100
80
Accuracy8856 Precision6400
Normal+flipped 20 Withoutprior
Normal Withprior
841 2Au3to-4con5text6 La7yer8 9 10 00 20 40Recal6l0 80 100
(a) (b)
Figure4.(a)AccuracyoftheRF-basedpriorindetectingspurious
skeletons.(b)Precision-recallcurvesfor3Dcliniciandetection.
we learn 100 shallow trees with a maximum depth of 10.
Figure 3. Examples of multi-view pose estimation results. Each
Figure 4(a) shows the detection accuracy of the RF-based
rowshowsamulti-viewframe. The3Dskeletonsobtainedafter
method in distinguishing spurious detections. The results
multi-viewenergyoptimizationareprojectedtotheviews.
show that the method always detects valid skeletons with
an accuracy superior to 84%. One observes that augment-
the single-view dataset significantly improves the results ing the training set by flipping the skeletons consistently
(Deep3DPS (RGB): 75.3% vs. 58.8% PCK), as it allows improvestheresultsbyenablingtheforesttolearnaricher
the network to adapt its representation for learning a bet- prior model that is not confused by the noisy side detec-
terencoderforsuchanenvironment. Wehavealsotrained tions. The results also indicate that auto-context enhances
the network to detect body parts using only depth data, theperformanceuptothefourthiterationandthentendsto
Deep3DPS (Depth), which achieves competitive results. overfit. WethereforeusetheoutputoftheRFtrainedonthe
The best performance is obtained when the network relies augmented training set at the fourth layer to identify spu-
onbothcoloranddepthimages: theresultingmodel,called riousskeletonsduringtheremainingevaluations. Figure2
Deep3DPS(RGB-D),isthereforeusedassingle-viewpose illustratestheresultsoftheapproachonsampleframesfrom
estimator during the rest of the experiments. But, we ob- themulti-viewdatasetandalsoonafewframesrecordedin
servethatonthisdata,trainingthenetworkfortheauxiliary a different room from totally different viewpoints. It can
taskssuggestedin[19],namelylocationrefinementandre- be seen that the method has correctly identified spurious
gression to other parts, degrades the performance. We be- skeletonsinbothdatasetsandgeneralizeswell. Theproper
lievethatthisisduetobothamuchsmallertrainingsetand generalizationisduetothefactthatthereferenceframeis
tothestrongforeshorteningofthebodypartsbecauseofthe definedontheflooratthecenteroftheoperatingtable,the
topviewsofthecameras. mainelementinanyoperatingroom.
As baseline, we evaluate the performance of the best
3.3.Multi-view3Ddetectionandposeestimation
3DPS model from [23], which relies on a 3D deformation
model similar to our approach, but with handcrafted color We set T to 30 cm to avoid merging skeletons across
s
and depth features. Our best model improves the perfor- personswhoareclosetoeachother. Weevaluate3Dclin-
manceoverthisbaselineby ∼7%onthesameexperimen- ician detection using the precision-recall curves. A detec-
tal setup. This highlights the benefits of deep ConvNets tion is accepted as a true positive if the distance between
inconstructingmorediscriminativebodypartdetectorsby theground-truthandthedetectionisbelow30cmforboth
automatically learning feature representations and also in- the head and neck. We use the fusion algorithm described
corporating a wider context. Evaluation of state-of-the-art inSection2.3.1togenerateasetof3Dskeletoncandidate
RGBmodels[19,41]trainedoncommoncomputervision perframe. Figure4(b)showsthe3Dcliniciandetectionre-
datasets shows that they do not generalize to the OR envi- sultsaftermulti-viewfusionwithandwithouttheRF-based
ronmentduetobothlooseclinicalclothesandthepresence prior. Thehighprecisionobtainedbyourmethodwhenthe
ofmanyvisuallysimilarsurfaces. ORpriorisusedindicatesthehighqualityofthegenerated
skeletons.
3.2.Randomforestbasedprior
To optimize part positions based on multi-
The Deep3DPS (RGB-D) model is applied to detect view cues, we generate four label sets {(k,s) :
skeletonsineachviewofthemulti-viewdatasetseparately. (3,50), (5,10), (7,2), (7,1)}, where the step sizes
The skeletons are back-projected into 3D and transformed are in centimeter. We solve the optimization in four
intoacommonreferenceframe. Weusethese3Dskeletons iterations by going from a large and coarse search space
totrainourprior,asexplainedinsection2.2. Wealsoaug- towards a small and fine search space, which allows us
mentthedatabyflippingtheskeletonstoexchangetheleft- to more efficiently explore the 3D space. The parameters
right body parts. Due to the small size of the training set, usedinallexperimentsareλ = 2andλ = 0.5, thatare
1 2
Part Oneview Twoviews Threeviews
name
initial afteropt. opt.-Φdepth initial afteropt. opt.-Φdepth initial afteropt. opt.-Φdepth
Head 7±4 7±4 7±4 6±3 6±3 6±3 5±2 5±2 5±2
Neck 7±4 7±4 7±4 5±3 5±3 5±3 4±2 4±2 4±2
Shld 25±25 19±16 21±18 22±16 15±10 19±13 14±14 10±7 12±9
Hip 28±22 27±19 28±20 24±13 23±13 24±14 18±10 17±9 18±10
Elbow 31±22 27±19 30±21 30±18 23±15 27±18 19±14 16±11 18±14
Wrist 42±34 32±21 35±24 34±22 25±16 28±18 24±18 18±13 20±15
avg† 32±26 26±19 29±21 28±17 22±14 25±16 19±14 15±10 17±12
Table2.Meanandstandarddeviationof3Dpartlocalizationerrorincentimeter. Theresultsarepresentedasafunctionofthenumberof
supportingviewsusedtogeneratetheinitial3Dskeletons(distribution: 1view: 30%; 2views: 43%; 3views: 27%). †Theaverageis
computedforallpartsexcepttheheadandnecksincetheyarenotincludedintheoptimization.SeeSection3.3fordetails.
selected using grid search over a set of 50 frames from 4.Conclusions
the multi-view dataset. The mean and standard deviation
(STD)ofthe3DEuclideandistancesbetweenthepredicted Inthispaper,weproposeamulti-viewRGB-Dapproach
bodypartpositionsandtheground-truthpositionsareused fordetectingandestimatingthebodypartpositionsofmed-
toevaluate3Dbodypartlocalizations. icalstaffin3D.AConvNet-basedbodypartdetectorcom-
binedwitha3Dpairwisedeformationmodelisusedtore-
coverbodyposesineachview. Amethodbasedonmulti-
In Table 2, we present the evaluation results for multi-
layerrandomforestsisthenproposedtoautomaticallylearn
view body part localization as a function of the number of
aprioriinformationabouttheORandremovespuriousde-
supportingviews. Pleasenotethatsincetheheadandneck
tections per view, which allows us to reliably detect the
localizationerrorsareclosetotheexpectederrorinlow-cost
bodyposesofpersonsinthescene. Then,thesedetections
RGB-Dcameras,wedonotupdatethesetwojointsduring
areback-projectedto3Dandmergedacrossviews. Finally,
ouroptimization. Thistablepresentslocalizationerrorsfor
a novel optimization function is introduced to update the
the initial 3D skeletons obtained by the fusion algorithm
partpositionsbyrelyingjointlyonthebodypartconfidence
andtheerrorafterperformingthemulti-viewoptimization.
maps, depth data and multi-view cues. The method has
Onecannoticethattheproposedmulti-viewfusionmethod
been quantitatively evaluated on a new multi-view dataset
correctly associates skeletons across views by consistently
acquired during live surgeries. Experimental results show
reducingthelocalizationerrorsasthenumberofsupporting
significant improvements over state-of-the-art methods for
viewsincreases. However,weobservethatifweignorethe
thetaskofsingle-viewposeestimationinmulti-personsce-
left and right labels of the detections and assign the label
narios,indicatingthebenefitofcombiningdeeppartdetec-
basedonshoulderdistanceswithgroundtruth,thelocaliza-
torsand3Dpairwiseconstraintsinbuildingrobustmodels.
tion errors decrease by ∼10 cm for skeletons with one or
The multi-view formulation also achieves very promising
two supporting views and ∼3 cm for skeletons with three
results showing the benefits of the deep ConvNet detector
supporting views. These results indicate that the side de-
and of depth data for correctly driving parts towards their
tection in individual views is not very reliable. But, if a
optimallocations. Tothebestofourknowledge,thisisthe
person is detected in all views, the proposed voting algo-
firstmulti-viewapproachthatperformsbothhumandetec-
rithmcanmakeamorereliableprediction. Themulti-view
tionandposeestimationinarealscenariowithoutanyprior
optimizationsignificantlyreducesthelocalizationerrorfor
knowledgeonthenumberofpersonspresent,aswellasthe
skeletons with any number of supporting views. Interest-
first multi-view RGB-D approach presented for pose esti-
ingly,theoptimizationimprovestheresultsevenforskele-
mation.
tonswithonesupportingviewbyproperlyincorporatingthe
depth-based reprojection costs and detection confidences.
ThedeepRGB-Dpartdetectoristhemaindriveroftheop- Acknowledgements
timization. Toevaluatetheeffectofthedepth-basedrepro-
jectioncost, wealsoreporttheresultswithoutthistermin ThisworkwassupportedbyFrenchstatefundsmanaged
column‘opt.-Φdepth’. Thedropinperformancehighlights by the ANR within the Investissements d’Avenir program
its importance. The 2D projections of 3D poses obtained under references ANR-11-LABX-0004 (Labex CAMI),
usingtheproposedmulti-viewoptimizationareshownfora ANR-10-IDEX-0002-02 (IdEx Unistra), ANR-10-IAHU-
fewframesinFigure3. 02(IHUStrasbourg)andANR-11-INBS-0006(FLI).
References [14] J.Gall,B.Rosenhahn,T.Brox,andH.-P.Seidel. Optimiza-
tion and filtering for human motion capture. International
[1] S.Amin,M.Andriluka,M.Rohrbach,andB.Schiele.Multi- JournalofComputerVision,87(1):75–92,2010.
view pictorial structures for 3d human pose estimation. [15] G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik. Us-
In British Machine Vision Conference (BMVC), September ingk-poseletsfordetectingpeopleandlocalizingtheirkey-
2013. points. In Proceedings of the 2014 IEEE Conference on
[2] S. Amin, P. Mu¨ller, A. Bulling, and M. Andriluka. Test- ComputerVisionandPatternRecognition,CVPR’14,pages
timeadaptationfor3dhumanposeestimation. InX.Jiang, 3582–3589.IEEE,2014.
J.Hornegger,andR.Koch,editors,PatternRecognition,vol- [16] D. Gowsikhaa, S. Abirami, and R. Baskaran. Automated
ume8753ofLectureNotesinComputerScience,pages253– humanbehavioranalysisfromsurveillancevideos:asurvey.
264.SpringerInternationalPublishing,2014. ArtificialIntelligenceReview,42(4):747–765,2014.
[3] M.Andriluka,L.Pishchulin,P.Gehler,andB.Schiele. 2D [17] K.He,X.Zhang,S.Ren,andJ.Sun. Deepresiduallearning
humanposeestimation: Newbenchmarkandstateoftheart forimagerecognition. CoRR,abs/1512.03385,2015.
analysis. In2014IEEEConferenceonComputerVisionand [18] M.HofmannandD.M.Gavrila.Multi-view3Dhumanpose
PatternRecognition,pages3686–3693,June2014. estimationincomplexenvironment.InternationalJournalof
ComputerVision,96(1):103–124,2011.
[4] A.Baak,M.Mu¨ller,G.Bharaj,H.Seidel,andC.Theobalt.A
[19] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka,
data-drivenapproachforreal-timefullbodyposereconstruc-
tionfromadepthcamera.InProceedingsoftheInternational andB.Schiele. DeeperCut: ADeeper,Stronger,andFaster
ConferenceonComputerVision(ICCV),pages1092–1099. Multi-personPoseEstimationModel,pages34–50.Springer,
Cham,2016.
IEEE,2011.
[20] A.JaimesandN.Sebe. Multimodalhumancomputerinter-
[5] V. Belagiannis, S. Amin, M. Andriluka, B. Schiele,
action: Asurvey. ComputerVisionandImageUnderstand-
N.Navab,andS.Ilic.3dpictorialstructuresformultiplehu-
ing, 108(12):116 – 134, 2007. Special Issue on Vision for
manposeestimation.InComputerVisionandPatternRecog-
Human-ComputerInteraction.
nition(CVPR),2014IEEEConferenceon,pages1669–1676.
[21] A.Jain,J.Tompson,M.Andriluka,G.W.Taylor,andC.Bre-
IEEE,2014.
gler. Learninghumanposeestimationfeatureswithconvo-
[6] V. Belagiannis, X. Wang, B. Schiele, P. Fua, S. Ilic, and
lutionalnetworks. InInternationalConferenceonLearning
N.Navab. Multiplehumanposeestimationwithtemporally
Representations(ICLR),April2014.
consistent3Dpictorialstructures. InChaLearnLookingat
[22] Y.Jia,E.Shelhamer,J.Donahue,S.Karayev,J.Long,R.Gir-
PeopleWorkshop,EuropeanConferenceonComputerVision
shick, S. Guadarrama, and T. Darrell. Caffe: Convolu-
(ECCV2014).IEEE,September2014.
tionalarchitectureforfastfeatureembedding.arXivpreprint
[7] V. Belagiannis, X. Wang, H. B. B. Shitrit, K. Hashimoto, arXiv:1408.5093,2014.
R. Stauder, Y. Aoki, M. Kranzfelder, A. Schneider, P. Fua, [23] A. Kadkhodamohammadi, A. Gangi, M. de Mathelin, and
S. Ilic, H. Feussner, and N. Navab. Parsing human skele- N.Padoy. Articulatedcliniciandetectionusing3Dpictorial
tonsinanoperatingroom.MachineVisionandApplications, structuresonRGB-Ddata. MedicalImageAnalysis,35:215
pages1–12,2016. –224,2017.
[8] M.Burenius,J.Sullivan,andS.Carlsson. 3dpictorialstruc- [24] N.Komodakis,G.Tziritas,andN.Paragios.Performancevs
turesformultipleviewarticulatedposeestimation.InCVPR, computationalefficiencyforoptimizingsingleanddynamic
pages3618–3625.IEEE,2013. mrfs: Settingthestateoftheartwithprimal-dualstrategies.
[9] K.Buys,C.Cagniart,A.Baksheev,T.D.Laet,J.D.Schut- ComputerVisionandImageUnderstanding,112(1):14–29,
ter,andC.Pantofaru. Anadaptablesystemforrgb-dbased 2008. SpecialIssueonDiscreteOptimizationinComputer
humanbodydetectionandposeestimation.JournalofVisual Vision.
CommunicationandImageRepresentation,2013. [25] X.Luo,B.Berendsen,R.T.Tan,andR.C.Veltkamp. Hu-
manposeestimationformultiplepersonsbasedonvolume
[10] M. Eichner and V. Ferrari. Better appearance models for
reconstruction. In Pattern Recognition (ICPR), 2010 20th
pictorial structures. In British Machine Vision Conference,
InternationalConferenceon,pages3591–3594,Aug2010.
BMVC2009,London,UK,September7-10,2009.Proceed-
[26] A.Newell, K.Yang, andJ.Deng. StackedHourglassNet-
ings,pages1–11,2009.
worksforHumanPoseEstimation,pages483–499.Springer,
[11] M. Eichner, M. Marin-Jimenez, A. Zisserman, and V. Fer-
2016.
rari. 2d articulated human pose estimation and retrieval in
[27] L.Pishchulin, E.Insafutdinov, S.Tang,B.Andres, M.An-
(almost)unconstrainedstillimages.InternationalJournalof
driluka, P. Gehler, and B. Schiele. DeepCut: Joint subset
ComputerVision,99(2):190–214,2012.
partition and labeling for multi person pose estimation. In
[12] P. F. Felzenszwalb and D. P. Huttenlocher. Pictorial struc- CVPR,2016.
turesforobjectrecognition. InternationalJournalofCom- [28] B. Sapp, D. Weiss, and B. Taskar. Parsing human motion
puterVision,61(1):55–79,2005. withstretchablemodels. InCVPR,pages1281–1288.IEEE,
[13] M.A.FischlerandR.A.Elschlager. Therepresentationand 2011.
matchingofpictorialstructures.IEEETransactionsonCom- [29] J. Schmidhuber. Deep learning in neural networks: An
puters,C-22(1):67–92,Jan1973. overview. NeuralNetworks,61:85–117,2015.
[30] J.Shotton, R.Girshick, A.Fitzgibbon, T.Sharp, M.Cook,
M.Finocchio,R.Moore,P.Kohli,A.Criminisi,A.Kipman,
andA.Blake. Efficienthumanposeestimationfromsingle
depthimages. PAMI,35(12):2821–2840,2012.
[31] L. Sigal, A. O. Balan, and M. J. Black. Humaneva: Syn-
chronizedvideoandmotioncapturedatasetandbaselineal-
gorithmforevaluationofarticulatedhumanmotion. Inter-
nationalJournalofComputerVision,87(1):4–27,2009.
[32] M. Sun and S. Savarese. Articulated part-based model for
joint object detection and pose estimation. In Proceedings
of the 2011 International Conference on Computer Vision,
ICCV’11,pages723–730.IEEE,2011.
[33] R. Tokola, W. Choi, and S. Savarese. Breaking the chain:
liberationfromthetemporalmarkovassumptionfortracking
human poses. In Proceedings of the International Confer-
enceonComputerVision,2013.
[34] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bre-
gler. Efficient object localization using convolutional net-
works. In2015IEEEConferenceonComputerVisionand
PatternRecognition(CVPR),pages648–656,June2015.
[35] J.Tompson,A.Jain,Y.LeCun,andC.Bregler.Jointtraining
ofaconvolutionalnetworkandagraphicalmodelforhuman
poseestimation. InZ.Ghahramani,M.Welling,C.Cortes,
N. D. Lawrence, and K. Q. Weinberger, editors, Advances
inNeuralInformationProcessingSystems27,pages1799–
1807.CurranAssociates,Inc.,2014.
[36] A.ToshevandC.Szegedy. DeepPose: Humanposeestima-
tionviadeepneuralnetworks. InCVPR,2014.
[37] Z.TuandX.Bai. Auto-contextanditsapplicationtohigh-
levelvisiontasksand3Dbrainimagesegmentation. IEEE
TransactionsonPatternAnalysisandMachineIntelligence,
32(10):1744–1757,2010.
[38] S.-E.Wei,V.Ramakrishna,T.Kanade,andY.Sheikh. Con-
volutionalposemachines. InCVPR,2016.
[39] W.Xu,P.c.Su,andS.c.S.Cheung.Humanposeestimation
usingtworgb-dsensors. In2016IEEEInternationalCon-
ferenceonImageProcessing(ICIP),pages1279–1283,Sept
2016.
[40] W.Yang,W.Ouyang,H.Li,andX.Wang.End-to-endlearn-
ing of deformable mixture of parts and deep convolutional
neuralnetworksforhumanposeestimation.InCVPR.IEEE,
2016.
[41] Y. Yang and D. Ramanan. Articulated human detection
withflexiblemixturesofparts. IEEETransactionsonPat-
ternAnalysisandMachineIntelligence,35(12):2878–2890,
2013.