Towards Accurate Multi-person Pose Estimation in the Wild

4.1 MB·

by  George Papandreou

Towards Accurate Multi-person Pose Estimation in the Wild
GeorgePapandreou,TylerZhu,NoriKanazawa,AlexanderToshev,
JonathanTompson,ChrisBregler,KevinMurphy
Google,Inc.
[gpapan, tylerzhu, kanazawa, toshev, tompson, bregler, kpmurphy]@google.com
7
1 Abstract Recently, there has been significant progress on this
0 problem, mostly by leveraging deep Convolutional Neural
2
Weproposeamethodformulti-persondetectionand2-D Networks(CNNs)trainedonlargelabeleddatasets[43,26,

n keypointlocalization(humanposeestimation)thatachieves 42, 10, 31, 2, 7, 6, 20, 24, 8]. However, most prior work
a state-of-the-artresultsonthechallengingCOCOkeypoints has focused on the simpler setting of predicting the pose
J
task. It is a simple, yet powerful, top down approach con- of a single person assuming the location and scale of the
6
sistingoftwostages. person is provided in the form of a ground truth bounding

] In the first stage, we predict the location and scale of boxornoisytorsokeypoint,asinthepopularMPII[2]and
V boxes which are likely to contain people; for this we use FLIC[38]datasets.
C theFasterRCNNdetectorwithanInception-ResNetarchi- In this paper, we tackle the more challenging setting of
. tecture. In the second stage, we estimate the keypoints of posedetection”inthewild”,inwhichwearenotprovided
s
c the person potentially contained in each proposed bound- with the ground truth location or scale of the person in-
[ ingbox. Foreachkeypointtypewepredictdenseheatmaps stances. Thisisharderbecauseitcombinestheproblemof

1 andoffsetsusingafullyconvolutionalResNet. Tocombine persondetection(localization)withtheproblemofkeypoint
v these outputs we introduce a novel aggregation procedure detection (landmarking). Furthermore, in crowded scenes,
9 toobtainhighlylocalizedkeypointpredictions. Wealsouse wherepeopleareclosetoeachother,itcanbequitedifficult
7 anovelformofkeypoint-basedNon-Maximum-Suppression tosolvetheassociationproblemofdeterminingwhichbody
7
(NMS), instead of the cruder box-level NMS, and a novel partbelongstowhichperson.
1
formofkeypoint-basedconfidencescoreestimation,instead
0 TherecentlyreleasedCOCOpersonkeypointsdetection
1. ofbox-levelscoring. dataset and associated challenge [29] provide an excellent
Ourfinalsystemachievesaverageprecisionof0.636on
0 vehicle to encourage research, establish metrics, and mea-
theCOCOtest-devsetandthe0.628test-standardsets,out-
7 sure progress on this task. It extends the COCO dataset
1 performingtheCMU-Posewinnerofthe2016COCOkey-
[30] with additional annotations of 17 keypoints (12 body
: pointschallenge. Further,byusingadditionallabeleddata
v joints and 5 face landmarks) for every medium and large
i weobtainanevenhigheraverageprecisionof0.668onthe sized person in each image. A large number of persons in
X
test-devsetand0.658onthetest-standardset,thusachiev-
the dataset are only partially visible. The degree of match
r ingaroughly10%improvementoverthepreviousbestper-
a betweengroundtruthandpredictedposesintheCOCOkey-
formingmethodonthesamechallenge.
pointstaskismeasuredintermsofobjectkeypointsimilar-
ity(OKS),whichrangesfrom0(poormatch)to1(perfect
match). Theoverallqualityofthecombinedpersondetec-
1.Introduction
tion and pose estimation system in the benchmark is mea-
Inthequestforcomprehensiveimageunderstanding,vi- sured in terms of an OKS-induced average precision (AP)
sualinterpretationofpeopleplaysacentralrole.Wewantto metric. In this paper, we describe a system that achieves
localize people, understand the activities they are involved state-of-the-artresultsonthischallengingtask.
in, understand how people move for the purpose of Vir- There are two broad approaches for tackling the multi-
tual/Augmented Reality, and learn from them to teach au- personposeestimationproblem: bottom-up,inwhichkey-
tonomoussystems. Amajorcornerstoneinachievingthese pointproposalsaregroupedtogetherintopersoninstances,
goals is the problem of human pose estimation, defined as and top-down, in which a pose estimator is applied to the
2-Dlocalizationofhumanjointsonthearms,legs,andkey- output of a bounding-box person detector. Recent work
pointsontorsoandtheface. [33, 24, 8, 23] has advocated the bottom-up approach; in
1
their experiments, their proposed bottom-up methods out- of-art performance on pose estimation is achieved using
performedthetop-downbaselinestheycomparedwith. CNNs [43, 26, 42, 10, 31, 2, 7, 6, 20, 24, 8]. The prob-
In contrast, in this work we revisit the top-down ap- lemcanbeencodedassimpleregression,asdonebyToshev
proach and show that it can be surprisingly effective. The andSzegedy[43],usingacascadeofdetectorsfortop-down
proposed system is a two stage pipeline with state-of-art poserefinementfromcroppedinputpatches. Alternatively,
constituent components carefully adapted to our task. In Jainetal.[26]trainedaCNNonimagepatches,whichwas
the first stage, we predict the location and scale of boxes appliedconvolutionallyatinferencetimetoinferheatmaps
which are likely to contain people. For this we use the (or activity-maps) for each keypoint independently. In ad-
Faster-RCNN method [35] on top of an Inception-Resnet dition, they used a “DPM-like” graphical-model post pro-
CNN[40]. Inthesecondstage,wepredictthelocationsof cessingsteptofilterheatmappotentialsandtoimposeinter-
each keypoint for each of the proposed person boxes. For jointconsistency. Followingthiswork,Tompsonetal.[42]
this we use a Resnet [21] applied in a fully convolutional used a multi-scale fully-convolutional architecture trained
fashion to predict activation heatmaps and offsets for each on whole images (rather than image crops) to infer the
keypoint,similartotheworksofPishchulinetal.[33]and heatmap potentials, and they reformulated the graphical
Insafutdinov et al. [24], followed by combining their pre- model from [26] - simplifying the tree structure to a star-
dictionsusinganovelformofheatmap-offsetaggregation. graph and re-writing the belief propagation messages - so
We avoid duplicate pose detections by means of a novel thattheentiresystemcouldbetrainedend-to-end.
keypoint-basedNon-Maximum-Suppression(NMS)mech- Chen et al. [10] added image-dependant priors to im-
anism building directly on the OKS metric (which we call proveCNNperformance. Bylearningalower-dimensional
OKS-NMS),insteadofthecruderbox-levelIOUNMS.We imagerepresentation,theyclusteredtheinputimageintoa
alsoproposeanovelkeypoint-basedconfidencescoreesti- mixtureofconfigurationsofeachpairofconsecutivejoints.
mator, whichwe show leads togreatly improved AP com- Dependingonwhichmixtureisactiveforagiveninputim-
paredtousingtheFaster-RCNNboxscoresforrankingour age, a separate pairwise displacement prior was used for
finalposeproposals.Ourfinalsystemachievesaveragepre- graphical model inference, resulting in stronger pairwise
cisionof0.636ontheCOCOtest-devsetandthe0.628test- priorsandimprovedoverallperformance.
standardsets,outperformingthewinnerofthe2016COCO Bulatetal.[7]useacascadednetworktoexplicitlyinfer
keypoints challenge [8], which gets 0.618 on the COCO partrelationshipstoimproveinter-jointconsistency,which
test-dev and 0.611 on the test-standard set. Further, if we the authors claim effectively encodes part constraints and
useadditionallylabeleddataweobtainanevenhigheraver- inter-jointcontext. Similarly,Belagiannis&Zisserman[6]
ageprecisionof0.668onthetest-devsetand0.658onthe alsoproposeacascadedarchitecturetoinferpairwisejoint
test-standard set, thus achieving a roughly 10% improve- (or part) locations, which is then used to iteratively refine
mentoverthenextbestapproachonthesamechallenge. unaryjointpredictions,whereunlike[7],theyproposeiter-
Intherestofthepaper,wediscussrelatedwork,andthen ativerefinementusingarecursiveneuralnetwork.
describe our method in more detail. We then perform an
Inspired by recent work in sequence-to-sequence mod-
experimental study, comparing our system to recent state-
eling, Gkioxari et al. [20] propose a novel network struc-
of-the-art,andwemeasuretheeffectsofthedifferentparts
ture where body part locations are predicted sequentially
ofoursystemontheAPmetric.
rather than independently, as per traditional feed-forward
networks. Body part locations are conditioned on the in-
2.RelatedWork
put image and all other predicted parts, yielding a model
For most of its history, the research in human pose es- which promotes sequential reasoning and learns complex
timation has been heavily based on the idea of part-based inter-jointrelationships.
models, as pioneered by the Pictorial Structures model of The state-of-the-art approach for single-person pose on
Fischer and Elschlager [16]. One of the first practical the MPII human pose [2] and FLIC [38] datasets is the
and well performing methods based on this idea is De- CNN model of Newell et al. [31]. They propose a novel
formable Part Model (DPM) by Felzenswalb et al. [15], CNN architecture that uses skip-connections to promote
whichspurredalargebodyofworkonprobabilisticgraphi- multi-scalefeaturelearning, aswellasarepeatedpooling-
calmodels(PGM)for2-Dhumanposeinference[3,12,37, upsampling(“hourglass”)structurethatresultsinimproved
45, 11, 27, 32, 38, 18]. The majority of these methods fo- iterative pose refinement. They claim that their network is
cusondevelopingtractableinferenceproceduresforhighly abletomoreefficientlylearnvariousspatialrelationshipas-
articulated models, while at the same time capturing rich sociatedwiththebody,evenoverlargepixeldisplacements,
dependenciesamongbodypartsandproperties. andwithasmallnumberoftotalnetworkparameters.
Single-PersonPoseWiththedevelopmentofDeepCon- Top-Down Multi-Person Pose The problem of multi-
volutional Neural Networks (CNN) for vision tasks, state- person pose estimation presents different challenges, un-
2
adressed by the above work. Most of the approaches for belandlocationinasecondstage. Wecanthinkofthefirst
multi-personposeaimatassociatinghumanpartdetections stage of our method as a proposal mechanism, however of
withhumaninstances. Thetopdownwaytoestablishthese only one type of object – person. Our second stage serves
associations,whichisclosesttoourapproach,istofirstper- asarefinementwherewe(i)gobeyondboundingboxesand
formpersondetectionfollowedbyposeestimation. Forex- predictkeypointsand(ii)rescorethedetectionbasedonthe
ample,Pishchulinetal.[34]followthisparadigmbyusing estimatedkeypoints. Forcomputationalefficiency,weonly
PS-basedposeestimation. Amorerobusttoocclusionsper- forwardtothesecondstagepersonboxdetectionproposals
son detector, modelled after poselets, is used by Gkioxari with score higher than 0.3, resulting in only 3.5 proposals
etal.[19]. Further,YangandRamanan[45]fusedetection perimageonaverage.Inthefollowing,wedescribeinmore
andposeinonemodelbyusingaPSmodel. Theinference detailthetwostagesofoursystem.
procedureallowsforposeestimationofmultipleinstancein
3.1.PersonBoxDetection
oneimageanalogoustoPS-basedobjectdetection. Asim-
ilar multi-person PS with an additional explicit occlusion
OurpersondetectorisaFaster-RCNNsystem[35],with
modelingisproposedbyEichnerandFerrari[13].
theimportantmodificationthatweuseResNet-Inceptionar-
Onarelatednote,2-Dpersondetectionisusedasafirst
chitecture [40]. This architecture combines Inception lay-
stepinseveral3Dposeestimationworks[39,4,5].
ers[41]withresidualconnections[21]andperformsbetter
Bottom-UpMulti-PersonPoseAdifferentlineofwork
thanbothofitscomponents.Thedetectorisinitiallytrained
istodetectbodypartsinsteadoffullpersons,andtosubse-
onall80categoriesintheCOCOdataset. Itisfurthermod-
quently associate these parts to human instances, thus per-
ifiedsoitonlypredictsthelocationofoneclass–‘person’.
forming pose estimation in a bottom up fashion. Such ap-
We fine-tune the modified model on data which contains
proaches employ part detectors and differ in how associ-
only person bounding boxes. Our system is based on the
ationsamongpartsareexpressed,andtheinferenceproce-
Faster-RCNNimplementationof[22].
dureusedtoobtainfullpartgroupingsintopersoninstances.
To achieve even better performance, we also consider
Pishchulin et al. [33] and later Insafutdinov et al. [24, 23]
a variant in which the person detector is an ensemble of
formulate the problem of pose estimation as part grouping
two Faster-RCNN models, one based on [40] as described
andlabelingviaaLinearProgram. Asimilarformulationis
above, andonebasedonResNet[21]forhighermodeldi-
proposedbyIqbaletal.[25].
versity. Tocombinethese, wefirstruntheregionproposal
Aprobabilisticapproachtopartgroupingandlabelingis
netforbothmodels,andthenapplytheboxrefinementand
proposed also by Ladickly et al. [28]. The authors model
scoring nets for both models on all proposals. We further
human part to human instance associations in addition to
improve performance by applying the ensembled detector
pixel-wise human and human part segmentation. They
to the whole image, as well as to 9 crops, arranged in a
leverageaHOG-basedsystemforpartdetections.
3x3 grid; this helps detect small people. We discuss the
Caoetal.[8]obtainedstate-of-the-artresultsonthe2016
improvementsobtainedbytheseextensionsinSection4.
COCO person keypoints challenge [30] by combining a
variation of the unary joint detector architecture from [44] 3.2.PersonPoseEstimation
with a part affinity field regression to enforce inter-joint
The pose estimation component of our system predicts
consistency. They propose a greedy algorithm to gener-
the location of all K = 17 person keypoints, given each
atepersoninstanceproposalsinabottom-upfashion. Their
personboundingboxproposaldeliveredbythefirststage.
best results are obtained in an additional top-down refine-
Oneapproachwouldbetouseasingleregressorperkey-
ment process in which they run a standard single-person
point,asin[43],butthisisproblematicwhenthereismore
pose estimator [44] on the person instance box proposals
than one person in the image patch (in which case a key-
generatedbythebottom-upstage.
point can occur in multiple places). A different approach
addressing this issue would be to predict activation maps,
3.Methods
asin[26],whichallowformultiplepredictionsofthesame
Our multi-person pose estimation system is a two step keypoint.However,thesizeoftheactivationmaps,andthus
cascade, as illustrated in Figure 1. In the first stage, we thelocalizationprecision,islimitedbythesizeofthenet’s
use a person detector to produce a bounding box around outputfeaturemaps,whichisafractionoftheinputimage
eachpersoninstance. Inthesecondstage,weapplyapose size,duetotheuseofmax-poolingwithdecimation.
estimatortotheimagecropextractedaroundeachdetected In order to address the above limitations, we adopt a
personinstanceinordertolocalizeitskeypoints. combinedclassificationandregressionapproach. Foreach
Ourapproachisinspiredbyrecentstate-of-artobjectde- spatial position, we first classify whether it is in the vicin-
tection systems such as [17, 41], which propose objects in ity of each of the K keypoints or not (which we call a
a class agnostic fashion as a first stage and refine their la- “heatmap”), then predict a 2-D local offset vector to get a
3
Faster R-CNN on ResNet-Inception ResNet

Expand box and crop
Stage 1: Person detection Stage 2: Pose estimation
Figure1: Overviewofourtwostagecascademodel. Confidential & Proprietary
more precise estimate of the corresponding keypoint loca- genet pretrained ResNet-101 model of [21], replacing its
tion. Notethatthisapproachisinspiredbyworkonobject last layer with 1x1 convolution with 3K outputs. We fol-
detection,whereasimilarsetupisusedtopredictbounding low the approach of [9]: we employ atrous convolution to
boxes, e.g. [14,35]. Figure2illustratesthesethreeoutput generatethe3Kpredictionswithanoutputstrideof8pixels
channelsperkeypoint. andbilinearlyupsamplethemtothe353x257cropsize.
In more detail, given the image crop, let f (x ) = 1 if
k i
thek-thkeypointislocatedatpositionx and0otherwise.
i
Herek ∈{1,...,K}isindexingthekeypointtypeandi∈
{1,...,N}isindexingthepixellocationsonthe353x257
image crop grid. Training a CNN to produce directly the
highlylocalizedactivationsf (ideallydeltafunctions)ona
k
fineresolutionspatialgridishard.
Instead, we decompose the problem into two stages.
First, for each position x and each keypoint k, we com-
i
putetheprobabilityh (x ) = 1if||x −l || ≤ Rthatthe
k i i k
pointx iswithinadiskofradiusRfromthelocationl of
i k
thek-thkeypoint. WegenerateK suchheatmaps,solvinga
Figure 2: Network input and ground-truth outputs. Left &
binaryclassificationproblemforeachpositionandkeypoint
Middle: left-elbowheatmapoverlay(redindicatesheatmap
independently.
of1). Right: offsetL2magnitude(2-Doffsetvectorshown
Inadditiontotheheatmaps, wealsopredictateachpo-
inred).
sitioniandeachkeypointkthe2-DoffsetvectorF (x )=
k i
l −x from the pixel to the corresponding keypoint. We
k i
ImageCroppingWefirstmakeallboxeshavethesame
generate K such vector fields, solving a 2-D regression
fixedaspectratio(height/width=1.37)byextendingeither
problemforeachpositionandkeypointindependently.
the height or the width of the boxes returned by the per-
Aftergeneratingtheheatmapsandoffsets,weaggregate
sondetectorwithoutdistortingtheimageaspectratio.After
themtoproducehighlylocalizedactivationmapsf (x )as
k i
that,wefurtherenlargetheboxestoincludeadditionalim-
follows:
agecontext: weusearescalingfactorequalto1.25during
evaluation and a random rescaling factor between 1.0 and (cid:88) 1
f (x )= G(x +F (x )−x )h (x ), (1)
1.5 during training (for data augmentation). We then crop k i πR2 j k j i k j
j
fromtheresultingboxtheimageandresizetoafixedcrop
ofheight353andwidth257pixels. where G(·) is the bilinear interpolation kernel. This is a
Heatmap and Offset Prediction with CNN We apply formofvotingprocess: eachpointj intheimagecropgrid
a ResNet with 101 layers [21] on the cropped image in a castsavotewithitsestimateforthepositionofeverykey-
fullyconvolutionalfashiontoproduceheatmaps(onechan- point, withthevotebeingweightedbytheprobabilitythat
nelperkeypoint)andoffsets(twochannelsperkeypointfor itisinthediskofinfluenceofthecorrespondingkeypoint.
thex-andy-directions),foratotalof3K outputchannels. The normalizing factor equals the area of the disk and en-
We initialize our model from the publicly available Ima- sures that if the heatmaps and offsets were perfect, then
4
Heatmap Thefinallossfunctionhastheform
L(θ)=λ L (θ)+λ L (θ), (3)
h h o o
whereλ = 4andλ = 1isascalarfactortobalancethe
h o
lossfunctionterms. Wesumthislossoveralltheimagesin
aminibatch,andthenapplystochasticgradientdescent.
CNN An important consideration in model training is how to
treat cases where multiple people exist in the image crop
in the computation of heatmap loss. When computing the
heatmap loss at the intermediate layer, we exclude contri-
Fused activation butionsfromwithinthedisksaroundthekeypointsofback-
maps
ground people. When computing the heatmap loss at the
final layer, we treat as positives only the disks around the
Offset
keypointsoftheforegroundpersonandasnegativesevery-
Figure3: Ourfullyconvolutionalnetworkpredictstwotar- thing else, forcing the model to predict correctly the key-
gets: (1) Disk-shaped heatmaps around each keypoint and pointsofthepersoninthecenterofthebox.
(2)magnitudeoftheoffsetfieldstowardstheexactkeypoint
position within the disk. Aggregating them in a weighted
PoseRescoring Attesttime,weapplythemodeltoeach
voting process results in highly localized activation maps.
imagecrop.Ratherthanjustrelyingontheconfidencefrom
The figure shows the heatmaps and the pointwise magni-
the person detector, we compute a refined confidence esti-
tude of the offset field on a validation image. Note that in
mate,whichtakesintoaccounttheconfidenceofeachkey-
thisillustrationwesuper-imposethechannelsfromthedif-
point. In particular, we maximize over locations and av-
ferentkeypoints.
erageoverkeypoints,yieldingourfinalinstance-levelpose
detectionscore:
fk(xi)wouldbeaunit-massdeltafunctioncenteredatthe 1 (cid:88)K
positionofthek-thkeypoint. score(I)= maxf (x ) (4)
The process is illustrated in Figure 3. We see that pre- K xi k i
k=1
dicting separate heatmap and offset channels and fusing
We have found that ranking our system’s pose estimation
thembytheproposedvotingprocessresultsintohighlylo-
proposals using 4 significantly improves AP compared to
calized activation maps which precisely pinpoint the posi-
usingthescoredeliveredbytheFaster-RCNNboxdetector.
tionofthekeypoints.
Model Training We use a single ResNet model with
4.ExperimentalEvaluation
two convolutional output heads. The output of the first
headpassesthroughasigmoidfunctiontoyieldtheheatmap
4.1.ExperimentalSetup
probabilitiesh (x )foreachpositionx andeachkeypoint
k i i
k. The training target h¯ (x ) is a map of zeros and ones, The person detection and pose inference models from
k i
with h¯ (x ) = 1 if ||x −l || ≤ R and 0 otherwise. The Sections3.1and3.2respectively, areimplementedinTen-
k i i k
corresponding loss function L (θ) is the sum of logistic sorflow[1]. Weusedistributedtrainingacrossseveralma-
h
losses for each position and keypoint separately. To ac- chinesequippedwithTeslaK40GPUs. Forthepersonde-
celeratetraining,wefollow[24]andaddanextraheatmap tectortrainingweuse9GPUs. WeoptimizewithSGDwith
prediction layer at intermediate layer 50 of ResNet, which momentum(momentumissetto0.9). Thestartinglearning
contributesacorrespondingauxiliarylossterm. rate is 0.001 and is decreased by one factor every 400000
For training the offset regression head, we penalize the steps. We stop training after 1.5 million steps, which hap-
difference between the predicted and ground truth offsets. pensafterapproximately3days. Forfinetuningonperson
Thecorrespondinglossis data we use the same setup, however we usually train for
lessthan1millionsteps.
(cid:88) (cid:88)
L (θ)= H(||F (x )−(l −x )||), (2) For the pose estimator we use the same optimizer with
o k i k i
the same optimization parameters and a batch size of 24.
k=1:Ki:||lk−xi||≤R
Training converges after 800000 steps which corresponds
whereH(u)istheHuberrobustloss,l isthepositionofthe toonedayoftraining.
k
k-th keypoint, and we only compute the loss for positions Allournetworksarepre-trainedontheImagenetclassi-
x withinadiskofradiusRfromeachkeypoint[35]. ficationdataset[36]. Tofine-tuneourposenetworkweuse
i
5
Figure 4: Detection and pose estimation results using our system on a random selection from the COCO test-dev set. For
each detected person, we display the detected bounding box together with the estimated keypoints. All detections for one
person are colored the same way. It is worth noting that our system works in really cluttered scenes (third row, rightmost
andlastrow,right);itdealswellwithocclusions(lastrow,left)andhallucinatesoccludedjoints. Lastbutnotleast,someof
thefalsepositivedetectionsareinrealitycorrectastheyrepresentpicturesofpeople(firstrow,middle)ortoys(fourthrow,
middle). Figurebestviewedzoomedinonamonitor.
6
two dataset variants; one that uses only external data (ext- Table 3: Performance (AP) on COCO test-dev: with and
only), and one that appends to this dataset samples from withoutourproposedOKS-basedNMS,usingext-onlyand
an internal dataset (int+ext). For the ext-only dataset we int+extdataandtheensembleperson-detector.
combine training samples from the MPII human pose [2]
NMSThreshold AP:ext-onlydata AP:int+extdata
(24,987 images with 40,614 person instances) and MS
0.1 0.615 0.646
COCO pose challenge [30] (45,174 images with 105,698
0.2 0.63 0.663
personinstances)datasets. Ourint+ext datasetappendsan 0.3 0.634 0.668
additional 73,024 images randomly selected from Flickr, 0.4 0.636 0.668
0.5 0.635 0.667
with 227,029 person instances labeled using an annotation
0.6 0.635 0.667
strategy similar to that described by Lin et al. [29]. These
0.7 0.632 0.666
additional training samples are verified to have no overlap 0.8 0.626 0.659
withtheMSCOCOtraining,validationortestsets. 0.9 0.617 0.65
NoNMS 0.601 0.628
Our pose-detector model is based on the ResNet-
101[21]imageclassificationarchitecture. ResNetisavery
deep neural network, which contains residual connections
overlap using intersection over union (IoU) of the boxes.
leading to state of art performance on a variety of vision
Weproposeamorerefinedvariant,thattakesthekeypoints
tasks[21]. Wepresentresultsofthisposenetworktrained
into account. In particular, we measure the OKS score
onboththeext-onlyandint+extdatasetvariants.
between corresponding keypoints for two candidate detec-
We present results using two person-detector architec-
tions,andaveragethisoverallkeypoints. Wethenusethis
turesasdetailedinSection3.1:onecomprisedofanexpen-
as a measure of multiple detections. Applying this OKS-
sive ensemble of person-detector networks, with a multi-
based NMS works well when we adjust the person detec-
crop and multi-scale sampling strategy (ensemble model),
torcandidateboxthresholdtobiastowardsfalsepositives.
andonethatusesasignificantlymoreefficientsingle-model
If we aggressively suppress candidates from the person-
detectornetwork,withsingle-cropsampling(singlemodel).
detector, the final OKS score is lower. We attribute this to
Wetrainboththeperson-detectormodelvariants(ensemble
theposenetwork’simprovedabilitytoreasonabouttheper-
andsinglemodels)ontheCOCOdetectiondatasetonly(our
son instance score as a sum of keypoint likelihood scores,
ext-onlysetwithoutMPIIsamples).
andsofalsepositivesfromtheperson-detectorcanbeeasily
removedifnokeypointsarepresentinthebounding-box,or
4.2.COCOKeypointsDetectionState-of-the-Art
ifthereissignificantoverlapinthejointannotationsofspu-
Table1showstheCOCOkeypointtest-devsplitperfor- riouspersondetections.
manceofourtwomodelandtwodatasetvariants,resulting We evaluate our proposed NMS algorithm using our
in 4 experiments in total; our pose detector trained on ext- ensemble person-detector, on both ext-only and int+ext
onlyandint+extdatasets,aswellastheensembleandsingle datasets,andwithmultipleNMSthresholdvalues. There-
person-detector networks. A random selection of test-dev sults are shown in Table 3. Using ext-only data, NMS im-
inferencesamplesareshowninFigure4(usingtheensem- provesAPby0.035atathresholdof0.4,whichalonebeats
bleperson-detectorandint+extpose-detectornetworks). thepreviouslyreportedstate-of-the-artonMSCOCO.Us-
Table 2 shows the COCO keypoint test-standard split ing int+ext data, NMS results in a significant 0.04 AP im-
results of our ensemble person-detector model with pose- provement.
detector trained on both dataset variants (ext-only and
4.4.Multi-scaleExperiments
int+ext).
All our model and dataset variants achieve state-of-the-
Animportanthyper-parameterinourcascadedsystema
art results on the MS COCO test-dev and test-standard scalingfactor,s,whichdeterminesthecroppedimagecon-
split. Ourbestmodelusestheexpensiveensembleperson-
textforinputtoourposedetectornetwork. Empirically,we
detector trained on ext-only data and our pose-detector
find a strong trade-off between pose detector input resolu-
trained on int+ext data, for an AP score of 0.668 on test-
tion and context, where too little resolution leads to inad-
dev. Usingext-onlydataforbothmodelsweachieve0.636
equatehigh-frequencytexturalinformation,whiletoolittle
on test-dev. The previous test-dev state-of-the-art is 0.618
context makes inferring body geometry difficult. During
fromtheCMU-Poseteam[8].
evaluation, we artificially enlarge the person-detector net-
work’sbounding-boxinstancesbyscalefactors,whichin-
4.3.OKS-basednonmaximumsuppression
troducesadditionalimagecontextaroundthesubjectofin-
Following standard practice, we use non maximal sup- terest. Theresultantimagecropisthenre-scaledtoafixed
pression (NMS) to eliminate multiple detections in the resolutionof353x257beforeinputtotheposedetectornet-
person-detector stage. The standard approach measures work. Table 4 shows our cascaded detector‘s AP perfor-
7
Table1: PerformanceonCOCOkeypointtest-devsplit.
AP AP.5 AP.75 AP(M) AP(L) AR AR.5 AR.75 AR(M) AR(L)
CMU-Pose[8]: 0.618 0.849 0.675 0.571 0.682 0.665 0.872 0.718 0.606 0.746
Thiswork:ext-only&ensemble 0.636 0.852 0.7 0.596 0.702 0.685 0.881 0.741 0.635 0.754
Thiswork:ext-only&single 0.624 0.840 0.685 0.591 0.681 0.667 0.866 0.720 0.608 0.749
Thiswork:int+ext&ensemble 0.668 0.863 0.734 0.630 0.733 0.716 0.896 0.776 0.669 0.782
Thiswork:int+ext&single 0.661 0.860 0.729 0.63 0.715 0.703 0.881 0.762 0.647 0.781
Table2: PerformanceonCOCOkeypointtest-standardsplit.
AP AP.5 AP.75 AP(M) AP(L) AR AR.5 AR.75 AR(M) AR(L)
CMU-Pose[8]: 0.611 0.844 0.667 0.558 0.684 0.665 0.872 0.718 0.602 0.749
Thiswork:ext-only&ensemble 0.628 0.840 0.689 0.599 0.682 0.685 0.880 0.741 0.631 0.758
Thiswork:int+ext&ensemble 0.658 0.851 0.723 0.629 0.713 0.717 0.895 0.778 0.662 0.792
Table4: PerformanceonCOCOtest-dev: varyingposede- Gursheesh Kour for managing our internal image annota-
tectorscale. Usingsinglenetworkpersondetection. tioneffort.
scalefactor(s) AP:int+extdata References
1.00 0.635
1.25 0.663 [1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,
1.50 0.657
C.Citro,G.S.Corrado,A.Davis,J.Dean,M.Devin,S.Ghe-
1.75 0.635
mawat,I.Goodfellow,A.Harp,G.Irving,M.Isard,Y.Jia,
R.Jozefowicz,L.Kaiser,M.Kudlur,J.Levenberg,D.Mane´,
R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster,
mancewhenvaryingscaleparameter,s.
J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,
Table4showsthatpersonscaleanddetectorcontextplay
V.Vanhoucke,V.Vasudevan,F.Vie´gas,O.Vinyals,P.War-
a important role in AP performance. Given a cropped in- den,M.Wattenberg,M.Wicke,Y.Yu,andX.Zheng.Tensor-
put,whereasinglesubjectiscenteredinthemiddleofthe Flow: Large-scalemachinelearningonheterogeneoussys-
croppedcontext, theposenetworkmustlearnaninference tems,2015. Softwareavailablefromtensorflow.org. 5
modelthat“chooses”thejointsofthecorrectsubject. With [2] M.Andriluka, L.Pishchulin, P.Gehler, andB.Schiele. 2d
toolittleortoomuchcontext,thistaskismadedifficult. In humanposeestimation: Newbenchmarkandstateoftheart
addition, increasing s in our formulation increases context analysis. InCVPR,2014. 1,2,7,10
attheexpenseofinputresolution,whichcanalsonegatively [3] M.Andriluka, S.Roth, andB.Schiele. Pictorialstructures
effectAPperformancewhenthecropscaleistoohigh. revisited: Peopledetectionandarticulatedposeestimation.
InCVPR,2009. 2
[4] V. Belagiannis, S. Amin, M. Andriluka, B. Schiele,
5.Conclusion
N. Navab, and S. Ilic. 3d pictorial structures for multiple
humanposeestimation. InCVPR,pages1669–1676,2014.
Inthisworkweaddresstheproblemofpersondetection
3
and pose estimation in cluttered images in the wild. We
[5] V. Belagiannis, S. Amin, M. Andriluka, B. Schiele,
present a simple two stage system, consisting of a person
N.Navab,andS.Ilic. 3dpictorialstructuresrevisited: Mul-
detectionstagefollowedbyakeypointestimationstagefor
tiplehumanposeestimation. InCVPR,2015. 3
each person. Despite its simplicity it achieves state-of-art
[6] V. Belagiannis and A. Zisserman. Recurrent human pose
resultsasmeasuredonthechallengingCOCObenchmark.
estimation. Inarxiv,2016. 1,2
In the future, we plan to investigate systems, that share [7] A.BulatandG.Tzimiropoulos. Humanposeestimationvia
featuresbetweemthedetectorandkeypointestimationand convolutionalpartheatmapregression. InECCV,2016. 1,2
canbetrainedend-to-end. Inthiswaywecouldhopefully [8] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime
boostourperformancefurtherwhilereducingcomputation multi-person 2d pose estimation using part affinity fields.
atinferencetime. arXiv:1611.08050,2016. 1,2,3,7,8
[9] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
Acknowledgments A. L.Yuille. Deeplab: Semantic imagesegmentation with
deep convolutional nets, atrous convolution, and fully con-
We are grateful to the authors of [22] for making their nectedcrfs. arXiv:1606.00915,2016. 4
excellentFaster-RCNNimplementationavailabletous. We [10] X. Chen and A. Yuille. Articulated pose estimation by a
would like to thank Hartwig Adam for supporting this graphicalmodelwithimagedependentpairwiserelations.In
project at its various stages, as well as Akshay Gogia and NIPS,2014. 1,2
8
[11] M. Dantone, J. Gall, C. Leistner, and L. V. Gool. Human [30] T.-Y.Lin,M.Maire,S.Belongie,J.Hays,P.Perona,D.Ra-
poseestimationusingbodypartsdependentjointregressors. manan,P.Dolla´r,andC.L.Zitnick. Microsoftcoco: Com-
InCVPR,2013. 2 monobjectsincontext. InECCV,pages740–755.Springer,
[12] M. Eichner and V. Ferrari. Better appearance models for 2014. 1,3,7
pictorialstructures. InBMVC,2009. 2 [31] A. Newell, K. Yang, and J. Deng. Stacked hourglass net-
works for human pose estimation. In ECCV, 2016. 1, 2,
[13] M.EichnerandV.Ferrari.Wearefamily:Jointposeestima-
tionofmultiplepersons.InECCV,pages228–242.Springer, 10
2010. 3 [32] L.Pishchulin,M.Andriluka,P.Gehler,andB.Schiele.Pose-
letconditionedpictorialstructures. InCVPR,2013. 2
[14] D.Erhan,C.Szegedy,A.Toshev,andD.Anguelov.Scalable
[33] L.Pishchulin, E.Insafutdinov, S.Tang,B.Andres, M.An-
objectdetectionusingdeepneuralnetworks.InCVPR,pages
driluka, P. Gehler, and B. Schiele. Deepcut: Joint subset
2147–2154,2014. 4
partition and labeling for multi person pose estimation. In
[15] P. Felzenszwalb, D. McAllester, and D. Ramanan. A dis-
CVPR,2016. 1,2,3
criminativelytrained,multiscale,deformablepartmodel. In
[34] L. Pishchulin, A. Jain, M. Andriluka, T. Thorma¨hlen, and
CVPR,2008. 2
B.Schiele.Articulatedpeopledetectionandposeestimation:
[16] M. A. Fischler and R. Elschlager. The representation and
Reshapingthefuture. InCVPR,pages3178–3185,2012. 3
matchingofpictorialstructures. InIEEETOC,1973. 2
[35] S.Ren,K.He,R.Girshick,andJ.Sun. FasterR-CNN:To-
[17] R.Girshick,J.Donahue,T.Darrell,andJ.Malik. Richfea-
wardsReal-Timeobjectdetectionwithregionproposalnet-
ture hierarchies for accurate object detection and semantic
works. InNIPS,2015. 2,3,4,5
segmentation. InCVPR,pages580–587,2014. 3
[36] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
[18] G.Gkioxari,P.Arbelaez,L.Bourdev,andJ.Malik. Articu- S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
latedposeestimationusingdiscriminativearmletclassifiers. A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual
InCVPR,2013. 2 RecognitionChallenge. IJCV,115(3):211–252,2015. 5
[19] G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik. Us- [37] B.Sapp,C.Jordan,andB.Taskar. Adaptiveposepriorsfor
ingk-poseletsfordetectingpeopleandlocalizingtheirkey- pictorialstructures. InCVPR,2010. 2
points. InCVPR,pages3582–3589,2014. 3 [38] B.SappandB.Taskar. Modec: Multimodaldecomposable
[20] G.Gkioxari,A.Toshev,andN.Jaitly. Chainedpredictions modelsforhumanposeestimation. InCVPR,2013. 1,2
usingconvolutionalneuralnetworks. InECCV,2016. 1,2 [39] M. Sun and S. Savarese. Articulated part-based model for
[21] K.He,X.Zhang,S.Ren,andJ.Sun. Deepresiduallearning jointobjectdetectionandposeestimation. InICCV,pages
forimagerecognition. InCVPR,2016. 2,3,4,7 723–730,2011. 3
[22] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, [40] C. Szegedy, S. Ioffe, and V. Vanhoucke. Inception-v4,
A.Fathi,I.Fischer,Z.Wojna,Y.Song,S.Guadarrama,etal. inception-resnet and the impact of residual connections on
Speed/accuracy trade-offs for modern convolutional object learning. arXiv:1602.07261,2016. 2,3
detectors. arXiv:1611.10012,2016. 3,8 [41] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
[23] E.Insafutdinov,M.Andriluka,L.Pishchulin,S.Tang,B.An- D.Anguelov, D.Erhan, V.Vanhoucke, andA.Rabinovich.
dres,andB.Schiele.Articulatedmulti-persontrackinginthe Goingdeeperwithconvolutions.InCVPR,pages1–9,2015.
wild. arXiv:1612.01465,2016. 1,3 3
[42] J.Tompson,A.Jain,Y.LeCun,andC.Bregler. Jointraining
[24] E.Insafutdinov,L.Pishchulin,B.Andres,M.Andriluka,and
ofaconvolutionalnetworkandagraphicalmodelforhuman
B.Schiele. Deepercut: Adeeper,stronger,andfastermulti-
poseestimation. InNIPS,2014. 1,2
personposeestimationmodel. InECCV,2016. 1, 2, 3, 5,
10 [43] A.ToshevandC.Szegedy. Deeppose: Humanposeestima-
tionviadeepneuralnetworks. InCVPR,2014. 1,2,3
[25] U.IqbalandJ.Gall. Multi-personposeestimationwithlo-
[44] S.-E.Wei,V.Ramakrishna,T.Kanade,andY.Sheikh. Con-
caljoint-to-personassociations. InECCV,pages627–642.
volutionalposemachines. InarXiv,2016. 3
Springer,2016. 3
[45] Y.YangandD.Ramanan. Articulatedposeestimationwith
[26] A.Jain,J.Tompson,M.Andriluka,G.Taylor,andC.Bregler.
flexiblemixturesofparts. InCVPR,2011. 2,3
Learninghumanposeestimationfeatureswithconvolutional
networks. InICLR,2014. 1,2,3
[27] S. Johnson and M. Everingham. Learning Effective Hu-
manPoseEstimationfromInaccurateAnnotation.InCVPR,
2011. 2
[28] L.Ladicky, P.H.Torr, andA.Zisserman. Humanposees-
timationusingajointpixel-wiseandpart-wiseformulation.
InCVPR,pages3578–3585,2013. 3
[29] T.-Y.Lin, Y.Cui, G.Patterson, M.R.Ronchi, L.Bourdev,
R. Girshick, and P. Dollr. Coco 2016 keypoint challenge.
2016. 1,7
9
SupplementaryExperiments
PoseGivenGroundTruthPersonBoxes
Inordertoidentifyhowmanyerrorsinoursystem’soutputareduetousinganimperfectpersonboxdetector,weconducted
anexperimenttomeasuretheperformanceofoursystemgivengroundtruthboundingboxesonasmallmini-valsetof4,301
images held out from the COCO 2014 validation set (which was not used in training either the box person detector or the
poseestimator). InTable5wecomparetheperformanceofoursystemusingourbestposeestimatortrainedonbothint+ext
inconjunctionwitheitherourbestsingleperson-detectororwiththegroundtruthboxessuppliedbythedataset. Weseethat
usinggroundtruthboxesincreasestheperformanceofoursystemintermsofAPfrom0.681to0.720,showingthatfuture
progressinpersonboxdetectioncanleadtosignificantlyimprovedresultsinourmulti-personposeestimationsetting.
Table5: PerformanceonCOCOkeypointmini-val: Usingdetected(int+ext&singlemodel)vs. groundtruthboxes.
AP AP.5 AP.75 AP(M) AP(L) AR AR.5 AR.75 AR(M) AR(L)
Detectedboxes 0.681 0.854 0.744 0.645 0.743 0.720 0.877 0.778 0.663 0.800
Ground-truthboxes 0.720 0.905 0.792 0.699 0.760 0.751 0.919 0.813 0.711 0.809
EffectofBoxScalingFactor
Table6expandsTable4ofthemainpaper,givingthefullrangeofAPandARresultsonCOCOtest-devwhenvarying
thevalueoftheboxscalingfactorsfrom1.00to1.75duringevaluation.
Table6: PerformanceonCOCOtest-dev,usingsinglenetworkpersondetectorandposeestimatortrainedonint+ext data:
Varyingtheboxscalingfactor.
scalefactor(s) AP AP.5 AP.75 AP(M) AP(L) AR AR.5 AR.75 AR(M) AR(L)
1.00 0.635 0.846 0.694 0.598 0.696 0.680 0.872 0.731 0.623 0.757
1.25 0.663 0.865 0.728 0.636 0.714 0.709 0.890 0.768 0.656 0.782
1.50 0.657 0.865 0.728 0.637 0.702 0.704 0.890 0.768 0.653 0.776
1.75 0.635 0.856 0.711 0.623 0.670 0.684 0.884 0.751 0.635 0.753
PerformanceonMPIISinglePersonPoseEstimationTask
Wehavealsoevaluatedtheperformanceofourposeestimationmodulealoneonthestandardsingle-personMPIIhuman
pose estimation dataset [2]. We have divided the training part of the dataset into a MPII-train split (14,464 images with
23,167 person instances) and a MPII-val split (3,612 images with 5,716 person instances). Training only on this MPII-
train split yields a [email protected] performance of 89.0% on MPII-val. This is comparable to the 88.5% number reported by
[24],whichusesaResNet-101poseestimationnetworkbackbonesimilartoours. PretrainingontheCOCOkeypointsdata
andfine-tuning onMPII-train [email protected] onMPII-valto 91.3%. Pretraining onboth theCOCOkeypoints and
ourinternalposedataandfi[email protected]%. Thepreviousbest
[email protected]%,achievedby[31]usingonlyMPIItrainingdata.Ourresultsaresummarized
inTable7.
Table7: Performanceofourposeestimationmoduleonthesingle-persontask,measuredonMPII-val: Effectofadditional
trainingdata.
trainingdata [email protected]
MPII 89.0
MPII+COCO 91.3
MPII+COCO+internal 92.0
10