Learning from Synthetic Humans

5.1 MB·

by  Gül Varol

Learning from Synthetic Humans
Gu¨lVarol∗† JavierRomero‡ XavierMartin† NaureenMahmood‡
MichaelBlack‡ IvanLaptev∗ CordeliaSchmid†
Abstract
7
1
0 Estimatinghumanpose,shape,andmotionfromimages
2 andvideosarefundamentalchallengeswithmanyapplica-

n tions. Recent advances in 2D human pose estimation use
a largeamountsofmanually-labeledtrainingdataforlearn-
J
ing convolutional neural networks (CNNs). Such data is

5
time consuming to acquire and difficult to extend. More-

over, manuallabelingof3Dpose, depthandmotionisim-
]
V practical. In this work we present SURREAL (Synthetic
C hUmans foR REAL tasks): a new large-scale dataset with
. synthetically-generated but realistic images of people ren-
s
deredfrom3Dsequencesofhumanmotioncapturedata.We
c
[ generate more than 6 million frames together with ground

truthpose,depthmaps,andsegmentationmasks. Weshow
1
that CNNs trained on our synthetic dataset allow for ac-
v
0 curatehumandepthestimationandhumanpartsegmenta-
7 tion in real RGB images. Our results and the new dataset
Figure 1. We generate photo-realistic synthetic images and their
3 openupnewpossibilitiesforadvancingpersonanalysisus-
1 corresponding ground truth for learning pixel-wise classification
ingcheapandlarge-scalesyntheticdata.
0 problems: human parts segmentation and depth estimation. The
. convolutionalneuralnetworktrainedonlyonsyntheticdatagener-
1
1.Introduction alizesonrealimagessufficientlyforbothtasks. Realtestimages
0
inthisfigurearetakenfromMPIIHumanPosedataset[4].
7
Convolutional Neural Networks provide significant
1
: gainstoproblemswithlargeamountsoftrainingdata.Inthe
v highlyusefulformanytasksasitcangeneraterichground
fieldofhumananalysis, recentdatasets[4,32]nowgather
i truthintermsofdepth,motion,body-partsegmentationand
X a sufficient number of annotated images to train networks
occlusionstomentionafew.
r for2Dhumanposeestimation[21,34]. Othertaskssuchas
a In this work we present SURREAL: a new large-scale
accurateestimationofhumanmotion,depthandbody-part
datasetwithsynthetically-generatedbutrealisticimagesof
segmentationarelaggingbehindasmanualsupervisionfor
people. Imagesarerenderedfrom3DsequencesofMoCap
suchproblemsatlargescaleisprohibitivelyexpensive.
data. Toensurerealism,thesyntheticbodiesarecreatedus-
Images of people have rich variation in poses, cloth-
ingtheSMPLbodymodel[18],whoseparametersarefitby
ing, hair styles, body shapes, occlusions, viewpoints, mo-
the MoSh [19] method given raw 3D MoCap marker data.
tionblurandotherfactors. Muchofthesevariations,how-
We randomly sample a large variety of viewpoints, cloth-
ever, can be synthesized using existing 3D motion capture
ingand lighting. SURREALcontains morethan6 million
(MoCap)data[3,16]andmoderntoolsforrealisticrender-
frames together with ground truth pose, depth maps, and
ing.Providedsufficientrealism,suchanapproachwouldbe
segmentation masks. We show that CNNs trained on syn-
∗WILLOWproject-team,De´partementd’Informatiquedel’E´coleNor- thetic data allow for accurate human depth estimation and
maleSupe´rieure,ENS/Inria/CNRSUMR8548,Paris,France. humanpartsegmentationinrealRGBimages,seeFigure1.
†Thoth project-team, Inria Grenoble Rhoˆne-Alpes, Laboratoire Jean
SURREALdatasetwillbecomepubliclyavailable[1].
Kuntzmann,France.
‡MaxPlanckInstituteforIntelligentSystems,Tu¨bingen,Germany. Therestofthispaperisorganizedasfollows. Section2
1
Figure2.Ourpipelineforgeneratingsyntheticdata.A3Dhumanbodymodelisposedusingmotioncapturedataandaframeisrendered
usingabackgroundimage,atexturemaponthebody,lightingandacameraposition.Theseingredientsarerandomlysampledtoincrease
thediversityofthedata.WegenerateRGBimagestogetherwith2D/3Dposes,surfacenormals,opticalflow,depthimages,andbody-part
segmentationmapsforrenderedpeople.
reviews previous work on the use of synthetic datasets in man trajectories from MoCap data. [28] trains CNNs with
computervision. Section3presentsourapproachforgen- syntheticdepthimages.
eratingrealisticsyntheticvideosofpeople. InSection4we The closest work to this paper is [7], where the authors
describe our convolution architecture for human body-part render large-scale synthetic images for predicting 3D pose
segmentation and depth estimation. Section 5 reports ex- withCNNs. Ourdatasetdiffersfrom[7]byhavingaricher,
periments. WeconcludeinSection6. per-pixelgroundtruth,thusallowingtotrainforpixel-wise
predictionsandmulti-taskscenarios. Inaddition,weargue
2.Relatedwork that the realism in our synthetic images is better (see [1]),
thus resulting in a smaller gap between features learned
Knowledge transfer from synthetic to real images has
fromsyntheticandrealimages. Themethodin[7]heavily
been recently studied with deep neural networks. Doso-
reliesonrealimagesasinputintheirtrainingwithdomain
vitskiy et al. [8] learn a CNN for optical flow estimation
adaptation. This is not the case for our synthetic training.
usingsyntheticallygeneratedimagesofrendered3Dmov-
Moreover, we render video sequences which can be used
ingchairs. Pengetal.[23]studytheeffectofdifferentvi-
fortemporalmodeling.
sualcuessuchasobject/backgroundtextureandcolorwhen
In this paper, we show that photo-realistic renderings
rendering synthetic 3D objects for object detection task.
of people under large variations in shape, texture, view-
Similarly, [33] explores rendering 3D objects to perform
pointandposecanhelptosolvepixel-wisehumanlabeling
viewpointestimation. Recently,Gaidonetal.[12]havere-
tasks. We use synthesized data to learn models of people
leased the Virtual KITTI dataset with synthetically gener-
and demonstrate improvements of body part segmentation
atedvideosofcarstostudymulti-objecttracking.
andbodydepthestimationinrealimages.
Several works focused on creating synthetic images of
humansforlearning2Dposeestimation[24,31], 3Dpose 3.Datageneration
estimation [7,9,13,30,37], pedestriandetection[20,24,
25], andactionrecognition[27,28]. Pishchulinetal.[25] ThissectionpresentsourSURREAL(SynthetichUmans
generatesyntheticimageswithagameengine. In[24],they foR REAL tasks) dataset and describes key steps for its
deform 2D images with a 3D model. More recently, Ro- generation (Section 3.1). We also describe how we obtain
gezandSchmid[30]usedimage-basedsynthesisengineto groundtruthdataforrealMoCapsequences(Section3.2).
augmentexistingrealimages. Ghezelghiehetal.[13]ren-
3.1.Synthetichumans
der synthetic images with 10 simple body models with an
emphasis on upright people; however, the main challenge Ourpipelineforgeneratingsyntheticdataisillustratedin
using existing MoCap data for training is to generalize to Figure2. Ahumanbody witha random 3Dpose, random
posesthatarenotupright. shapeandrandomtextureisrenderedfromarandomview-
Asimilardirectionhasbeenexploredin[27,28].In[27], point for some random lighting and a random background
actionrecognitionproblemisaddressedwithsynthetichu- image. Belowwedefinewhat“random”meansinallthese
2
Figure3.SamplesyntheticframesfromourSURREALdataset.Imagescontainrealisticallyrenderedpeoplewithalargevarietyofposes,
shapes,clothings,viewpointsandbackgroundclutter.
cases. Sincethedataissynthetic,wealsogenerateground It is often challenging to realistically and automatically
truthdepthmaps,opticalflow,surfacenormals,humanpart retarget MoCap skeleton data to a new model. Conse-
segmentationsandjointlocations(both2Dand3D).Asre- quently we do not use the skeleton data but rather use
sult,weobtain6.5millionframesgroupedinto67,582con- MoSh[19]tofittheSMPLparametersthatbestexplainraw
tinuous image sequences. See Table 1 for more statistics 3DMoCapmarkerlocations. Thisgivesboththe3Dshape
andFigure3forsamplesfromtheSURREALdataset. ofthesubjectandthearticulatedposeparametersofSMPL.
Toincreasethediversity,wereplacetheestimated3Dbody
Body model. Synthetic bodies are created using the shapewithasetofrandomlysampledbodyshapes.
SMPL body model [18]. SMPL is a realistic articulated We render each CMU MoCap sequence three times us-
model of the body created from thousands of high-quality ingdifferentrandomview-points. Moreover,wedividethe
3Dscans, whichdecomposesbodydeformationsintopose sequences into chunks of 100 frames with 30%, 50% and
(kinematicdeformationsduetoskeletalposture)andshape 70%overlapsforthesethreerenderings. Everyposeofthe
(body deformations intrinsic to a particular person that sequence is rendered with consistent parameters (i.e. body
makethemdifferentfromothers).SMPLiscompatiblewith shape, clothing, light, background etc.) within each 100-
mostanimationpackageslikeBlender[2]. SMPLdeforma- framechunk.
tionsaremodeledasacombinationoflinearblendskinning
Human texture. We texture the body model with tex-
andlinearblendshapesdefinedbyprincipalcomponentsof
turemapsextractedfromtwotypesofrealscans. Thefirst
bodyshapevariation. SMPLposeandshapeparametersare
oneusesCAESARscans,whichcomewithacolortexture
convertedtoatriangulatedmeshusingBlender,whichthen
per3Dpoint, andweextractSMPLtexturemapsfromthe
appliestexture,shadingandaddsabackgroundtogenerate
scans. Thesetexturemapsarevariedintermsofskincolor
thefinalRGBoutput.
andidentity. However,theirqualityisnotidealduetolow
resolution,uniformtight-fittingclothing,andvisiblemark-
Bodyshape. Inordertorendervaried,butrealistic,body ers placed on the face and body. Anthropometric markers
shapes we make use of the CAESAR dataset [29], which areautomaticallyremovedfromthetextureimagesandin-
wasusedtotrainSMPL.Tocreateabodyshape,weselect painted.
one of the CAESAR subjects at random and approximate Inordertoprovidemorevariety,weextractasecondset
theirshapewiththefirst10SMPLshapeprincipalcompo- oftexturesfrom3Dscansof7differentsubjectswithnor-
nents. Tenshapecomponentsexplainmorethan95%ofthe mal clothing from a total of 98 different capture sessions.
shapevarianceinCAESAR(attheresolutionofourmesh) These scans are registered with 4Cap as in [26]. The tex-
andproducequiterealisticbodyshapes. ture of real clothing substantially increases the realism of
generated images, even though SMPL does not model 3D
Body pose. To generate images of people in realistic deformationsofclothes.
poses,wetakemotioncapturedatafromtheCMUMoCap 20%ofourdataisrenderedwiththefirstset(208CAE-
database [3]. CMU MoCap contains more than 2000 se- SAR textures randomly sampled from 4000), and the rest
quencesof23high-levelactioncategories,resultinginmore withthesecondset(701clothedtextures). Topreservethe
than10hoursofrecorded3Dlocationsofbodymarkers. anonymity of subjects, we replace all faces in the texture
3
Table1.SURREALdatasetinnumbers.EachMoCapsequenceis
mapsbytheaverageCAESARface. Theskincolorofthis
rendered3times(with3differentoverlapratios).Clipsaremostly
average face is corrected to fit the face skin color of the
100frameslong.Weobtainatotalof6,5millionframes.
originaltexturemap.Thiscorrectedaveragefaceisblended
smoothlywiththeoriginalmap,resultinginarealisticand #subjects #sequences #clips #frames
anonymizedbodytexture. Train 115 1,964 55,001 5,342,090
Test 30 703 12,528 1,194,662
Light. ThebodyisilluminatedusingSphericalHarmon-
ics with 9 coefficients [14]. The coefficients are randomly Total 145 2,607 67,582 6,536,752
sampled from a uniform distribution between −0.7 and
0.7,apartfromtheambientilluminationcoefficient(which
MoSh [19] to fit the SMPL body shape and pose to the
has a minimum value of 0.5) and the vertical illumination
raw MoCap marker data. This provides a good fit of the
component, which is biased to encourage the illumination
model to the shape and the pose of real bodies. Given the
fromabove. SinceBlenderdoesnotprovideSphericalHar-
providedcameracalibration, weprojectmodelstoimages.
monics illumination, a spherical harmonic shader for the
We then render the ground truth segmentation, depth, and
bodymaterialwasimplementedinOpenShadingLanguage
2D/3Djointsjustasabove. Theonlydifferencehereisthat
(OSL).
these now are in correspondence with real pixel values in
Camera. Theprojectivecamerahasaresolutionof320× the dataset. As MoSh provides almost perfect fits of the
240, focal length of 60mm and sensor size of 32mm. To model,weconsiderthisdatatobe“groundtruth”. SeeFig-
generate images of the body in a wide range of positions, ures6and7forgeneratedexamples.
we take 100-frame MoCap sub-sequences and, in the first
4.Approach
frame, render the body so that the center of the viewport
pointstothepelvisofthebody,atarandomdistance(sam-
Inthissection,wepresentourapproachforhumanbody
pledfromanormaldistributionwith8metersmean,1meter
partsegmentation[22,5]andhumandepthestimation[10,
deviation)witharandomyawangle. Theremainderofthe
11,17],whichwetrainwithsyntheticand/orrealdata,see
sequencetheneffectivelyproducesbodiesinarangeoflo-
Section5foranevaluation.
cationsrelativetothecamera.
Our approach builds on the stacked hourglass network
Background. We render the person on top of a static architecture introduced originally for 2D pose estimation
backgroundimage. Toensurethatthebackgroundsarerea- problem [21]. This network involves several repetitions
sonablyrealisticanddonotincludeotherpeople, wesam- of contraction followed by expansion layers which have
ple from a subset of LSUN dataset [36] that includes total skipconnectionstoimplicitlymodelspatialrelationsfrom
of 400K images from the categories kitchen, living room, different resolutions that allows bottom-up and top-down
bedroomanddiningroom. structuredprediction. Theconvolutional layerswith resid-
ual connections and 8 ‘hourglass’ modules are stacked on
Ground truth. We perform multiple rendering passes in topofeachother,eachsuccessivestacktakingtheprevious
Blender to generate different types of per-pixel ground stack’s prediction as input. The reader is referred to [21]
truth. The material pass generates pixel-wise segmenta- formoredetails. Avariantofthisnetworkisusedforscene
tionofrenderedbodyparts,givendifferentmaterialindices depth estimation [6]. We choose this architecture because
assigned to different parts of our body model. The veloc- itcaninferpixel-wiseoutputbytakingintoaccounthuman
ity pass, typically used to simulate motion blur, provides bodystructure.
us with a render simulating ground truth optical flow. The Our network input is a 3-channel RGB image of size
depth and normal passes, used for emulating effects like 256×256croppedandscaledtofitahumanboundingbox
fog, bokeh or for performing shading, produces per-pixel using the ground truth. The network output for each stack
depth maps and normal maps. The final texture rendering hasdimensions64×64×15inthecaseofsegmentation(14
passcompositestheshaded,texturedbodyovertherandom classesplusthebackground)and64×64×20fordepth(19
background. Together with this data we also save camera depth classes plus the background). We use cross-entropy
parameters, lighting parameters as well as the 2D and 3D lossdefinedonallpixelsforbothsegmentationanddepth.
positionsofbodyjoints. Thefinallossofthenetworkisthesumover8stacks. We
trainfor50Kiterationsforsyntheticpre-trainingusingthe
3.2.Generatinggroundtruthforrealhumandata
RMSpropalgorithmwithmini-batchesofsize6andalearn-
Human3.6M dataset [16, 15] provides ground truth for ing rate of 10−3. Our data augmentation during training
2D and 3D human poses. We complement this ground includesrandomrotations,scalingandcolorjittering.
truth and generate predicted body-part segmentation and We formulate the problem as pixel-wise classification
depth maps for people in Human3.6M. Here again we use task for both segmentation and depth. When addressing
4
segmentation, each pixel is assigned to one of the pre- Input Predsegm GTsegm Preddepth GTdepth
defined 14 human parts, namely head, torso, upper legs,
lowerlegs,upperarms,lowerarms,hands,feet(separately
for right and left) or to the background class. Regarding
the depth, we align ground-truth depth maps on the z-axis
by the depth of the pelvis joint, and then quantize depth
values into 19 bins (9 behind and 9 in front of the pelvis)
Wesetthequantizationconstantto45mmtoroughlycover
thedepthextentofcommonhumanposes. Thenetworkis
trained to classify each pixel into one of the 19 depth bins Figure4.Segmentationanddepthpredictionsonoursynthetictest
orbackground.Attesttime,wefirstupsamplefeaturemaps set.
ofeachclasswithbilinearinterpolationbyafactorof4to
Input Real Synth Real+Synth GT
outputtheoriginalresolution. Then,eachpixelisassigned
to the class for which the corresponding channel has the
maximumactivation.
5.Experiments
Wetestourapproachonseveraldatasets. First, weval-
idate whether our models for segmentation and depth esti-
mation perform well on the synthetic test set of our SUR-
REAL dataset. This gives an idea about the model’s abil-
ity to learn the given task for the same domain. Second,
weevaluatethesegmentationtaskonrealimagesfromthe
Freiburg Sitting People dataset [22]. Next, we measure
segmentation and depth estimation performance on videos Figure5.PartsegmentationontheFreiburgSittingPeopledataset,
fromtheHuman3.6Mdataset[16,15],wherethe3Dinfor- training only on FSitting (Real), training only on synthetic im-
mation is also available. Finally, we qualitatively evaluate ages (Synth), fine-tuning on 2 training subjects from FSitting
(Real+Synth).Fine-tuninghelpsalthoughonlyfor200iterations.
our approach on the MPII Human Pose dataset [4] to see
success and failure cases in a challenging domain adapta-
tionscenario.
ThesplitisconstructedsuchthatagivenCMUMoCapsub-
5.1.Evaluationmetrics ject is assigned as either train or test. However some sub-
jectshavealargenumberofinstances,somesubjectshave
Wechooseintersectionoverunion(IOU)andpixelaccu-
unique actions, and some actions are very common (walk,
racymetricsforevaluatingthesegmentationapproach. The
run,jump). Overall,30subjectsoutof145areassignedas
finalmeasureistheaverageover14humanpartsasin[22].
test. 28 test subjects cover all common actions, and 2 test
Depthestimationisformulatedasaclassificationproblem,
subjectshaveuniqueactions. Remainingsubjectsareused
but we take into account the continuity when we evaluate.
for training. Although our synthetic images have different
Weuseroot-mean-squared-error(RMSE)thatiscomputed
bodyshapeandappearancethanthesubjectintheoriginat-
betweenthepredictedquantizeddepthvalue(class)andthe
ing MoCap sequence, we still found it appropriate to split
ground truth quantized depth on the human pixels. How-
bysubjects.Weseparateasubsetofourbodyshapes,cloth-
ever,tobeabletointerprettheerrorintermsofrealworld
ing and background images for the test set. This ensures
coordinates,wemultiplytheerrorbythequantizationcon-
that our tests are unbiased with regards to appearance, yet
stant that we used, which is 45mm. Furthermore, we re-
are still representative of all actions. Table 1 summarizes
portascaleandtranslationinvariantRMSE(st-RMSE)by
thenumberofframes, clipsandMoCapsequencesineach
solving for the best translation and scaling in z-axis to fit
split. Clipsarethecontinuous100-frameimagesequences
the prediction to the ground truth. Since inferring depth
wherewehavethesame randombodyshape, background,
fromRGBisambiguous, thisisacommontechniqueused
clothing,cameraandlighting. Anewrandomsetispicked
inevaluations[11].
ateveryclip. Notethatafewsequenceshavelessthan100
5.2.Validationonsyntheticimages frames,thereforenotalltheclipshave100-frames.
Train/testsplit. Toevaluateourmethodsonsyntheticim- Results on synthetic test set. The evaluation is per-
ages, we separate 20% of the synthetic frames for the test formed on the middle frame of each 100-frame clip on
setandtrainallournetworksontheremainingtrainingset. the aforementioned held-out synthetic test set, totaling in
5
Table2.Partssegmentationresultson4testsubjectsofFreiburg
train the same network from scratch using 2 training sub-
SittingPeopledataset. IOUforhead,torsoandupperlegs(aver-
jects. This network quickly overfits as there are few sub-
aged over left and right) are presented as well as the mean IOU
jects to learn from and the performance is quite low. Our
andmeanpixelaccuracyover14parts. Themeansdonotinclude
‘Synthetic’resultisobtainedusingthenetworkpre-trained
backgroundclass.Byaddinganupsamplinglayer,wegetthebest
on synthetic images without fine-tuning. We get 51.88%
resultsreportedonthisdataset.
pixelaccuracyand40.1%IOUwiththismethodandclearly
Head Torso Legs mean mean
up outperform training from real images. Furthermore, fine-
Trainingdata IOU IOU IOU IOU Acc.
tuning(‘Synthetic+Real’)with2trainingsubjectshelpssig-
Real+Pascal[22] - - - 64.10 81.78
nificantly. SeeFigure5forqualitativeresults. Giventhelit-
Real 58.44 24.92 30.15 28.77 38.02 tle amount for training in FSitting, the fine-tuning quickly
Synth 73.20 65.55 39.41 40.10 51.88 convergesafter200iterations.
Synth+Real 72.88 80.76 65.41 59.58 78.14 In [22], the authors introduce a network that outputs a
Synth+Real+up 85.09 87.91 77.00 68.84 83.37 high-resolutionsegmentationafterseverallayersofupcon-
volutions. For a fair comparison, we modify our network
Table3.PartssegmentationresultsonHuman3.6M.Thebestresult
tooutputfullresolutionbyaddingonebilinearupsampling
isobtainedbyfine-tuningsyntheticnetworkwithrealimages.Al-
layerfollowedbynonlinearity(ReLU)andaconvolutional
thoughtheperformanceofthenetworktrainedonlywithrealdata
layerwith3×3filtersthatoutputs15×300×300instead
outperformstrainingonlywithsynthetic,thepredictionsvisually
of15×64×64asexplainedinSection4. Ifwefine-tune
areworsebecauseofoverfitting,seeFigure6.
this network (‘Synth+Real+up’) on FSitting, we improve
IOU Accuracy
performance and outperform [22] by a large margin. Note
Trainingdata fg+bg fg fg+bg fg
that [22] trains on the same FSitting training images, but
Real 49.61 46.32 58.54 55.69 addedaround2,800Pascalimages. Hencetheyusesignifi-
Synthetic 46.35 42.91 56.51 53.55
cantlymoremanualannotationthanourmethod.
Synthetic+Real 57.07 54.30 67.72 65.53
5.4.SegmentationanddepthonHuman3.6M
Showcasingourapproachrequiresexpensiveannotation
12,528 images. For segmentation, the IOU and pixel ac-
if applied on real data. This is the reason for existing real
curacy are 69.13% and 80.61%, respectively. Evaluation
datasetsbeingsmallandnotsuitablefortrainingdeepnet-
of depth estimation gives 72.9mm and 56.3mm for RMSE
works. As a result, there is no available dataset to carry
and st-RMSE errors, respectively. Figure 4 shows sample
out a clean evaluation. For this reason, we came up with
predictions. Forbothtasks, theresultsaremostlyaccurate
generatingnearlyperfectgroundtruthforimagesrecorded
on synthetic test images. However, there exist a few chal-
withacalibratedcameraandgiventheirMoCapdata. Hu-
lenging poses (e.g. crawling), test samples with extreme
man3.6Miscurrentlythelargestdatasetwheresuchinfor-
close-upviews, andfinedetailsofthehandsthatarecaus-
mation is available. There are 3.6 million frames from 4
ingerrors. Inthefollowingsections,weinvestigateifsimi-
cameras. Weusesubjects1,5,6,7,8fortraining,subject9
larconclusionscanbemadeforrealimages.
forvalidationandsubject11fortestingasin[30,35]. We
5.3.SegmentationonFreiburgSittingPeople useallframesfromonerepetitionofeachactionfortrain-
ing, and every 64th frame from all repetitions at test. The
Freiburg Sitting People (FSitting) dataset [22] is com-
frames have resolution 1000 × 1000 pixels, we assume a
posed of 200 high resolution (300x300 pixels) front view
256×256croppedhumanboundingboxisgiventoreduce
imagesof6subjectssittingonawheelchair. Thereare14
computationalcomplexity. Weevaluatetheperformanceof
humanpartannotationsavailable. SeeFigure5forsample
both segmentation and depth, and compare with the base-
test images and corresponding ground truth (GT) annota-
lineforwhichwetrainanetworkonrealimagesonly.
tion. We use the same train/test split as [22], 2 subjects
for training and 4 subjects for test. The amount of data is
5.4.1 Segmentation
limited for training deep networks. We show that our net-
workpre-trainedonlyonsyntheticimagesisalreadyableto Table 3 summarizes the parts segmentation results on Hu-
segmenthumanbodyparts. Thisshowsthatthehumanren- man3.6M. We report both the mean over 14 human parts
deringsinthesyntheticdatasetarerepresentativeofthereal (fg) and the mean together with the background class
images,suchthatnetworkstrainedexclusivelyonsynthetic (fg+bg). Training on real images instead of synthetic im-
datacangeneralizequitewelltorealdata. agesincreasesIOUby3.4%andpixelaccuracyby2.14%.
Table 2 summarizes segmentation results on FSitting. This is expected because the training distribution matches
We carry out several experiments to understand the gain the test distribution in terms of background, camera posi-
from synthetic pre-training. For the ‘Real’ baseline, we tion and action categories (i.e. poses). The amount of real
6
Input Real Synth Real+Synth GT Input Real Synth Real+Synth GT
Figure6.PartssegmentationontheHuman3.6Mdataset,trainingonlyonrealimagesandMoSH-generatedground-truthfromHuman3.6M
(Real),trainingonlyonsyntheticimagesfromSURREAL(Synth),andfine-tuningonrealHuman3.6Mdata(Real+Synth). The‘Real’
baselineclearlyfailsonupperarmsbyfittingtheskincolor. Thesyntheticpre-trainednetworkhasseenmorevarietyinclothing. Best
resultisachievedbythefine-tunednetwork.
Table4.DepthestimationresultsonHuman3.6M(inmillimeters).
dataissufficienttoperformCNNtraining. However,since
Thedeptherror(RMSE)andscale-translationinvarianterror(st-
there are very few subjects available, we see that the net-
RMSE)arereportedonforegroundpixels.Theerrormeasurement
work doesn’t generalize to different clothing. In Figure 6,
onlyongivenhumanjointsisalsoprovided(PoseRMSE).
the ‘Real’ baseline has the border between shoulders and
Trainingdata RMSE st-RMSE PoseRMSE st-PoseRMSE
upper arms exactly on the t-shirt boundaries. This reveals
that the network learns about skin color rather than actual Real 96.3 75.2 122.6 94.5
Synthetic 111.6 98.1 152.5 131.5
bodyparts. Ourpre-trainednetwork(Synth)performsrea-
Synthetic+Real 90.0 67.1 92.9 82.8
sonablywell,eventhoughtheposedistributioninourMo-
Cap is quite different than that of Human3.6M. When we
fine-tune the network with real images from Human3.6M Therefore,wetestthegeneralizationofourmodelonmore
(Real+Synth), the segmentations look very similar to the challengingimages.Currently,MPIIHumanPose[4]isone
groundtruthandweoutperformthebaselinebyalargemar- of the biggest datasets with diverse viewpoints and clutter.
gin. Moreover, our model is capable of distinguishing left However,thisdatasethasnoannotationforpartsegmenta-
and right most of the time on all 4 views since it has been tion nor depth. For this reason, we qualitatively show our
trainedwithrandomlysampledviews. predictions. In Figure 8, several success and failure cases
areillustrated. Ourmodelgeneralizesreasonablywell,ex-
5.4.2 Depthestimation
ceptwhentherearemultiplepeopleclosetoeachother,and
DepthestimationresultsonHuman3.6Mforvariousposes in extreme viewpoints and occlusions, which have not ap-
and viewpoints are illustrated in Figure 7. Here, the pre- pearedduringtraining.
trainednetworkfailsattheverychallengingposes,although
6.Conclusions
it still captures partly correct estimates (left column, third,
fourth rows). Fine-tuning on real data compensates for Inthisstudy,wehaveshownsuccessfullarge-scaletrain-
these errors and refines estimations. In Table 4, we show ingofCNNsfromsyntheticallygeneratedimagesorpeople.
RMSEerrormeasuredonforegroundpixels, togetherwith We have addressed two tasks, namely, human body parts
thescale-translationinvariantversion(seeSection5.1). We segmentation and depth estimation, for which large-scale
alsoreporttheerroronlyonknown2Djoints(PoseRMSE) manual annotation is infeasible. Our generated synthetic
to have an idea of how well a 3D pose estimation model datasetcomeswithrichpixel-wisegroundtruthinformation
would work based on the depth predictions. One would andcanpotentiallybeusedforothertasksthanconsidered
need to handle occluded joints to infer 3D locations of all here. Unlikemanyexistingsyntheticdatasets,thefocusof
joints,andthisisbeyondthescopeofthecurrentpaper. SURREALisontherealisticrenderingofpeople,whichis
achallengingtask. Inourfuturework,weplantointegrate
5.5.QualitativeresultsonMPIIHumanPose
the person into the background in a more realistic way by
The datasets we evaluated our approach with are rela- takingintoaccountthelightingandthe3Dscenelayout.We
tivelysimple, i.e.nobackgroundclutter, fewsubjects, sin- alsoplantoaugmentthedatawithmorechallengingscenar-
glepersonperimage,fullbodyvisible. Exceptthefirsttwo iosbycreatingocclusionsandrenderingmultiplepeoplein
properties, this setup matches our synthetic training data. thesamescene.
7
Input Real Synth Real+Synth GT Input Real Synth Real+Synth GT
Figure 7. Depth segmentation on the Human3.6M dataset, columns represent same training partitions as in Figure 6. The pre-trained
network(Synth)failsduetoscalemismatchinginthetrainingsetandlowcontrastbodyparts,butfine-tuningwithrealdata(Real+Synth)
tendstorecoverfromtheseproblems.
Figure8.QualitativeresultsonchallengingimagesfromMPIIHumanPosedataset.Multi-person,occlusionandextremeposesaredifficult
casesforourmodel.Giventhatthereisnofine-tuninginvolved,themodelisabletogeneralizesufficientlywellonclutteredrealdata.
8
Acknowledgements [17] F.Liu,C.Shen,andG.Lin.Deepconvolutionalneuralfields
fordepthestimationfromasingleimage. InCVPR,2015. 4
This work was supported by the MSR-Inria joint lab,
[18] M.Loper,N.Mahmood,J.Romero,G.Pons-Moll,andM.J.
Google and Facebook Research Awards, the ERC starting
Black. SMPL:Askinnedmulti-personlinearmodel. SIG-
grant ACTIVIA and the ERC advanced grant ALLEGRO. GRAPHAsia,2015. 1,3
WealsoacknowledgethefundingprovidedbytheAlexan- [19] M.M.Loper,N.Mahmood,andM.J.Black.MoSh:Motion
dervonHumboltFoundation. and shape capture from sparse markers. SIGGRAPH Asia,
2014. 1,3,4
References [20] J. Marin, D. Vazquez, D. Geronimo, and A. M. Lopez.
Learning appearance in virtual scenarios for pedestrian de-
[1] http://www.di.ens.fr/willow/research/ tection. InCVPR,2010. 2
surreal/. 1,2
[21] A. Newell, K. Yang, and J. Deng. Stacked hourglass net-
[2] Blender - a 3D modelling and rendering package. http: worksforhumanposeestimation. InECCV,2016. 1,4
//www.blender.org. 3 [22] G.Oliveira,A.Valada,C.Bollen,W.Burgard,andT.Brox.
[3] Carnegie-MellonMocapDatabase. http://mocap.cs. Deeplearningforhumanpartdiscoveryinimages. InICRA,
cmu.edu/. 1,3 2016. 4,5,6
[4] M.Andriluka,L.Pishchulin,P.Gehler,andB.Schiele. 2D [23] X.Peng,B.Sun,K.Ali,andK.Saenko. Learningdeepob-
humanposeestimation: Newbenchmarkandstateoftheart jectdetectorsfrom3Dmodels. InICCV,2015. 2
analysis. InCVPR,2014. 1,5,7 [24] L. Pishchulin, A. Jain, M. Andriluka, T. Thormhlen, and
[5] L.-C.Chen,Y.Yang,J.Wang,W.Xu,andA.L.Yuille. At- B. Schiele. Articulated people detection and pose estima-
tentiontoscale: Scale-awaresemanticimagesegmentation. tion:Reshapingthefuture. InCVPR,2012. 2
InCVPR,2016. 4 [25] L. Pishchulin, A. Jain, C. Wojek, M. Andriluka,
[6] W.Chen,Z.Fu,D.Yang,andJ.Deng. Single-imagedepth T. Thormhlen, and B. Schiele. Learning people detection
perceptioninthewild. InNIPS,2016. 4 modelsfromfewtrainingsamples. InCVPR,2011. 2
[7] W.Chen,H.Wang,Y.Li,H.Su,Z.Wang,C.Tu,D.Lischin- [26] G. Pons-Moll, J. Romero, N. Mahmood, and M. J. Black.
ski,D.Cohen-Or,andB.Chen.Synthesizingtrainingimages Dyna: Amodelofdynamichumanshapeinmotion. ACM
forboostinghuman3Dposeestimation. In3DV,2016. 2 TransactionsonGraphics,(Proc.SIGGRAPH),2015. 3
[8] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, [27] H.RahmaniandA.Mian. Learninganon-linearknowledge
V. Golkov, P. van der Smagt, D. Cremers, and T. Brox. transfermodelforcross-viewactionrecognition. InCVPR,
FlowNet: Learning optical flow with convolutional net- 2015. 2
works. InICCV,2015. 2 [28] H.RahmaniandA.Mian. 3Dactionrecognitionfromnovel
[9] Y.Du,Y.Wong,Y.Liu,F.Han,Y.Gui,Z.Wang,M.Kankan- viewpoints. InCVPR,2016. 2
halli,andW.Geng. Marker-less3Dhumanmotioncapture [29] K. Robinette, S. Blackwell, H. Daanen, M. Boehmer,
withmonocularimagesequenceandheight-maps.InECCV, S.Fleming,T.Brill,D.Hoeferlin,andD.Burnsides.Civilian
2016. 2 American and European Surface Anthropometry Resource
[10] D.EigenandR.Fergus. Predictingdepth, surfacenormals (CAESAR),FinalReport. 2002. 3
and semantic labels with a common multi-scale convolu- [30] G.RogezandC.Schmid. MoCap-guidedDataAugmenta-
tionalarchitecture. InICCV,2015. 4 tionfor3DPoseEstimationintheWild. InNIPS,2016. 2,
6
[11] D.Eigen,C.Puhrsch,andR.Fergus. Depthmapprediction
[31] J.Romero,M.Loper,andM.J.Black. FlowCap:2Dhuman
from a single image using a multi-scale deep network. In
NIPS,2014. 4,5 posefromopticalflow. InGCPR,2015. 2
[32] B. Sapp and B. Taskar. Multimodal decomposable models
[12] A.Gaidon,Q.Wang,Y.Cabon,andE.Vig. Virtualworlds
forhumanposeestimation. InCVPR,2013. 1
asproxyformulti-objecttrackinganalysis. InCVPR,2016.
[33] H.Su,C.R.Qi,Y.Li,andL.J.Guibas. RenderforCNN:
2
Viewpoint estimation in images using CNNs trained with
[13] M.F.Ghezelghieh,R.Kasturi,andS.Sarkar.Learningcam-
rendered3Dmodelviews. InICCV,2015. 2
eraviewpointusingcnntoimprove3Dbodyposeestimation.
[34] S.-E.Wei,V.Ramakrishna,T.Kanade,andY.Sheikh. Con-
In3DV,2016. 2
volutionalposemachines. CVPR,2016. 1
[14] R. Green. Spherical harmonic lighting: The gritty details.
[35] H.Yasin,U.Iqbal,B.Krger,A.Weber,andJ.Gall. Adual-
InArchivesoftheGameDevelopersConference,volume56,
sourceapproachfor3Dposeestimationfromasingleimage.
2003. 4
InCVPR,2016. 6
[15] C.Ionescu,L.Fuxin,andC.Sminchisescu.Latentstructured
[36] F.Yu,Y.Zhang,S.Song,A.Seff,andJ.Xiao. Lsun: Con-
modelsforhumanposeestimation. InICCV,2011. 4,5
structionofalarge-scaleimagedatasetusingdeeplearning
[16] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu.
withhumansintheloop. arXiv:1506.03365,2015. 4
Human3.6M: Large scale datasets and predictive methods
[37] X.Zhou,M.Zhu,S.Leonardos,K.Derpanis,andK.Dani-
for 3D human sensing in natural environments. IEEE
ilidis. Sparsenessmeetsdeepness: 3Dhumanposeestima-
TransactionsonPatternAnalysisandMachineIntelligence,
tionfrommonocularvideo. InCVPR,2016. 2
36(7):1325–1339,2014. 1,4,5
9