Novel Views of Objects from a Single Image

6.3 MB·

by  Konstantinos Rematas

1
Novel Views of Objects from a Single Image
Konstantinos Rematas, Chuong Nguyen, Tobias Ritschel, Mario Fritz, and Tinne Tuytelaars
Abstract—Takinganimageofanobjectisatitscorealossyprocess.Therichinformationaboutthethree-dimensionalstructureofthe
worldisflattenedtoanimageplaneanddecisionssuchasviewpointandcameraparametersarefinalandnoteasilyrevertible.Asa
consequence,possibilitiesofchangingviewpointarelimited.Givenasingleimagedepictinganobject,novel-viewsynthesisisthetaskof
generatingnewimagesthatrendertheobjectfromadifferentviewpointthantheonegiven.Themaindifficultyistosynthesizetheparts
thataredisoccluded;disocclusionoccurswhenpartsofanobjectarehiddenbytheobjectitselfunderaspecificviewpoint.Inthiswork,
weshowhowtoimprovenovel-viewsynthesisbymakinguseofthecorrelationsobservedin3Dmodelsandapplyingthemtonewimage
instances.Weproposeatechniquetousethestructuralinformationextractedfroma3Dmodelthatmatchestheimageobjectintermsof
viewpointandshape.Forthelatterpart,weproposeanefficient2D-to-3Dalignmentmethodthatassociatespreciselytheimage
6 appearancewiththe3Dmodelgeometrywithminimaluserinteraction.Ourtechniqueisabletosimulateplausibleviewpointchangesfor
1 avarietyofobjectclasseswithinseconds.Additionally,weshowthatoursynthesizedimagescanbeusedasadditionaltrainingdatathat
0
improvestheperformanceofstandardobjectdetectors.
2

g IndexTerms—NovelViewSynthesis,2Dto3DAlignment,ImageBasedRendering
u (cid:70)
A
5 1 INTRODUCTION
1
GIVEN a single image of an object, humans can clearly appearanceandgeometry,anecessarystepfortheviewsynthesis,

] “imagine” what this object would look like from different canhelpintaskssuchasshapemanipulation,appearancetransfer,
V
viewpoints.Evenwhentheparticularobjectwasneverseenbefore, etc. Having information about the 3D structure of an object is
C its visible parts are enough to facilitate the imaginary synthesis essentialalsoforimageunderstanding.Afterall,anobjectisnot
. thathappensinourmind.Forexamplewhenweseethesideview just a two-dimensional box but rather a real world entity with
s
c ofacar,wecanhallucinatethefrontpart,includingdetailssuch certaingeometry,materialsandinteractionwiththeenvironment.
[ as lights, hood, wind shield etc. What enables us to do so? The The work presented in this paper focuses on the synthesis

answer lies in our knowledge of the structural information for of new views of objects from single images and the necessary
2
v a particular visual class. The object instances of that class have prerequisitestoachievethis.Weapproachthesynthesisproblem
8 specific3Dshapes,symmetries,materials,colors,etc.Severalof by considering the 3D models of a class as proxies to transfer
2 theseattributesaresharedamongtheclassinstancesandthusthey theappearancetothenovelviews.Westartbyassumingmanual
3 canbeusedascuesforthesynthesisprocess.Probablythemost alignment between the 2D appearance and the 3D shape of the
0
importantcuesforsynthesisarethoserelatedwiththe3Dstructure object,andwesynthesizedirectlythepixelsinthenovelviewas
0
oftheobjects.Thereasonisthatthistasksharesthesameprinciples acombinationofpixelsintheoriginalview.Next,weextendour
.
2 withimageformation:animageisnotjustpixelsinagridbutitis approachtoauser-centeredsetting,whereinteractionandspeed
0
the2Dinterpretationofa3Dworld. are essential, and viewpoint-dependency of appearance is taken
6
The synthesized view needs to be a plausible simulation of intoaccount.Wepresentamethodforaligningthegeometryand
1
: how the object would appear in the new view. There are two theappearancewithminimaluserinteraction.Then,werevisitthe
v parametersinsuchasetting:a)theunderlyinggeometryandb)the viewsynthesisandweextendittoincludeviewpointdependency
i
X associatedappearance.Thenovelviewneedstohavethegeometric for the reconstruction in an efficient way. Our key contributions
r characteristics of the object’s class and the appearance of the canbesummarizedas:
a depicted object instance. If only one of the two is inconsistent • A method to synthesize realistic novel 3D views of an
then the result is perceived as fake or unrealistic. For example, objectfromasingle2Dimage
simplereplacementofacarwitharenderedcar3Dmodelkeeps • Aproceduretoaligna3Dshapestoasingleimage
thegeometricstructureoftheclass,buttheappearancewillcome • Amplificationoftrainingdataindetectionandclassification
fromthe3Dmodelandmaynotblendinwell.Ontheotherhand, tasksbynovel-viewsynthesis.
ifwediscardtheunseengeometrytheninanewviewtheobject
This article is organised as follows: the previous work is
willhaveemptyareas,similartoafacadeinaPotemkinvillage.
described in Sec. 2. In Sec. 3 we introduce our core method
Theabilitytosynthesizenovelviewsopensnewpossibilities
fornovelviewsynthesis.Sec.4focusesonthematchingbetween
inseveralimagemanipulationtasks.Adirectapplicationisthe3D
the geometry and the appearance of the object. In Sec. 5 we
manipulation of objects in images. In addition, the novel views
revisitthenovelviewsynthesisfromaviewer-centeredperspective.
can be used as additional training data for conventional object
Experimental results are reported in Section Sec. 6 and further
detectors/classifiers.Theemergenceof3Dorlightfieldorstereo
discussedinSectionSec.7.SectionSec.8concludesthepaper.
displaysprovideanotherdomainthatbenefitsfromthegeneration
ofadditionalviewsofobjects.Uncoveringtherelationshipbetween
2 PREVIOUS WORK
View Synthesis: Generating novel views from 2D images
• Projecthttp://homes.cs.washington.edu/∼krematas/ViewSynthesis/
is an image-based rendering problem, with several applications
2
Novel view 3
Novel view 2
Input image Novel view 1
Fig.1.Givenaninputimage(left),wesynthesizenovelviewsofthedepictedobjectatinteractiveframesrateswithview-dependentapearance(right).
in computer vision [1] and computer graphics [2], [3]. Image- annotations.Thesituationismoredifficultifnoparametricmodel
basedrenderingtechniques[4]canconvincinglysynthesizenovel isavailable.Therecentworkof[23]performs3Dreconstruction
viewpointswithcomplexmaterialandilluminationeffectsfroma ofobjectsinimagecollections,buttheirmethodisnotinteractive
setofimages.Ithasbeenshownthatqualitycangreatlybenefitifa as it requires the images to be processed jointly. When a shape
notionof3Dgeometryisprovidedtothesystem[5].Kholgadeetal. deformationmodelisavailable,the3Dshapecanbedeformedto
[6]showedthatgivenamanual3Dshapealignmentconvincingand fittingthe2Dsilhouettealsoleadingtoa2D-to-3Dalignment[24].
detail-preservingnovelviewscanbegeneratedevenfromasingle Wecombinesilhouetteandluminancepattern-basedapproachesin
image.Viewinterpolationincludingcomplexshadingispossible onediscriminativelearningframework.
byusingsurfacelightfields[7]orlumigraphs[4].Whendeforming Arecentsuggestionistouse3Dshapesandextractdiscrimi-
andblendingimages,detectingholesandfillingthemisthebiggest nativepatches[25]tofindthealignment,e.g.topaintings[26]or
challenge[8],usuallysolvedeitherbystretchingforegroundover singleobjects[27],[28].Suchanapproachhasbeenshownable
background or by inpainting [9], [10], but no approach we are toreliablyfindinstancesfromaclassforwhichalargesetof3D
awareofusesthestructureofa3Dtemplatetoguidethisinpainting. shapes is available. They obtain good precision, in real images
Appearancethatdependsonashapetemplatewasproposedforthe with clutter, but at high computational cost [27]. Limet al. [29]
diffusecaseandmanualinterventiontocapturetheappearanceof proposeaRANSAC-basedsamplingapproachinordertospeedup
artinthe“LitSphere”metaphor[11].Ourapproachcanbeseen the search - but this only works for exact 3D shapes and cannot
astheautomaticandcontinuousgeneralizationfromonetomany, copewithwithin-classvariability.Usingtemplates,itcanrequire
context-dependent Lit Spheres. Image Analogies [12] learns the up to 2 minutes to align a single chair in a cluster of 80 cores,
transformationfromapairofimages,toapplytoanovelimage. whichisfarfrombeingusableinaninteractiveapplication[27].
Similarly, we learn the relation between two rendered 3D views Theresultsinthisarticlearecomputedanorderofmagnitudefaster
of an image to generate appearance in new views from a single whileachievinghigherquality.Ourapproachisalsorelatedtothe
imageforgeneralappearanceandaccountfortherelationof3D recentworkof[30]intermsofextendingthefineviewalignment
geometryandmaterials[13].Simpler,jointbilateralupsampling stepto3Dshapepartstoallowforshapedeformations.
[14]hasbeenusedtoreconstructahigh-resolutionimagefroma
low-resolutionimageusingahigh-resolutionguidancesignal.Our Trainingfromsyntheticdata: Thereisanincreasinginterest
approachusesmultiplesynthesizedhigh-resolutionimagestoguide totraincomputervisionmodelsfromsynthesizeddatainorderto
view synthesis, including upsampling. For 3D reconstruction of achievemorescalablelearningschemes.Workonfullysynthetic
occludedareasinscannedobjects,approachesbasedontemplates data has shown mostly successful in depth data [31], [32], [33],
areroutinelyused[15]tofillholesinsurfaces,buttheseoperate shape-based representations [34], [35], textures [36], [37] and
on3Dsurfacesanddonotaccountforappearancesuchaswedo. scenes[38].Incontrast,wetakeanimage-basedapproachinorder
Anotherapproachtoreconstructoccludedregionsis[16],butthe to leverage real-world image statistics as well as the intra-class
methodrequiresvisibilityoftheregionoritssymmetricpartinthe variationavailabletousinimagedata.Previousimage-basedwork
videosequence.Commonly,acquiringview-dependentappearance synthesizestrainingimagesbyrecombiningposesandappearance
intermsofaBRDFrequirestoalsocapturelightprobe,evenwhen of objects in order to create new instances [39], [40], [41], [42],
thegeometryisknown[17].Wecircumventthisrequirementby [43].Morerecently,Suetal.[44]andPengetal.[45]rendered3D
notfactoringoutreflectanceandlighting,whilestillallowingfor modelstogenerateadditionaldataforobjectdetectionandpose
plausibleview-dependentappearance(suchadecompositioncanbe estimationrespectivelyusingConvolutionalNeuralNetworks.In
achievedusing[18]).Morerecently,theworkof[19]estimatesthe contrast,ourworkfocusesonsynthesisacrossviewpointswiththe
appearanceofanobjectusingReflectanceMaps[20]butrequires appearancegivenfromanimageanddealswithdisocclusionsthat
singlematerialobjects. arenotaddressedinpreviouswork.
2D-to-3D alignment: If the object belongs to a parametric 3 3D MODEL-GUIDED NOVEL-VIEW SYNTHESIS
familyofshapes,alignmentcanbeautomated[21].Forcarsand We pose novel view synthesis as a problem of reconstructing
bicycles, Zia et al. [22] have shown accurate alignments with a appearanceasafunctionofsceneattributesfromobservedsamples.
wireframe model based on a deformable 3D shape model and Inputtoouralgorithmisanimageandanaligned3Dmodeland
robustpartdetectors,startingfromCADmodelswithmanualpart outputisanimagefromanovelview.Firstly,theoriginalandthe
3
Fig.2.Wecomputeanovelviewimagef2fromtheinputimagef1using3Dinformationofanaligned3Dtemplate(Lefttoright:radiance,normals,
reflectance,positions)asguidance,evenifitsrenderingsL1andL2haveappearancelargelydifferentfromf1.qisthesamplingquality,indicating
thevisibilityintheoriginalview,andwistheflowbetweentheviews,whichisoptionalandusedtospeedupthesampling.
novelview ofthe 3Dmodelcan berendered easily.This allows wherev(x)=V(cid:48)−1p(x)/||V(cid:48)−1p(x)||2isthenormalizedviewing
2
to put pixel locations from the original to the novel view into directionandz returnsthedepthcomponentofavector.
correspondence(flow)andtodetectdisocclusions,i.e.,pixelsin Furtherwedefineametriconallattributesasfollows(Fig.3):
thenovelviewthatwerenotpresentintheoriginalview.Simply As a distance on positions ∆ we use Euclidean distance; for
p
copyingpixelsalongtheirflowwillresultintheclassicprojective normals,weusethedotproductasthedistance∆ ;forreflectance,
n
texturingusedinimage-basedrendering[8].Thechallengeistofill weapplyaperceptuallylinearPhongBRDFdistance∆ similar
r
thedisocclusionsbyinpainting.Themostsimplesolutionwould to [47]; radiance and locality of two pixels ∆ and ∆ is again
L l
betoreplacedisoccludedpixelswithpixelsfromtherendered3D measuredusingEuclideandistance.Usingalltheabove,wecan
model.This,however,cannotproduceaconsistentappearanceas nowwritetheprobabilityofassigningtheappearancefrompixel
precisematerials,lightingandgeometryareunknown. positionx tothenovelappearanceatlocationx as
1 2
Thekeyobservationisthatthe3Dmodelprovidesseveral3D
c(x ,x )=q(x )/(w ∆ (p (x ),p (x ))+
properties,suchasposition,normal,reflectance,foralldisoccluded 1 2 1 p p 1 1 2 2
pixels(Fig.2).Wecanusethisinformationtoguidetheinpainting wn∆n(n1(x1),n2(x2))+
ofappearanceforsuchpixels.Tothisend,wecopyappearancefrom w ∆ (r (x ),r (x ))+
r r 1 1 2 2
theoriginalview,thatissimilarintermsofposition,normaland
w ∆ (L (x ),L (x )))+
L L 1 1 2 2
reflectance.Briefly,eachpixelinthenovelviewcanbemodeled
w∆(x ,x )).
asaweightedsumofthepixelsintheoriginalview.Theseweights l l 1 2
areessentiallyacompatibilityscorebetweentheposition,normal
andreflectancevaluesofpixelsinthetwoviews. 3.2 Reconstructingappearance
Reconstructinganovelviewisnowaweightedsumovertheinput
3.1 Samplingappearance
image.Theresultimagef isreconstructedatlocationx ,as
2 2
Inputtooursystemisappearanceintheformofa2DRGBimage
(cid:90) (cid:30)(cid:90)
f1 : R2 → R3 anda3DmodelS ⊆ R3 withPhongreflectance f2(x2)= c(x1,x2)sf1(x1)dx1 c(x1,x2)sdx1 , (1)
F : S → R9 defined on it (diffuse color, specular color and
glossiness). We want to compute the image f that shows the where s is a sharpness parameter. If s is low, the reconstructed
2
objectfromaviewdifferentbythematrixV(cid:48) ∈R4×4. appearance is combined from many sources. It is more reliable,
The3Dmodeltypicallycontainstheobjectinquestion,aswell but also more smooth. If s gets larger, fewer, but higher-quality
as its context, e.g., a car standing on a planar ground, which is observationsdominatethesolution.Fordiscreteimagestheintegral
verycommonforcars.Theparticulartype(triangular,(NURBS) aboveturnsintoasuminpractice.Insteadofaligningthe3Dmodel
patches,procedural,etc.)andtechnicalpropertiesofthe3Dmodel S totheoriginalimagef in3D,we,fornow,deformtheoriginal
(facecount,orientability,UVcoordinates)arenotrelevantforour 2D image [1] to align it to the images p ,n and r of the 3D
1 1 1
approach,asweonlyrequirethatthemodelcanberenderedinto modelS.Sec.5willdiscusshowtoautomatizethisstep.
animagefromaview. Wedonotneedtoiterateovertheentireimagebutonlyover
First, let p1 : R2 → R3 be the position image, which is the alocalneighborhood.Ifx2 isthenovelimagepixelposition,we
projectionofS intheoriginalviewafterhidden-surfaceremoval. runoverafixed-sizeneighborhoodaroundlocationx +w(x ),
2 2
We produce such images using deferred shading [46] with z- where w : R2 → R2 is the flow between the novel view and
buffering, i.e., generating positions instead of shaded colors in the original view, i.e., where every pixel in the novel view is
rendering.Usingdeferredshading,wealsocomputen1 : R2 → comingfromintheoriginalview.Thisisbecausemorecorrelated
R3, the normal image computed from the derivative of S; r1 : pixelsarefoundintheneighborhoodofthepixelposition,thata
R2 →R9,thereflectanceimage;andL1 :R2 →R3,theradiance world space position had in the original view. It is produced as
(Fig. 2). Position, normal and reflectance images p2,n2,r2 and follows:WeagainrasterizeS fromtheviewV(cid:48) butstorew(x)=
L2 fromthenovelviewV(cid:48) areproducedinthesameway.Finally, x−ρ(V(cid:48)p(x))atthepixelwithlocationx,whereρ:R3 →R2
wecomputesamplingqualityandocclusioninaz-buffer-liketest, isthe(perspective)projectionfromworldtoimagespace.Flowis
formalizedbyafunction undefinedforpositionsthatwereoccludedintheoriginalview.
(cid:26) max(0,n(x)·v(x)) ifz(p(x)−V(cid:48)p(x))>(cid:15) We use a GPU to produce all guide images, parallel over
q(x)=
0 else, output pixels, usingdeferred shading and toevaluate Eq. 1on a
4
Fig.3. Reconstructionweightscanddistances∆ ofthreeoutputlocationsx inrespecttoallpositionsfromtheinputimage.
{L,n,p,r} {2,3,4}
512×512 image using a 128×128 neighborhood in less than a a) b) c) d) e) f)
second,allowingtointeractivelyexplorenovelviewsbymovinga
virtualcamera.
Fig. 5. Reflectance Map for the input image f1 seen in Fig. 4: a):
Reconstructedusingnearest-neighbor.Notetheincreasingdensityto-
wardsfrontalviews.b):Ourreconstructionwithoutposition,materialand
radianceinformation(onlynormaldifferences).c–f):Ourreconstruction
withallinformationincludedfordifferentmaterialsoftheobject(body,
rim,tireandwindshield).
Fig.4.Contributionofdistancestothereconstructionweights(Seetext).
Forreconstruction,theweightswaretunedbyvisualinspection ontoasphere,includingproperocclusion.
of the result, which is easy thanks to the interactive feedback
(Fig.4).Thenovelviewcanbesynthesizedas2Dimagedirectly
4 2D-TO-3D ALIGNMENT
usingEq.1,butthereconstructioncanalsobeexpressedinterms
ofReflectanceMaps[20](orLitSpheres[11]).AReflectanceMap Thenovelviewsynthesismethodfromtheprevioussectionassumes
capturestheorientation-dependentappearanceofanobject/object thatthe3Dmodelandthe2Dobjectarewellaligned.Inthissection
part.Inotherwords,itisanassociationbetweensurfacenormals wepresentanapproachto2D-image-to-3D-model-alignmentthat
andtheircolorvalues.Here,aReflectanceMapcanbecomputed requiresminimaluserinteraction.
bysynthesizingtheappearanceonasphereusingEq.1. Thealignmentcomponentcomprisesanoff-lineandanon-line
We now discuss the individual terms that contribute to the phase. In the off-line phase (Sec. 4.1), a database of 3D shapes
reconstruction:First(Fig.4,+n),ifallweightsexceptnormalare isanalyzedtolearnasetofdiscriminativeshape-viewdetectors.
zero, our approach is equivalent to the Lit Sphere model [11]. There is one discrete detector for each 3D shape in each view.
All details are missing here, except a global dependency on Tothisendallshapesarerenderedfromallviewsusingdifferent
surface orientation. Second (Fig. 4,+r), adding a reflectance illuminations.Learningthesedetectorscantakeseveralhours,but
term is equivalent to a Reflectance Map for each material. This isonlyconductedonce.Besidesluminance,ourdetectorsalsouse
is equivalent to a “continuous” Reflectance Map depending on coarse segmentations. Also in the offline phase, 3D shapes are
materialparameters.Third(Fig.4,+L),theradiancetermprefers segmentedintoparts.
appearancethatissimilarifmultipleappearancesareequivalent. Theon-linephase(Sec.4.2)proceedsinacoarse-to-fineand
This is the case for the ground plane, that is either shadows discrete-to-continuous scheme. First, the discrete 3D shape and
or unshadowed in L1 and L2, which is transferred from f1. viewforthequery2Dimagearefoundusingthepreviouslylearned
Finally (Fig. 4,+p), adding a dependency on position prefers detectors.Detectorsaccountbothforluminanceandasegmentation
localappearanceifeverythingelseissimilar.Inourexample,the produced from user input. Having found the best discrete view
cobblestonepatternthatisnotvisibleinanyfeature,butcorrelates however, will not result in a precise alignment. To this end, the
withpositionisreproduced.Note,thatwhenaddingonecomponent, viewisrefinedcontinuouslyinthenextstep.Alternativeviewsof
noothercomponentisdegraded. thebest3Dshapearerenderedon-the-flytofindtheviewresulting
The3Dmodelisfitintotheunitcubetonormalizepositions inthebestmatchtothequery2Dimage.Finally,astheclosest3D
and make them comparable for different scenes. The settings in shapeinthedatabaseisoftennotidenticaltotheoneinthequery
Fig.4arew =1,w =2,w =0.1,w =0.01,w =1.The 2Dimageduetowithinclassvariabilityintheconsideredobject
n r L p l
sharpness is set to s = 3. We keep those constant for all results categories,aper-partalignmentisfoundthatcontinuouslyrefines
reportedthroughoutthesection.Avisualizationofthesamplingin the3Dshape.
thedirectionaldomainisseeninFig.5. Producingaphysicallyorfunctionallymeaningful3Dshape
Regardingillumination,thereferenceimagesareilluminatedus- fromsuchpart-basedalignmentwouldrequireanoptimizationthat
ingone“representative“lighting(skylightwithambientocclusion), reliablyaccountsforphysicalandfunctionalpartrelationsforall
thatcanbeassumedtoalsobepresentinthe2Dimageforcars. objectsinanentire3Dshapedatabase.Insteadofaddressingthis
Forbackground,weusealargewhitebackgroundsphereforall dauntingtask,weadvocatetousethispart-basedalignmentonlyfor
3Dmodelssuchthatthefunctionp,n,r,L,etc.arealwaysdefined. improvedlocalizationwhencapturingview-dependentappearance
WhenevaluatingEq.1onbackground,distancetoforegroundis for the novel view-synthesis (Sec. 5). For each material of the
so large that it does not contribute, so similarity is based on 3D 3Dshape,directional-dependentappearanceissampledusingthe
positionandnormalalone.Effectivelythebackgroundisprojected normalonallimageareaswithpartsofthatmaterial.Thisallows
5
Shapes
e ... e Old
as ... as
h h
p p
Offline 3D m...odel Views Sh...ape-vie...w ... Online NewPos. NormalsMat. IdAppearance Radiance
database sampling Input Coarse alignment Fine alignment Part-based alignment Novel-view synthesis Result
Fig.6. Overviewofourapproachcomprisinganoff-lineandanon-linephase.Theoff-linephasesamplesshapesfromadatabaseinmanyviews.In
theon-linephase,coarse,fineandper-part2D-to-3Dalignmentisperformedwhichisbasedonacoarseimagesegmentationderivedfromuserinput
(scribbles).Finally,novelviewsaresynthesizedbyrenderingthe3Dshapeintheoldandnovelviewsandtransferringview-dependentappearance
betweenthoseimages.
to re-render the shape from novel views with fine spatial details Rendering: Toproducea2Dimageforeachshape-viewsample,
andfaithfulview-dependentappearanceinreal-time,asweshow weuseimage-basedlighting[8],fromanenvironmentmap(“Uffizi”
inSec.5. by Debevec et al. [8]) at the default orientation in proximity of
thecameraincombinationwithambientocclusion[49].Intotal,
we produce on average n = 60,000 images per class, each in
s
4.1 Off-linephase a resolution of 512×512 with 2×2 supersampling. Transparent
Input to the off-line phase is a database of 3D shapes. First, we materialsarerenderedassuch,butwithoutreflectionsorrefractions.
render2Dimagesfromtheseshapesbyrandomlysamplingview After rendering, every image is converted into a Histogram-of-
parameters.Wecallthesesamples“shape-viewsamples”(Fig.7). Oriented-Gradients (HOG)1 [52] image representation based on
Foreachsamplewetrainanexemplar-baseddiscriminativeshape- the contained object’s bounding box. The spatial sampling of
viewdetector.Inordertofacilitatethepartalignmentinthelast the HOG cells is chosen to result in approximately 150 cells –
stage, we also segment the mesh into parts that allow for a fine- correspondingtoa1350-dimensionalfeaturevector.Additionalto
granularalignmentaccountingforthevariationsobservedinthe rendering luminance, a binary foreground mask of the object is
instancesatruntime. renderedandstoredinaHOGrepresentationinthesameway.
Discriminative shape-view detectors: The construction of
a) b) View pos. Bounding
Shape box shape-viewdetectorsisbasedontheglobalmaskandluminance
Vie templates. For each template, a classifier is computed based on
apes w dir. Roll LinearDiscriminantAnalysis(LDA)[53].Theclassificationscore
h
S between a template’s HOG representation d and a test HOG
Views FOV representation x is defined as S(cid:48)(d,x) = (Σ−1(d − µ))Tx,
with Σ and µ being the covariance and mean of the negative
Fig.7.Samplingoftheshape-view-space(a)andsamplingofaviewfor data respectively. These were computed using a set of 100,000
aspecificshape(b). random patches extracted from natural images. For the mask
negative statistics we use the segmentation masks from the
Pascal VOC 2012 [54] images. Since the HOG dimension of
3D Shapes: Ourmethodisbasedonadatabaseof3Dshapes. the templates varies, it is intractable to calculate Σ and µ for
We do not require any specific (e.g., meshing) fidelity as long allpossibleHOGsizes.ThereforeΣisestimatedusingtheauto-
as the models can be rendered from many views. Therefore our covariance matrix Γ as done by Hariharan et al. [53]. To make
approachisabletotapintorichon-lineresources,whichprovide theresponsesofthedifferentexemplarclassifierscomparable,the
a vast amount of 3D shapes, capturing many shape classes with affinecalibrationapproachofAubryetal.[27]isusedbasedon
theirrespectiveintra-classvariation.Forourexperiments,weuse negativedata,producingacalibratedclassifierS =αS(cid:48)+β.
3Dmodelsfrom3DWarehouse,astheyareindexedin[48].Our
currentdatabaseconsistsof10objectclasses(AIRLINER,FIGHTER Part segmentation: Toimprovenovelview-synthesisquality,
JET, BICYCLE, DOUBLE BUS, FOLDING CHAIR, OFFICE CHAIR, weproposeper-partalignment(Sec.4.2.3).Tothisend,allinput
STATIONWAGON,SEDAN,SPORTMOTORBIKE,SOFA)thatareon 3D shapes need to be segmented into parts. Segmenting a mesh
averagerepresentedby50-250instancesforeachclass. into meaningful parts is a difficult problem that we try to avoid.
Instead, we advocate for a simple mesh segmentation approach.
Initially,themeshissegmentedintoconnectedcomponentsbased
Sampling: Wesamplefromtheseven-dimensionalspaceofall
on triangle connectivity and material index. We then iteratively
possibleviews,thatis,allviewpositions(3D),vieworientations
mergethepartwiththesmallestareaintooneofitsneighborparts
(2D)camerarollangle(1D)andfieldofviews(FOV,1D)asshown
untilaclass-specificnumberofparts(typicallyeight)remains.
inFig.7(b).Inordertoreasonablyboundthechoiceofviewsat
trainingtime,thepositionsareselectedfromashellbetweenthe
4.2 On-linephase
bounding sphere and a sphere twice as large. The orientation is
selected by picking a random point inside the bounding sphere. Inputtotheon-linephaseisa2Dqueryimageandoutputisa3D
Roll is picked randomly uniform. The FOV is randomly chosen shape,aviewmatrixandaper-partmodeltransformationmatrix.
correspondingtotherangeoffocallengthbetween20and200mm.
1.Ourapproachcanalsobeusedwithdifferentfeatures(e.g.frompretrained
Finally, the distance along the view direction is set so that the
CNNs).WechooseHOGbecauseoftheirprovenabilitytomatchdifferent
completeshapefitstheviewport. modalities[26],[27],[50],[51].
6
Theon-linephasehasthreesteps:a)coarseanddiscreteshape-view ofmagnitudefasterthanactuallycomputingadetectionresponse.
alignmentbydetection,b)fine-grainedalignmenttocontinuously We conclude that our pruning is an effective means to reduce
refinetheviewbyproducingexampleson-the-flyandc)part-based computationaleffortfor2D-to-3Dalignment.
alignmentthatindependentlyandcontinuouslymovesindividual
3Dshapeparts.Theeffectofeachstepwithrespecttothe2D-to-3D 4.2.2 Finealignment
alignmentisshowninFig.8 Thefinealignmentstartsfromthecoarsealignmentandrefinesthe
view.Furthermore,asourdatabaseshapesdonotalwaysmatchthe
Input Coarse Fine Part onesintheinputimage,weaddmorevariationstoour3Dmodel
byallowingnon-uniformscalealongthex,y andz−axes.Tothis
end, we parameterize the continuous 10-dimensional space, that
is, rotations (3D), translations (3D), FOV (1D) and model scale
(3D).Thebeststatethatminimizesacostfunctionisfoundusing
simulated annealing. Luminance and mask HOG distance to the
2D query image are used as a cost function with equal weights,
withoutanytrainingbeingperformed.Ineachiteration,thecurrent
state is changed to an alternative state by modifying 3 random
Fig.8. Effectofalignmentstepsonappearanceandgeometrymatching:
Thecoarsematchistheclosestexemplarofexamplesandfailstoalign components (out of 10) of the current state. An alternative view
eveninarigidsense.Finealignmentfindsagoodrigidalignment,but of the 3D shape is then rendered into a 2D image and the cost
stillcannotproduceamatchasthe3Dshapeisdifferentfromtheone functioniscomputed.Analternativestatethatdecreasesthecurrent
inthedatabase.Onlythepart-basedalignmentputsthearmrestsand
costfunctionisalwayskeptwhilestateresultinginanincreaseis
wheelsintherightlocationfromwhereappearancecanbetransferred.
keptaccordingtoaprobabilitybasedontheBoltzmannfactor[57].
Weassumetheinputimagetobesegmentedintoforeground Withprogressingiterations,thealternativestatedifferslessfrom
andbackground,eitherusinganautomatedapproach(e.g.semantic thecurrentstate.Resultsofthissteparearefinedviewandproject
imagesegmentation[55])orasufficientamountofuserinteraction matrices,andascaled3Dmodel.300iterationsareusedforallthe
[56]. Userinteraction isrequired toindicate wherethe objectto resultsshowninthepaper.
beliftedislocatedinthequeryimageanyway,forexamplewhen
multipleobjectsarepresentinthequeryimage.Atypicalwayof 4.2.3 Per-partalignment
selectionwouldbetodrawarough2Dboundingbox.Ourresults Inthisfinalstep,theviewisheldfixedandtheindividualpartsof
alwaysshowtheboundingboxorscribblesthatwereappliedby the3Dshapearetransformedindependently.Therefinementuses
theuserontheimagesforliftingto3D.Inallresults,theuserinput multiplepassesofsimulatedannealing.Ineachpass,arandompart
isminimalandontheorderofafewseconds. ischosenandsimulatedannealing(inthespiritoffinealignment
step) is used. For each part, we allow rotation (3D), translation
4.2.1 Coarsealignment (3D)andmodelscale(3D).Wefurtherclamptheseparameterstoa
Weapproachcoarsealignmentofa3Dmodelstoan2Dinputimage thresholdsothatonlysmallperturbationsfromfinealignmentare
withamaskbyretrievingthebestmatchingdiscreteshapeandview allowed.Again,themaskandluminanceHOGdistanceisusedto
usingthepreviouslytrainedshape-view-detectors.Ourproposed eitherkeepordiscardtheparttransformation.Weuse20passes,
speed-upsarekeytoincorporatingthischallengingsub-taskinan eachpasswith20iterationsforallresultsshowninthepaper.
interactivesystem. Theoutcomeisnotaconsistent3Dshape,aspartdeformation
neglects physical or functional part relationships. However, our
Detection: First, the input image and its mask are converted novelview-synthesis(Sec.5)onlyrequiresindividualpartstobe
to a HOG representation x and x respectively. Next, the well-placedabsolutelytocaptureappearancewhiletheirrelative
mask
alignmentscoreisthecombinationwithequalweightsofthescores placementisnotrelevant.
S(d,x) and S(d ,x ) of all samples d and d . The
mask mask mask
samplewiththebestscoreisfurtherconsideredintheprocessing.
Using a single global HOG representation is faster than e.g. the
discriminativepartbasedapproachadvocatedby[27].
Pruning: Pruningisusedtoacceleratedetection.Thenumber
of shape-view samples becomes large when a finer alignment is
desired, different FOVs are used and the number of shapes is
increased.Wefoundthesegmentationmaskcreatedfromtheuser
Proj. texturing Materials Result
click,strokeorboundingboxtobeanexcellentmeansofculling
many samples with very little effort. To this end, samples with
Fig. 9. Novel views synthesis: For every material index (2nd col.), a
boundingboxesthataredifferentinsizebymorethanoneHOG ReflectanceMapisconstructedbasedontheappearanceoftheobject
cellfromthequeryboundingboxareexcluded.Boxesofacertain intheimage(1stcol.)andthenormalsinviewspace.Theseappearance
mapsarethenusedtosynthesizeappearanceforanovelview(3rdcol.).
rangeofsizescanbefoundintimecomplexitysub-lineartothe
numberofsamples,typicallyeliminatingalargepartofthesamples
early-on.
For instance, in our simplest setting where only 128 views 5 NOVEL-VIEW SYNTHESIS REVISITED
per 3D model are considered, on average only 633 out of 6400 In Sec. 3 we presented a method to synthesize novel views of
samplesperclassneedtobetested.Thepruningaloneis1order an object with the guidance of a 3D model. While the obtained
7
resultshavehighlevelofrealism,thesynthesisisview-independent. of specularities or artifacts. In Sec. 3 such visual elements were
Thismeansthatwhenwechangeviewpointin3D,specificvisual consideredas”painted”ontheobject.
elements that are view dependent such as specularity remains
unaffected.However,inascenariothatinvolvesuserinteraction
6 EXPERIMENTS
andmanipulationoftheobjectin3D,aviewdependentsynthesisis
Inthissectionwepresentqualitativeandquantitativeresultsforthe
required.ForthisreasonweextendthesynthesismethodofSec.3
novelviewsynthesisand2D-to-3Dalignmentmethodspresented
tocaptureviewspecificinformation.
inthispaper.
The2D-to-3Dalignmentfromtheprevioussectionidentified
foranimagef theshapeS (apolygonalmesh),afinely-aligned
1
viewV(acameratransformation)andamodelmatrixM foreach 6.1 SyntheticViewpointDataset
j
partj.Wenowwouldliketosynthesizea2Dimagefromanovel WeapplyournovelviewsynthesisapproachfromSec.3inorder
viewV(cid:48) inreal-time,withdetailedappearancealsointhepartsthat togeneratenewviewsforasetof26sideviewimagesofcarsfrom
wereunobservedinthequeryimage.Inparticular,wewouldlike PASCALVOC2007[58].Here,wemanuallyalignedtheimages
theappearancetobeview-dependent,i.e.,ahighlightonacarbody with the 3D models (about 2h effort) and we generate for each
shouldnotappearasifitwaspainted,butmoveinafaithfulway image9syntheticviewsbysamplingtheviewingsphere.Examples
toconveythematerialandvisualessenceofacar. ofthesynthesizeddataareshowninFig.11.Notehoweffectsof
Tothisend,wedefinetheRGBappearanceAi(ω) ∈Ω+ → globalilluminationonthevehicleaswellasshadowbelowthecar
R3 ofthei-thmaterialfromdirectionω inviewspaceas arepreservedinmanyimages.Thedisocclusionareasarefilledin
(cid:90) (cid:90) inaplausibleandnaturalway.Thetransitionbetweenthevisible
A (ω)= γ (ω,x)f (x)dx/ γ (ω,x)dx, (2)
i i 1 i and“hallucinated”partisseamless.
where f (x) is the RGB image appearance at 2D position x ∈
1
(0,1)2 andγ (ω,x)isaweightingkernel
i
γ (ω,x)=q (x)exp(−α|1−(cid:104)Rn(x),ω(cid:105)|2), (3)
i i
where q (x) is a compatibility function that is 0 if the image at
i
position x does not belong to the mask, otherwise 1 if material
indexatxisiandfinally,0.0001ifmaterialatxisnoti.Thelast
weightisusedtoenforcecolorsformaterialsthatarenotobserved
atanyxinf .n(x)istheworldspace-normal,Ristheupper-left
1
3×3sub-matrixoftheinversetransposeofthe4×4matrixV
andαisasharpnessparameter,typicallysettoα=4.Thenovel
imagefromV(cid:48) isdefinedas
f (x)=A(cid:48)(n(x))=A (R(cid:48)n(x)), (4)
2 i i
formaterialiandtherotationalpartR(cid:48) ofV(cid:48).
Inthediscretecase,thefunctionA becomesalook-uptable
i
ofdirections(weuse32×32directions).Thistableisessentially
aReflectanceMap[20].
Eachtableentryiscomputedasaweightedsumoverallinput
imagepixelsandbasedonthenormalandmaterialidofS rendered
from V using a per-part model matrix M . Fig. 9 illustrates the
j
concept.By thealigned 3Dgeometrywe gatherdataoffthe2D
imageinordertoestimateReflectanceMapmodelsthatareused
torendernovelviews.Theavailablematerialinformationtogether
allowsforamuchcrisperrenditionandtruthfulinteractionoflight Fig. 11. Startting from PASCAL VOC input training 2D images (1st
withthematerialincontrasttoprojectivetexturing.AsA (ω)isa column)weproducearbitrarynovel-viewtrainingimages(othercolumns).
i
convexcombinationofinputpixels,wenoticedalossincontrast.
Asasolution,weperformCIELABhistogrammatchingbetween
thesmoothedhistogramofA(cid:48)andthehistogramoftheinputimage 6.2 Trainingdataaugmentation
f . Objectdetectionandclassificationapproacheshaveseensubstantial
1
Atrendertime,asinglematerial-dependentReflectanceMapis improvements over the last decade. One driving factor is the
required(Fig.10),allowingtoproduceanovelviewwithseveral availability of training data that is representative for the test
hundred frames per second in high resolutions, in our images scenariosofinterest.However,theconstructionofsuchdatasets
combinedwithambientocclusion[49]. is tedious and yet does not capture all aspects of variability in
RegardingtheReflectanceMaps,therearetwomaindifferences theclassesthatitcontains.Inparticularthesamplingofuntypical
compared to Sec. 3. First, in this section the goal is to estimate examplesorviewpointsisoftenlacking[59].Morespecifically,the
directly the entries of the Reflectance Map, while previously popularPASCALVOCbenchmark[58]providesagoodsampling
we were estimating the appearance for every disoccluded pixel ofintra-classvariationsbutwithoutexhaustiveview-pointsampling.
in the image. In addition, here we consider viewer-centered OtherdatasetslikeEPFLCars[60]providedenseviewsampling
normals. Therefore, a change of viewpoint affects the location (withoutazimuth)butdonotcaptureintra-classvariationwell.
8
Material RMs
Input Image Material Ids Synthesized
Fig.10. Theimprovednovel-viewsynthesis(Sec.5)segmentstheinputimageintopartsusingthematerialIDsofa3Dmeshasaguide.To
synthesizeanewimage,areflectancemapisextractedforeverymaterial–shownontheleft–,allowinghighspeedandview-dependentappearance.
NextweusetheaforementionedSyntheticViewpointDataset Original Rendered Synthetic
to“augment”thePASCALVOCtrainingdatainordertorepresent
theintra-classvariationtogetherwithabetterviewpointsampling
thatincludesatypicalviews.Ourstudyfocusesonthe“car”class.
Giventhisaugmenteddatasetwerunaseriesofexperimentto
underlinethevalidityofourapproach.Inallexperiments,weuse
thestate-of-the-artDeformablePartModel(DPMversion5)[61].
Fig.12.Pilotstudytrainingdata:a)originalimage,b)rendered3Dmodel,
c)oursynthesizedobject.
TABLE1
PerformanceoftheDPMdetectorwhentrainedindifferentdataandwith
differentnumberofcomponents(N).
Data Avg.precision(%) #Views improvements between 14 to 16.5% by adding our synthesized
N =1 N =3 Real Synth. views.WenowcomparethemodeltrainedonthefullVOCtraining
Side 15.4 16.2 26 - set(Full)toaversiontowhichweaddoursyntheticviews.Inthis
Rendered 11.5 12.7 - 26 setting we do not observe an improvement in average precision,
Synthetic 15.0 14.5 - 26
but rather comparable performance. However, if we inspect the
associatedprecisionrecallcurveinFig.14(left)moreclosely,we
observethatourmodelimprovesinthehighprecisionregime.It
Pilot study – Re-synthesis: As an initial test, we investigate
produces no single false positive until 15% of the positives are
howmuchthesynthesizedviewsaffecttheperformance,compared
detected(recall).
withtherealimagesanddirectrenderingsofthe3Dmodels.We
perform this study on the 26 sideviews which are resynthesized Inordertogeneratefurtherinsightsintoourmodel,weperform
usingthemethodfromSec.3.2bytreatingthevisiblepartofthe astudysimilartotheoneproposedin[59].Herewefocusonrare
carasadisocclusion.UsingthisdatawetrainaDPMdetectorand viewpointsofthecarsbyselectingasubsetofthePASCALVOC
wetestonthewholeVOCtestset.Table1showstheperformance testset where the top of the car becomes visible. Our reasoning
numbersinaverageprecision(PASCALVOCstandardcriterion) isthatthosecasesaredifficultforthestandardmodelasthispart
fordifferentnumberofcomponents(columns)intheDPM.The oftheviewpointdistributionispoorlysampled.Fig.13confirms
first line represents training on the real 26 sideviews. Then we outintuitionabouttheviewdistributionintheVOCdataset.The
repeatthetrainingbutinthiscasewehaveremovedthecarusing statisticshowsthatallviewsinvolvingthevisibilityofthetopare
the“ContextAware”-toolfromAdobePhotoshopandwereplaced under-represented.ThelowerpartofTbl.2performsananalysisfor
itwitharenderingofthe3Dmodelthatcorrespondstothattype detectionofsuchrareviewpoints.Weseestrongimprovementson
of car (Sedan, SUV or Compact) (Fig. 12). We observe that the thoserareviewpointsofupto22.6%fortrainingfromsideviews
performance is much lower, due to the lack of variation in the andupto4.8%forthefulltrainingset.
car appearance. The last row corresponds to training on the 26
resynthesized cars using our approach. The performance is very
closetothetrainingontherealcarswhichprovidesfirstevidence
TABLE2
that our method is indeed able to generate the kind of realism PerformanceoftheDPMdetectoronPASCALVOCdatasetwithvarying
thatisneededtosuccessfullytrainanobjectdetectionalgorithm trainingsetanddifferentnumberofcomponents(N).Evaluationis
performedonthefulltestsetaswellasasubsetofrareviewpoints.
withoutastronglossinperformancethatisoftenobservedinsuch
settings.
Test Train Avg.precision(%) #Views
PASCAL VOC data set: We continue by using our whole
N =3 N =4 N =5 Real Synth.
syntheticviewpointdatasetandmixingitwithdifferentportionsof
realdata.Quantitativeresultsvaryingthenumberofcomponents V Side 16.2 18.4 16.7 26 -
O Side+synth 30.2 31.4 33.2 26 728
intheDPMareshowninTbl.2andperformanceplotsinFig.14. C
Full 51.7 53.4 50.7 1250 -
2
WefirstfocusontheupperhalfofTable2wherewetestonthe 00 Full+synth 50.2 53.1 50.9 1250 728
7
standardPASCALVOCtestset.Westartbytrainingonlyonthe26
Side 11.9 11.6 10.3 26 -
V
sideviews(Side)thatweconsiderforaugmentationandcompare O Side+synth 23.2 30.2 32.9 26 728
the performance to a model on the same 26 views amplified by C Full 51.9 52.5 51.8 1250 -
728viewssynthesizedbyourapproach(Side+synth).Weobserve rare Full+synth 55.0 57.3 53.1 1250 728
9
section. For the full UCLA test set (upper half of Table 3) we
400 observedrasticimprovementsbyaddingoursynthesizeddata.For
the sideviews the improvement is between 38.9 and 44.3% and
p 300
Front-side-to Front-side Front-top Front Rear-side-top Rear-side Rear-top Rear Side-top Side Top 120000 fttarhocaercinetmhsesedodtfoouenlll4tthrs8aeeittnaibmemedeptwsolinefimeetnohdre6e2.f69ureslalaindlPdeAtsr9vSa.iiC6enw%AinsLg.(RSdeeixadmtaaemas+repsktylae(nbFstlhuy()l,2lo)o6uuwstrpihddeiecerfvhtoeicerhmwtaodssr
0 vs. 1250 real examples). The improvements can be seen more
clearlyinFig.15.Wealsoprovidetherareviewpointanalysisfor
Fig.13.ObjectsidevisibilityforcarVOC2007testdataset.
this dataset. Here the improvements are even more pronounced.
Fromtheprecisionrecallcurveweseethatthemodeltrainedon
1 1 VOC theamplifiedsideviews(Side+synth)isoutperformingthemodel
Side trained on the full set. In this case 26 real training examples in
ws
Full0.5 are vie0.5 SFuidlle+our ctoomthbeinbaetsitopnewrfiothrmoaunrcaeuognmtehnitsatdioatnam(Feuthllo+douisrse)n.oughtogetclose
R
Full+our
6.3 2D-to-3Dalignment
0 0.5 1 0 0.5 1
This section evaluates our approach in terms of quantitative
performance according to different metrics and computational
Fig.14.Performance(Average-Precision)ofthe“side”and“full”versions
ofPASCALVOCdatasetwithandwithoutnovelviewsproducedusing speed,auserstudyandvisualqualityoftherenderings.
ourapproach.EvaluationisperformedonthefullPASCALVOCtestset
(left)aswellasasubsetofrareviewpoints(right). 6.3.1 Quantitativeresults
Wecompareour2D-to-3Dalignmentapproachtopreviousmethods
UCLA data set: In order to get a more realistic estimate of in terms of different metrics. As previous methods we consider
eightalternatives:themethodofAubryetal.[26],amethodusing
theperformanceacrossviewpointsweturntotheUCLAcarsdata
lightfielddescriptors[63],amethodusingnearestneighborandfive
set[62]whichhasbeendesignedwithamoreuniformviewpoint
simplifiedvariantsofourapproach.Asmetricsweuseintersection-
samplinginmind.Thetestsetconsistsof200imagesthatcover
over-union(IoU)ofsegmentationmasks,normal-awareIoUonthe
bettertheviewingsphere.
same, and a novel metric using image differences of novel-view
rendering.
TABLE3
PerformanceoftheDPMdetectoronUCLAdatasetwithvaryingtraining
set(imagesfromVOC)anddifferentnumberofcomponents Methods: ThemethodbyAubryetal.[26]usesdiscriminative
(N).Evaluationisperformedonthefulltestsetaswellasasubsetof patches on luminance. We consider it our reference and would
rareviewpoints.
like to achieve similar or better performance, but in less time.
To quantify the effect of discriminative learning, we include
Test Train Avg.precision(%) #Views nearest-neighbormatchinginHOGspaceofluminance(NN).Light
fielddescriptors[63](LFD)areapopularapproachincomputer
N =3 N =4 N =5 Real Nov.
graphics [24] thanks to their simplicity (they are based only on
Side 41.5 43.5 41.1 26 -
U Side+synth 83.0 82.4 85.4 26 728 the silhouette of the object). We would like to know if adding
C
L Full 75.4 78.7 78.3 1250 - silhouetteinformationtotheluminance,aswellasdiscriminative
A
Full+synth 84.0 86.0 85.2 1250 728 learning, improves the alignment performance. Additionally we
U Side 40.5 44.2 39.7 26 - considerseveralincrementsinsamplingvariationforourapproach:
C
L Side+synth 82.2 81.9 85.1 26 728 atemplateLDAclassifiertrainedonthesamerenderedimagesas
A
Full 69.8 73.7 72.5 1250 -
rare Full+synth 81.4 83.6 83.3 1250 728 Asaumbpryle(sEwxeimthpleanrv),iraodndminegntvimewapporionttastiaomnpl(eEsnv(Vmieawp)),,aaddddiittiioonnaall
samples with camera roll (Roll) and finally the two steps of our
full approach (Coarse and Fine) that include mask information.
1 1
UCLA Ineachofthesestepsweincreasethenumberoftrainingimages
Side by including a certain type of variation in the rendering process
Full0.5 are views0.5 SFuidlle+our (Teh.ge.sedetnyspeersvoiefwvparoiianttiosnamarpelienxgp,edciftfeedretnotalpigphetanriningdthireercetiaolnism,eatgce.)s.
R Full+our duringtesting.
Metrics: Intersection-over-unionisaclassicmeasureofalign-
0 0.5 1 0 0.5 1
ment quality but remains agnostic to what is aligned. Therefore,
Fig.15.Performance(Average-Precision)ofthe“side”and“full”versions we also consider a normal-aware Intersection-over-union metric
of training set (VOC) with and without novel views synthesized by
extensiontothismetricwhereagreeingpixelsarenotcountedas
approach. Evaluation is performed on the full UCLA test set (left) as
1,butasthedotproductoftheunderlyingscreenspacenormals.
wellasasubsetofrareviewpoints(right).
Doingso,onlysurfaceswithasimilarorientationarecountedas
In Tab. 3 and Fig. 15 we show the performance plots of the a success, which is essential to the targeted synthesis approach.
DPM detectors trained on the same dataset as in the previous Themainobjectiveofourapproachistoproduceanimageofthe
10
Intersection-over-union Intersection-over-union (normals) Image-based
ce]110%
en Mean Bicycle Bus Chair Motorbike Car
er
ef105%
o r
o t
ati100%
R
e [
c
n
a 95%
m
or
erf
P eference[Aubry 2012] NNExemplarViewEnvmapRollCoarseFine NNExemplarViewEnvmapRollCoarseFine NNExemplarViewEnvmapRollCoarseFine NNExemplarViewEnvmapRollCoarseFine NNExemplarViewEnvmapRollCoarseFine NNExemplarViewEnvmapRollCoarseFine
R
Fig.16. Comparingperformanceofseveralvariantsofourapproachandabaseline[26].Performanceisshownasverticalbarsinunitsofratioto
thebaseline(moreisbetter).Agroupofbarsistheoutcomeononedataset,wherethefirstoneisthemeanofalltheothers.Insideeachgroup,
everytripletofbarsisavarinatofouralgorithm:nearestneighbor(NN);exemplarSVM,thesamewithincreasingvariation(view,envmap,roll)and
finallytheourfullmethod,includingfinealignment.Finally,wecompareperformanceusingdifferentmetricsencodedasdifferentcolors:Aclassic
intersection-over-union,ourextenstionusingnormalsandanimage-basedmetricthatassesstheimagesimilaritytothefinalimageproduced.
object in the input from a novel view. Therefore, we suggest to as colors in Fig. 16 and Fig. 17). Performance is stated as the
alsouseimagedifferencesbetweensynthesizednovelview-images meanoverclasses,aswellasforthefiveindividualclasses.While
asathirdmetric.Fortwo2D-to-3DalignmentsA(groundtruth) wehavealsoconductedexperimentsusinglightfielddescriptors
andB(theother),itisdefinedastheimagedistance(e.g.1−the [65],weleavethemoutoftheplotsastheyhaveshownverypoor
normalized L difference in rgb values) between a novel view- performanceofapproximately80%onaverageconsistentlyacross
2
synthesisresultimagewhenusingAandwhenusingB,divided allmetrics.
bytheunionofthepixelsthatthenovelviewscover.Toproduce In Fig. 16 we present the effect of including complementary
thenovelviews,weapplyprojectivetexturingtobothAandBand typesofvariationinthetrainingdatafor5differentclasses.We
thenrotatethem10◦ inthey axis.Thismetricdirectlymeasure show that by adding the aforementioned sampling schemes we
theeffectofalignmenterrorsonthesynthesizedimages. improve the performance w.r.t. to the reference consistently for
allmetrics.NotethatExemplarperformsbetterthanNNforthe
Intersection-over-union Intersection-over-union (normals) Image-based sametrainingdata,similartowhatisreportedin[50],[51],thus
ce] 110% weshowthecomparisonfortheinitialsettingonly.
en Fine Moreover, the results in Fig. 17 show that our accelerated
e [Ratio to refer 110050%% Reference Coarse Roll Envmap View Exemplar waoulnsiitglhytnoommucteoonsdnatescpreraricotfiueccteliidnovugsesrlteyhinreaepdauecdshrefreto-hsfrermsipemaennedocdlrei-enu(epeess.lsgau.bipn“otertoeaxrteaemmfsaspcaoltmaforrtp”io)lmi.fneT≈g.hOis6scu0har0lemlwmoowietshsst
c
orman 95% NN cporomcpeldeuxrsea(m“cpolianrgses”c)he–maech–iceovnesstiatnutiinmgporouvrefimnaelnctooafrs2e-3al%ignwmheinlet
erf stillmaintainingan≈40foldspeedup.Ourfine-alignmentstage
P 1 10 100 1000
withon-the-flyrenderinggivesusonaveragea5%improvement,
Speed-up [Time ratio to reference]
whilestillbeingroughly5timesfaster.Theresultsareconsistent
acrossthe3metricsthatwehaveinvestigated.
Fig. 17. Relation of performance shown in Fig. 16 to speedup. The
verticalaxisisinthesameunitsasFig.16:ratiotoareferencesaccording Regarding the absolute computation time of our framework,
todifferentmetricsencodetascolors.Thehorizontalaxisshowsthe mostofitisspendinfinealignment.Onaverage,0.45sarespend
speedup-ratiotothereferenceinlogscale.Thereferenceisshownin
forcoarse,2.4sforfine,and2.1sforperpartalignment.
greywithaspeedupandperformanceratioofone.Weseethatourfull
methodisbothfasterandbetter.
User study: We asked human subjects to compare our fine
alignment result to the exemplar result in order to assess the
Results: 2D-to-3D alignment was performed on 80 random
images2 from 5 subclasses of Pascal3D [64] (Imagenet subset), alignmentquality.Theexperimentcomprisedof50trials,wherein
each,participantswereshowntheresultofbothalignmentsforone
namelystationwagon,sportmotorbike,foldingchair,doublebus,
mountain bicycle, using the nine methods just described and outof10randominstancesfrom5classesofImagenet(STATION
the result was evaluated according to each of the three metric WAGON, SPORT MOTORBIKE, FOLDING CHAIR, DOUBLE BUS,
introduced.AsgroundtruthweusetheannotationfromPascal3D, MOUNTAINBICYCLE).Alignmentwasshownastwoimagesina
randomhorizontallayoutwhereineachthe3Dshapeisrendered
even if it is noisy in many cases. Performance is stated as the
usingthenormalshadingandcomposedontopofthequeryimage,
ratiototheoutcomeofthereferencemethod(Aubryetal.[27]),
asseeninthefirstcolumnofeverysetinFig.18.WhenN =28
e.g.performanceidenticaltothereferencewouldbe100%.Using
subjectswereaskedina2AFCtoindicate“whichimageshowsa
theratio,allmeasuresweproposebecomecomparable(encoded
morecorrectalignmentoftheshapetotheimage”theyreferredto
2.Alltestimagesandresultsareavailableintheprojectwebsite. ours in 59% of the cases (significant p < .0001, binomial test),