Table Of Content3D Reconstruction of Simple Objects from A Single View
Silhouette Image
XinhanDi PengqianYu
7
1 TrinityCollegeIreland NationalUniversityofSingapore
0 [email protected] [email protected]
2
n Nov2016
a
J
7 Abstract
Amongavarietyofthetechniquesaimingat3Drecon-
1
struction, 3D reconstruction from a single view is chal-
] Whilerecentdeepneuralnetworkshaveachievedpromis- lenging. Inordertoovercomethischallenge,shapepriors
V
ing results for 3D reconstruction from a single-view im- areappliedinordertocompensatefortheproblemofill-
C age, these rely on the availability of RGB textures in posedness. For example, smoothness in the definition of
. images and extra information as supervision. In this consistencyofsurfacesislearnedandappliedforthere-
s
c work, we propose novel stacked hierarchical networks construction of curved surfaces [19, 20, 21, 26, 32, 11,
[ andanendtoendtrainingstrategytotackleamorechal- 22]. Geometric relation is developed as another kind of
1 lenging task for the first time, 3D reconstruction from a priortodissolveambiguitiesinthereconstructionprocess
v single-view 2D silhouette image. We demonstrate that [18,4,6,10,11]. Highleverpriorsarealsousedtoover-
2 our model is able to conduct 3D reconstruction from a comesingle-viewreconstruction.Semanticrelationpriors
5 single-viewsilhouetteimagebothqualitativelyandquan- which infer structure information of objects are applied
7
titatively. Evaluation is performed using Shapenet for [10,16,6,11]. Also,thelearnedshapepriorsareusedfor
4
0 the single-view reconstruction and results are presented thereconstructionoffacades[16],humanbodies[2],and
. incomparisonwithasinglenetwork,tohighlighttheim- category specific objects [27, 14] through learning from
1
provementsobtainedwiththeproposedstackednetworks definedconsistencyrules.
0
7 andtheendtoendtrainingstrategy. Furthermore,3Dre-
1 constructioninformsofIoUiscomparedwiththestateof Thanks to the development of the deep learning tech-
: art3Dreconstructionfromasingle-viewRGBimage,and niques, 3D reconstruction from a single view reduces
v
i theproposedmodelachieveshigherIoUthanthestateof the dependence on classical priors through employing a
X
artofreconstructionfromasingleviewRGBimage. trainedreconstructionmodel. A3Drecurrentreconstruc-
r tion neural network(3D-R2N2) is developed to learn a
a
mapping from images of objects to the 3D shapes and
1 Introduction is presented as rebuilding 3D shape from a single image
[3], 3D Interpreter network is designed for single-view
3D reconstruction techniques rebuild 3D world from 2D reconstruction with application of two pre-trained mod-
informationandcontributionhasbeenconsistentlymade els and a projection layer during training a deep learn-
since the latest two decades. Theoretically, framework ing framework [29], the prediction of both an RGB im-
arebuiltinthisareasuchasspacecarving[17],SfM[23], age and a depth map of the object are made through a
SLAM[5]andCMVS/PMVS[8]. Practically,implemen- convolutional network [25], another joint prediction of a
tation to reconstruct the 3D world is developed such as depth map and intrinsic images from single-image input
VisualSfM[28]andCMPMVS[12]. is conducted through training a joint convolutional neu-
1
Figure1:CategorySpecific3DReconstructionfromASingle-viewSilhouetteImage.TheLeftpartrepresentssingle-
viewsilhouetteimageslocatedrandomlyonaspherearoundaobject. Therightpartrepresentsaproposedframework
toreconstructa3Dshapefromasingle-viewsilhouetteimage.
ral field(JCNF) model [15]. Stereo pair images are used rion to minimize the error for the final network which is
to train an auto-encoder without requiring a pre-training differentfromcombinedcriterionforeachnetwork.Asil-
stageforsingle-viewdepthprediction[9]. Volumetric3D lustratedinFig1,theproposedarchitectureachievesgood
object reconstruction is achieved from the designed net- performancefor3Dcategoryspecificreconstructionfrom
workwithextrainputofthepre-trainedprojectiontrans- asingle-viewsilhouetteimage,andthesilhouetteimageis
formationcode[31]. randomlylocatedaroundasphere,andpreviousnetwork
frameworkfor3Dreconstructionfromsingleviewisnot
Inthispaper,weproposeastackednetworkfor3Dre- demonstratedtodealwiththislargeviewvariation.
construction from a single-view silhouette image that is
demonstrated to be able to reconstruct category-specific
objects from a randomly located single-view silhouette
image. The overall architecture of the architecture is il- The main contributions of our work are three-fold.
lustratedinFig1. Thearchitectureconsistsoftwomajor Firstly, we proposed a stacked hierarchical network for
components:(1) the replicate layer, which works to pro- 3D reconstruction from a single view and its end-to-end
duceasingle-view2Dsegmentationto3Dvolumecoarse trainingstrategy. Andwedemonstratethatthedesignof
segmentation through simple replication operation. (2) the stacked networks architecture and its training strat-
stacked reconstruction networks, which uses the 3D vol- egy works much better than a single network. Secondly,
ume segmentation to reconstruct a 3D shape. The pro- we demonstrate that category specific 3D reconstruction
posednetworkarchitectureis trained endtoendthrough from a single-view silhouette image without RGB tex-
following a proposed training strategy for stacked hier- tures or other extra input is conducted through our pro-
archical network without a pre-training stage or separate posed model. Thirdly, we demonstrate that for 3D re-
training for each partial network. The proposed train- construction from a single-view silhouette image, single
ing strategy works as controlling the output of each net- views are randomly and widely located on a sphere sur-
work during training process of the staked hierarchical face while previous network frameworks work for small
networksthroughgradientcalculation. Inthestackedhi- view variation, such as single view images located on a
erarchicalnetworkarchitecture,itfollowsauniquecrite- circle.
2
2 Related Work
Networkisemployedasadata-drivenmethodtoprovide
usefulpriorsfor3Dreconstructionfromasingleviewim-
age recently. WarpNet is exploited to align an object in
one image with a different object in another which al-
lows single-view reconstructions with quality [13]. Vir-
tualviewnetworks(VNN)[1]arebuilttoproducesmooth
rotations through the class object collection and points
matchingforpointcloudreconstructionfromasingleim-
age.Theviewpriorofasingleimageisestimatedthrough
exploiting a deep CNN architecture with a high learning
capacity[24]. Twodeepnetworkstacksareemployedto Figure2: Stackedhierarchicalnetworks.
produceadepthpriorforasingleimagewithouttheneed
forsuperpixelation[7].
3 Stacked Hierarchical End-to-end
Furthermore, deep learning networks are applied to
Training Networks
build frameworks for 3D reconstructions from a single-
view image. Among many frameworks that are applied,
the most relevant works are learned NRSfM model with A common choice of completing complex task through
defined consistency and smoothness criterion [14], 3D- networks is a design of stack networks. And train each
R2N2 network with adding 3D convolutional LSTM in- singlenetworkinmultiplestagesthatseparatetrainingis
side [3], 3D-INN framework with usage of 2 pre-trained conductedfortrainingeachnetwork. However,thisisnot
models and a projection layer [29] , volumetric recon- strictlytrainedendtoend. However,weproposeanovel
struction of objects with pre-trained projective transfor- design and end to end training strategy for stacked net-
mationcode[31]anddepthpredictionfromCNNneural worksandemployitforthechallengingtaskof3Drecon-
networks [25, 15, 9]. A single network is built for 3D structionfromasingle-viewsilhouetteimage.
reconstructionfrommultiple-viewsilhouetteimageswith
multiple-viewinformation[30].
3.1 Networks Architecture and Training
Comparedwiththeaboveframeworks,ourframework
Strategy
is considered as advantages than the exploited networks
for3Dreconstructionfromasingleview. Firstly,despite Stacked hierarchical networks are built for training a
thatasingle-viewRGBimage[3,29,31,25]isusedasin- modeltocompleteacomplextask.
put,ourframeworkreleasesRGBdependencetoasingle-
view silhouette image. Secondly, the testedsingle views
[31, 3, 29, 25] are not largely located in space, but our Definition 3.1 Let Y = F(X) be stacked hierarchical
frameworkworkstoreconstructa3Dshapefromasingle networks where X is the input, Y is the output of the
viewrandomlyandwidelylocatedonaspherearoundthe stacked networks, F is the overall function of the stack
object, while the above frameworks achieves 3D recon- networks,nisthenumberofthesinglenetworksthatthis
structionfromanumberofsingleviewsorarbitraryviews stacked networks are comprised of. Consider F can be
around circles. Thirdly, we reduce the input of single- split as n stacked functions f ,k = 1...n and f is the
k k
viewsilhouetteimagewithoutkeypointsusedin[14].Fi- overallfunctionofkthsub-networkinthestackedhierar-
nally,webuildastackedhierarchicalnetworksandtrain- chical networks. Also consider Y = f (Y −1),k =
k k k
ingthenetworkendtoenddespiteofseparatetrainingof 1...n, Y is the output of the kth sub-network. And
k
eachnetworkorusageofpre-trainedcodes[31,29]. Y =X,Y =Y.
0 n
3
Strategy 3.1 Consider the stacked hierarchical net-
worksY =F(X),inordertoguaranteeaneffectiveend-
to-end training for stacked hierarchical networks com-
binedwithanumberofsinglenetworks. Bothamultiple
gradientdecentlearningstrategyandanuniqueerrorcri-
terionforthefinalsinglenetworkofthestackednetworks
areemployed. Thisstrategyisemployedinordertofinda
globalsolutionforstackednetworkstoovercomethedif-
ficulty of searching global optimization from end-to-end
training.
Considermultiplegradientdescentlearningstrategyas
theEq.(1)andEq.(2). Figure3: Stackedhierarchicalnetworksfortheproposed
application.
n
(cid:88)
∇F(X)= λ ∇f (Y ) (1)
k k k−1
k=1 S ,k =1,..,n−1,GT =η1×X+η2×GT,(cid:80)2 ηi =
k k k k i=1 k
∇fk(Yk−1)=|fk(Yk−1)−GTk|12 (2) 1,followingtheproposeddefinitionandstrategyandkis
thekthsinglenetwork,k ∈{1,2,...,n}. Thedeeparchi-
where∇F(X)isthemultiplegradientforthestacked
tectureforthisapplicationisillustratedinFig3.
networksduringtraining,∇f isthegradientforthekth
k
singlenetwork. Theparameterλ follows(cid:80)n λ =1.
k k=1 k
GT isthegroundtruthforthekthsinglenetwork. 3.2 Application for Category Specific 3D
k
Consider learning the unique error criterion as the Reconstruction from A Single-view sil-
Eq.(3).
houetteimage
Inordertoconduct3Dreconstructionfromasingle-view
Θ∗ =argmin|F(X)−GTn|12,Θ=θ1,...,θn (3) silhouette image, stacked hierarchical networks are pro-
Θ
posed. Followingtheproposedapplicationanddefinition
It works as learning the global optimization parame-
above, the stacked networks are combined of two single
ters set Θ∗ = θ∗,...,θ∗ for each single network in the
1 n networks. Both of the single network share the same ar-
stacked hierarchical networks. The criterion only focus
chitecture. Eachonecontainsfour3Dconvolutionlayers
on the minimization of the error of the final single net-
andfour 3Ddeconvolution layers. Thesizeof thefilters
work in the stacked networks despite of a combination
forconvolutionlayersare4×4×4,2×2×2,2×2×
criterionfortheminimizationoferrorfromallsinglenet-
2,2 × 2 × 2. The size of filters for deconvolution lay-
works.Boththestackedhierarchicalnetworkarchitecture
ersare2×2×2,2×2×2,2×2×2,4×4×4. The
andit’send-to-endtrainingstrategyareillustratedinFig2.
parameterofstrideforalllayersis2. Thenumbersoffil-
tersforthefourconvolutionlayersare64,256,512,1024.
Application3.1 ConsiderasinglenetworkY =f(X), The numbers of filters for the four deconvolution layers
where X is the input, Y is the output, GT is the ground are1024,512,256,64.
truth,f isthesinglenetworkoverallfunction,S isthear- Also, before the stacked networks, a replication layer
chitectureofthesinglenetwork. Practically,thisnetwork is added to work as replication of single-view 2D seg-
isabletoachievetheerrorboundaryE∗ =|y−y∗|p1,p(cid:61) mentation for producing 3D segmentation. In the task
1. In the real world situation, in order to increase the of 3D reconstruction from a single-view silhouette im-
learning ability of the single network, that is to reduce age, the single-view silhouette image input is 50 × 50,
the error boundary from E∗ to E(cid:48),E(cid:48) < E∗, stacked andthereplicationlayersimplyworkstoreplicatethe2D
networks are designed as Y = F(X), where S = segmentation 50 times to produce a 3D segmentation in
k+1
4
anddemonstratetheimprovementoftheproposedstacked
network both qualitatively and quantitatively. Further-
more,resultsarecalculatedinvoxelIoUinordertomake
comparison with the state of art single-view reconstruc-
tionwork.
Figure4:Twonetworkarchitectures.Thetoparethecon-
cisearchitecture(topleft)ofstackedhierarchicalnetworks 4.1 Dataset
and the detailed architecture(top right). The bottom are
theconcisearchitecture(bottomleft)ofthesinglenetwork ShapeNet is a richly-annotated, large-scale repository of
[30]andthedetailsarchitecture(bottomleft),thereplicate shapesrepresentedby3DCADmodelsofobjects. Itcon-
layerisnotrepresented. tainsmorethan3,000,000models,220,000modelsoutof
whichareclassifiedinto3135categories.Wetakeuseof2
categoriesoftheShapeNet,planes,andcarsfortheevalu-
the3Dgridspace. Whilevoxelsinthespacegetsbinary
ationoftheourreconstructionframework. ForeachCAD
value,1representsthatthevoxelisinthesegmentation,0
model of the training and testing dataset, we project a
represents that the voxel is not in the segmentation. The
CADmodelfromalargesetofsingleviewstogetsingle-
architectureofthestackeddeeparchitectureisillustrated
viewsilhouetteimagesoftheobject.
inFig4.
Furthermore,inordertotraintheproposeddeeparchi-
tecturefromendtoend. Theproposedtrainingstrategyis 4.2 Training
applied. Here,followingtheproposedtrainingstrategyin
Eq.(4),Eq.(5),Eq.(6)andEq.(7). We evaluate the proposed single-view reconstruction
framework for 2 object categories including cars and
S =S ,f =f =f (4) planes. For each object category, we randomly pick 100
1 2 1 2
different CAD models. We train 78 CAD models sepa-
rately for each object category. As presented, for each
∇f1 =|Y1−0.5×X−0.5×GT|12 (5) CAD model, 180 single-view silhouette images are pro-
ducedthroughprojectionfromviewswidelylocatedona
∇f2 =|Y2−GT|12 (6) sphere around the model. Therefore, we train 78×180
single-view silhouette images for each object category.
Similarly, for the test, we test 22 CAD models and pick
up 180 single-view silhouettes images for each model.
Θ∗ =argmin|F(X)−GT|12,Θ={θ1,θ2} (7) Therefore,22×180single-viewsilhouetteimagesaretest
Θ
for each object category. In order to measure the qual-
ityofthereconstructionshapequantitatively,wemeasure
4 Evaluation thevoxeloverlapbetweenthereconstructedshapeandthe
groundtruthshape(voxelIoU).Also,torepresenttheac-
Experiments for 3D reconstruction from a single-view curacy improvement of our stacked networks compared
silhouette image are conducted to evaluate the perfor- with the single network [30], we evaluate rebuilt shapes
mance of the proposed stacked networks and the end- produced from the two networks for comparison. As il-
to-end training strategy. To demonstrate the proposed lustrated in Fig5, for the reconstruction experiments of
model’s ability for class-specific reconstruction from a carsandplanes,wefollowtheEq.(8)
single-view silhouette image, experiments are conducted
fortwoobjectcategoriesincludingthecarandtheplane.
Wealsoconduct3Dreconstructionexperimentsforboth α =10l ,β =20l ,l ∈{1,2,...,18},l ∈{1,2,...,18}
1 1 2 2 1 2
the staked networks and the single network [30] Fig4, (8)
5
Figure5: Awidelylocatedsingleviewonasphere.
toget180singleviewsonaspherearoundeachobject.
And2Dsegmentationoftheobjectisprojectedfromeach Figure6:Reconstructionvisualizationfromahardsingle-
singleview. Fig1demonstratesanexampleforthechoice viewsilhouetteimage(Experiment1). Theleftpartrepre-
ofsingleviewspickedforacar. sents visualization for cars and the right part represents
visualizationforplanesForeachpart,thefirstcolumnrep-
resentspickedhardsingle-viewsilhouetteimageofanob-
4.3 Evaluation on Sphere-surface Located
ject. Thesecondcolumnrepresentstherebuilt3Dshape
Single-view Silhouette Images (Experi-
reconstructedfromasinglenetworkgiventhecorrespond-
ment1)
ingsilhouetteimages. Thethirdcolumnrepresentsthere-
built 3D shape reconstructed from the proposed stacked
For this test, we test 22 CAD models for each category
networks. Thefourthcolumnrepresentsthegroundtruth
and 180 single-view silhouette images are rendered for
shape.
the test of each model. A single view for each single-
view silhouette image is picked following the same rule
inthetraining.Thatistosaytahttheviewspickedfortest
isthesameviewsinthetraining. Theseviewsarewidely isnotreasonableasthesilhouetteimagesfromthesesin-
located on the sphere around of the object. Therefore, gle views do not contain necessary information, such as
there are totally 22×180 single-view silhouette images where to put the airfoil or the tail, what dose the head
fortestforeachobjectcategory. looklikewhenseenfromtheback. However,ourstacked
As illustrated in Fig6, for 3D reconstruction of cars networks are able to reconstruct good 3D shape for this
fromasingle-viewsilhouetteimage,andonlyasmallpart case,fixthetailairfoilattherightplaceandreconstructa
of 3D shape is available from a single view, such as the goodheadeventheseisnoinformationfromtheseviews.
bottom view, the top view, the front view and the back Also,thenumberofhardsingle-viewsilhouetteimagesof
view,it’schallengingtoreconstructagoodqualityof3D planesforthistestis440.
shapefroma2Dsilhouetteimagewheremostpartofthe Also, as illustrated in Fig7, for 3D reconstruction of
3Dshapeareunavailable(hardsingle-viewsilhouetteim- carsfromasingle-viewsilhouetteimage,andmostpartof
ages). It’sreasonablethatthesinglenetworkisonlyable theshapeisavailableineachsilhouetteimagefromasin-
toreconstructaverycoarse3Dshapeinthiscase. How- gle view(easy single-view silhouette image), such as the
ever,fortheproposedstackednetworkanditsend-to-end side views, the left views and the right views, the single
trainingstrategy,agoodquality3Dshapeisreconstructed networkisabletobuildareasonableshapebuttheseshape
evenmostpartoftheshapeisunavailablefromthesingle- haveobviousdrawbackssuchasbigholes,fractures,par-
viewsilhouetteimages. Wetest440hardsingle-viewsil- tialshapedamage,partialcoarseshape.However,thepro-
houetteimagestotallyforthereconstructionofcarsfrom posed stacked networks and its end to end training strat-
asingle-viewsilhouetteimage. egyhelpthereconstructiontoovercomethesedrawbacks,
Similarly, as illustrated in Fig6, for 3D reconstruction and the 3D reconstructed shapes are very close to the
of planes from a single-view silhouette image, the sin- ground truth shapes. We test 3520 easy single-view sil-
gle network is able to rebuild a 3D shape but the shape houetteimagestotallyforthereconstructionofcarsfrom
6
Figure7:Reconstructionvisualizationfromaeasysingle-
view silhouette image for cars(Experiment1). The first
columnrepresentspickedeasysingle-viewsilhouetteim-
ages of an object. The second column represents the re-
built3Dshapereconstructedfromasinglenetworkgiven
the corresponding silhouette images. The third column
represents the rebuilt 3D shape reconstructed from the
proposed networks. The fourth column represents the
groundtruthshape.
Figure8:Reconstructionvisualizationfromaeasysingle-
viewsilhouetteimageforplanes(Experiment1). Thefirst
columnrepresentspickedeasysingle-viewsilhouetteim-
ages of an object. The second column represents the re-
built3Dshapereconstructedfromasinglenetworkgiven
asingle-viewsilhouetteimage. the corresponding silhouette images. The third column
represents the rebuilt 3D shape reconstructed from the
proposedstackednetworks.Thefourthcolumnrepresents
Also, as illustrated in Fig8, for 3D reconstruction of
thegroundtruthshape.
planesfromasingle-viewsilhouetteimage,andmostpart
of the shape is available in each silhouette image from
a single view, the single network is able to rebuild most
part of 3D shape of a plane and some parts are missing
such as the tail, the wing, the head, the back. Also, it
sometimesfixthetailatthewrongposition. However,the
proposed stacked networks and its training strategy help
the reconstruction to solve these issues and rebuild good
shapes. Also, the number of easy single-view silhouette
imagesofplanesfortestis3520.
7
4.4 Evaluation on Sphere-surface Ran-
domly Located Single-view Silhouette
Images(Experiment2)
For this test, we test 22 CAD models for each category
and 180 single-view silhouette images are rendered for
thetestofeachmodel. Asingleviewforeachsilhouette
imageispickedfollowingadifferentruleEq.(9)fromthe
training to ensure the all the views are randomly located
onaspherearoundaobjectandtheseviewsaredifferent
with the trained views. Where l ∈ {1,2,...,18},l ∈
1 2
{1,2,...,18},γ ∈(0,1)isarandomnumber.
α =10(l −γ),β =20(l −γ) (9)
1 1 2 2
Thisexperimentismorechallengingthanexperiment1
asthatthesilhouetteimagesareprojectedfromrandomly
located single views on the sphere, and the views are
different from the views used for training. And we test
22×180 single-view silhouette images in total for each
objectcategory.
As illustrated in Fig9, for reconstruction of cars from
a single-view silhouette images when the views are to-
wardsrandomlylocated. andforasingleviewthatsmall
Figure 9: Reconstruction visualization from a ran-
partofshapeisavailableoralargepartofshapeisavail-
domly located single-view silhouette image for
able,thesinglenetworkisabletoreconstructasmallpart
cars(Experiment2). The first column represents picked
ofthecar,rebuildacoarse3Dshapethatisfarfromacar
single-view silhouette images of an object. The second
structure. Alsothesinglenetworksometimesfailstopro-
column represents the rebuilt 3D shape reconstructed
duceareasonableshapeandmuchnoiseoccursinthegrid
fromasinglenetworkgiventhecorrespondingsilhouette
space. However, thestackednetworksareabletorecon-
images. Thethirdcolumnrepresentstherebuilt3Dshape
structgoodqualityofshapeswithoutthosedrawbacks.
reconstructed from the proposed stacked networks. The
Similarly, asillustratedinFig10, forreconstructionof
fourthcolumnrepresentsthegroundtruthshape.
planesfromasingle-viewreconstructionwhentheviews
arerandomlylocatedonaspheresurface,andforasingle-
viewsilhouetteimagefromviewswherebothasmallpart
of shape or a large part of shape is available, the sin-
glenetworkrebuildplanesthatpartsoftheplanesuchas
wings,tailaremissingorthesepartsarefixedatthewrong
position. Also, it sometimes fails to reconstruct reason-
able shapes of planes. However, the proposed stacked
networksreconstructgoodqualityofshapeswhichover-
comestheseissues.
8
3D-R2N2 S-Hard-E1 S-All-E1 SS-Hard-E1 SS-All-E1 S-All-E2 SS-All-E2
Cars 0.798 0.699 0.765 0.817 0.828 0.601 0.686
Planes 0.513 0.373 0.469 0.473 0.474 0.384 0.430
Table1: AverageIoUforcarcategoryandplanecategory. Inthetable,Srepresentsthesinglenetwork,SSrepresents
theproposedstackednetworks,E1andE2representsexperiment1andexperiment2,Hardrepresentshardsingle-view
silhouetteimages,Allrepresentssilhouetteimagesforalltestsingleviews.
Figure 10: Reconstruction visualization from a ran-
domly located single-view silhouette images for
planes(Experiment2). Thefirstcolumnrepresentspicked
single-view silhouette images of an object. The second
column represents the rebuilt 3D shape reconstructed
fromasinglenetworkgiventhecorrespondingsilhouette
images. Thethirdcolumnrepresentstherebuilt3Dshape
reconstructed from the proposed stacked networks. The
Figure 11: Voxel IoU measure of a single network and
fourthcolumnrepresentsthegroundtruthshape.
stacked networks for experiment1. For cars, voxel IoU
for each hard single-view silhouette image are repre-
sented(top). Similarly, the IoU for hard single-view sil-
houetteofplanesarerepresented(bottom).
9
4.5 Comparison between Stacked workforeachsingle-viewsilhouetteimage.
Networks,3D-R2N2,SingleNetworks. AsillustratedinTab.1,fortheexperimentofrandomly
locatedsingle-viewsilhouetteimages,74.8737%of3960
Comparison between the single network [30], proposed single-view silhouette images of cars achieves high IoU,
stacked networks and the state of art 3D-R2N2 [3] is thepercentageforplanesis52.8283%of3960randomly
made. Reconstruction results are calculated in voxel locatedsingle-viewsilhouetteimagesofplanes. TheIoU
intersection-over-Union(IoU) for the above two experi- valueisabout0.07lowerthan3D-R2N2[3]. Weconsider
ments. Experimentsareconductedwiththesameconfig- contribution as the reconstructed shape is in good qual-
uration for the single network and stacked networks ex- ity as illustrated in Fig9 and Fig10. Also, the proposed
pect3D-R2N2whichworksfortheinputwithRGBtex- networks work from single-view silhouette images with-
tures.As3D-R2N2worksforRGBimagesrenderedfrom outRGBtexturesandtheviewsarewidelyandrandomly
Shapenet [3] as input in a specific sequence which dose located around a sphere in spite of a circle [3]. Further-
not works for a single-view silhouette image. We com- more,inthisexperiment,alltheviewsaredifferentinthe
pareIoUoftheexperimentsfromboththesinglenetwork training.
and the stacked networks with the same object category Also, in order to ensure that the all the average IoU
test [3] in Shapenet. And also compare the average IoU values are credible, we also calculate the standard error,
forthesameobjectcategoryfromShapenetthat3D-R2N2 andthestandarderrorisaround10−4whichisverylow.
getsfromasingle-viewRGBimage[3].
As illustrated in Tab.1, the average IoU value of the
stackednetworksishigherthan3D-R2N2forcarsinthe 5 Conclusion
experiment1. Althoughthestackednetworksachievesthe
sameIoUleverforplanesthatis0.04lowerthan[3],but Firstly, the proposed reconstruction framework is repre-
weconsiderourproposedframeworkisabletobuildob- sented to reconstruct good quality 3D category specific
ject category specific reconstruction from a single view shapefromasingle-viewsilhouetteimage,whilethestate
silhouette image in a good quality also as represented ofartof3Dreconstructionfromasingleviewdependson
in Fig6, Fig7, Fig8, Fig9 and Fig10. And the network RGB textures and other extra input. To our best knowl-
is demonstrated to be able to do 3D reconstruction from edge, for object specific category, we firstly demonstrate
single-view with reduced dependence from RGB texture thatnetworkshelpthetask,3Dreconstructionfromsingle
images to 2D silhouette images. Furthermore, the diffi- view, reduce the dependence of RGB textures to binary
cultyofexperimentinourworkishigherthan3D-R2N2 segmentation.
[3] that the input is reduced to a single-view binary sil- Secondly, a novel network design is proposed for
houetteimage. Andtheviewsgetmorevariationthatthe stacked hierarchical networks combined of a number of
renderedRGBimagesareprojectedrandomly5timeson singlenetworks. Alsoanend-to-endtrainingstrategyfor
a circle [3], while the single-view silhouette images are the stacked networks are proposed to help the complex
rendered180timesfromwidelylocatedviewsonasphere stacked networks find global optimization solution dur-
whichistowardsrandomsingle-viewconfiguration. ing training. As the application of the 3D reconstruc-
Also, the stacked networks achieve higher IoU value tion from a single-view silhouette image demonstrates,
than single network Tab.1. Even the increased IoU is thiscontributionimprovesthenetworks’abilityinthisre-
about 0.1 and 0.07 higher, but as the Fig6, Fig7, Fig8, constructiontasktoovercomemanydrawbacksandeven
Fig9 and Fig10 demonstrate, this improvement demon- failures. Further work to the development of complex
strates that the proposed stacked network reconstruct stackednetworksinthisend-to-endtrainingdirectionmay
much higher quality shapes than the single network. In bepromising.
detail,wealsodemonstrateeachIoUforthehardsingle- Thirdly, the proposed networks are demonstrated to
viewsilhouetteimageinexperiment1betweenthesingle build 3D reconstruction from a single view that are both
networkandthestackednetworksFig11.Thestackednet- widely and randomly located on a sphere around a ob-
worksachievesdistinctlyhigherIoUthanthesinglenet- ject. Whiletoourbestknowledge, othernetworkframe-
10
