A Learning-based Variable Size Part Extraction Architecture for 6D Object Pose Recovery in Depth

4.2 MB·

by  Caner Sahin

1
A Learning-based Variable Size Part Extraction
Architecture for 6D Object Pose Recovery in Depth
Caner Sahin1, Rigas Kouskouridas2 and Tae-Kyun Kim1
Abstract—State-of-the-art techniques for 6D object pose of the objects with simple depth pixels [8] and integrate them
recovery depend on occlusion-free point clouds to accurately into random forests. Most particularly, iterative random forest
register objects in 3D space. To deal with this shortcoming, we
algorithms such as Latent-Class Hough forest (LCHF) [15]
introduce a novel architecture called Iterative Hough Forest with
anditerativeMulti-OutputRandomforest(iMORF)[9]obtain
7 Histogram of Control Points that is capable of estimating the
1 6D pose of occluded and cluttered objects given a candidate 2D state-of-the-art accuracy on pose estimation. On the other
0 bounding box. Our Iterative Hough Forest (IHF) is learnt using hand, these methods rely on scale-invariant features, while
2 parts extracted only from the positive samples. These parts the exploitation of rich discriminative information inherently
are represented with Histogram of Control Points (HoCP), a embeddedintothescale-variabilityisoneimportantpointbeen
n “scale-variant” implicit volumetric description, which we derive
a from recently introduced Implicit B-Splines (IBS). The rich overlooked.
J discriminative information provided by the scale-variant HoCP Unlike the aforementioned learning-based methods, the

9 features is leveraged during inference. An automatic variable ones presented by Novatnack et al. [10], [11] utilize the
size part extraction framework iteratively refines the object’s detailed information of the scale variation in order to register

] initial pose that is roughly aligned due to the extraction of
range images in a coarse-to-fine fashion. Although promising,
V
coarsest parts, the ones occupying the largest area in image
they extract and match conventional salient 3D key points.
C pixels. The iterative refinement is accomplished based on finer
(smaller) parts that are represented with more discriminative However, real depth sensors have several imperfections such
.
s control point descriptors by using our Iterative Hough Forest. as missing depth values, noisy measurements, foreground
c Experiments conducted on a publicly available dataset report fattening, etc. As a result, salient feature points used in
[
that our approach show better registration performance than [10] tend to be located on these deficient parts of the depth
1 the state-of-the-art methods. images, and hence, they are rather unstable [12]. In such a
v
6 Index Terms—Object registration, 6 DoF pose estimation, scenario,3Dreconstructionmethodsthatprovidemorereliable
scale-variant HoCP features, one class training, random forest, shapeinformationcanbeutilized[6].ImplicitB-Splines(IBS)
6
1 iterative refinement. [7], [13] are techniques that can provide shape descriptors
2 through their zero-sets and reconstruct surfaces. They are
0 I. INTRODUCTION based on locally controlled functions that when combined
1. Object registration is an important task in computer vision with their control points produce a very rich part-based object
0 that determines the position and the orientation of an object representation.
7 in camera-centered coordinates [33]. An object of interest Our architecture is originated from these observations. We
1
that was detected beforehand in a coarse 2D bounding box integrate the coarse-to-fine registration approach presented in
:
v is fed into a registration system that can superimpose the [10]intotherandomforestframework[7],[9]usingHistogram
Xi desired translation and rotation of the object onto the raw ofControlPoints(HoCP)featuresthatweadaptfromrecently
camera image. By utilizing such a system, one can propose introduced IBSs [13]. We train our forest only from positive
r
a promising solutions for various problems related to scene samples and learn the detailed information of the scale-
understanding, augmented reality, control and navigation of variability during training. We normalize every training point
robotics, etc. Recent developments on visual depth sensors cloud into a unit cube and then generate a set of scale-space
andtheirincreasingubiquityhaveallowedresearcherstomake images, each of which is separated by a constant factor. The
useoftheinformationacquiredfromthesedevicestofacilitate parts extracted from the images in this set are represented
challenging registration scenarios. with the scale-variant HoCP features. During inference, the
Iterative Closest Point (ICP) algorithm [1], point-to-model parts centered on the pixels that belong to the background
based methods [2], [3] and point-to-point techniques [4], [5] andforegroundcluttersareremovediterativelyusingthemost
demonstrate good registration results. However, the perfor- confident hypotheses and the test image is updated. Since this
mance of these approaches is severely degraded in cases of removal process decreases the standard deviation of the test
heavyocclusionandclutter,andsimilarlookingdistractors.In point cloud, subsequent normalization applied to the updated
ordertoaddressthesechallenges,severallearningbasedmeth- testimageincreasestherelativescaleoftheobject(foreground
odsformulateocclusionawarefeatures[6],derivepatch-based pixels) in the unit cube. More discriminative control point
(local) descriptors [15] or encode the contextual information descriptors are computed at higher scales and this ensures
the refinement of the object pose. In our prior work [14], we
1Imperial Computer Vision and Learning Lab (ICVL), Imperial College
have evaluated the registration performance of the proposed
London, 2Wirewax {c.sahin14, tk.kim}@imperial.ac.uk,
[email protected] architecture by only using fixed-size parts. We extend the
2
n
o
citamotuaelbaeirzaisv-noitcartxe trap initial regis. t.r a.tion ts1 .reti ...iterative posets a1-l re.retifineme tsalnt.reti... . . . tsalnoitareti ts a1-lnoitaretiiterative pose refitsnoitare1t inement initartsiger laitiniitial reg.
de duno . . .
zilamronolc tniopitareneg . . . OUTPUTtsa1-l.reti ts.re1t i
T
U
P
N
I
Fig.1. Sampleresultofourarchitecture:theobjectofinterest(lower-leftcorner)isfirstroughlyalignedbyextractingcoarsestparts,theonesoccupyingthe
largestareainimagepixels.Thisalignmentistheniterativelyrefinedbasedonfiner(smaller)partsthatarerepresentedwithmorediscriminativedescriptors
(TheRGBimageoftheobjectofinterestisforvisualizationpurposes,thecolor-codedpartsarecenteredonthesamepixels).
work engineering an automatic variable size part extraction methods to the learning based approaches. Iterative Closest
framework in such a way that we can further exploit the Point (ICP) algorithm, originally presented in [1], requires
discriminative information provided by the HoCP features. a good initialization in order not to be trapped in a local
This framework first roughly aligns the object of interest minimum during fine tuning. This issue is addressed in [20]
by extracting coarsest parts, the ones occupying the largest providing globally optimal registration by the integration of
area in image pixels, and then iteratively refines its alignment a global branch-and-bound (BnB) optimization into the local
based on finer (smaller) parts that are represented with more ICP. The point-wise correspondence problem is converted
discriminativecontrolpointdescriptors.Notethatweemploya into a point-to-model registration in [2]. The object model
compositionalapproach,i.e.,weconcurrentlydetecttheobject is represented with implicit polynomials (IP) and the distance
in the target region and estimate its pose by aligning the parts between the test point set and the object model is minimized
in order to increase robustness across clutter. Figure 1 depicts via the Levenberg-Marquardt algorithm (LMA). Zheng et al.
a sample result of our architecture. To summarize, our main [21] propose a 6 DoF pose estimation technique utilizing 3D
contributions are as follows: IPsonultrasoundimages.Intheoff-linephase,objectmodelis
• To the best of our knowledge this is the first time an im- represented with 3D IPs and by utilizing its gradient flow, 2D
plicit object representation, Implicit B-Spline, is adapted ultrasoundimageisregisteredintheon-lineprocess.In[22],a
into a “scale-variant” part descriptor and is associated coarse-to-fine fast IP-driven registration method is presented.
with the random forests. A rough pose estimation is quickly acquired with a coarse IP
• We introduce a novel iterative algorithm for the Hough model (low degree curve fitting) and finer models refine the
forests: it finds out an initial hypothesis and improves its parametersofthisroughestimation(highdegreecurvefitting).
confidence iteratively by extracting more discriminative Hinterstoisser et al. [30] extract holistic templates from 3D
“scale-variant” descriptors due to the elimination of the models of the objects and match to the scene at test time.
background/foreground clutter. These studies have demonstrated good registration results on
• We engineer an automatic variable size part extraction thetargetpointsetsthatareocclusion-freeand/oraresubjected
framework for the random forests: it first roughly aligns to the artificial Gaussian noises and outliers.
the object of interest by extracting coarsest parts and it- Unlike the abovementioned methods, more realistic regis-
erativelyimprovesitsconfidencebasedonfiner(smaller) tration scenarios have been addressed by the point-to-point
ones. techniques that build point-pair features for sparse represen-
The rest of the paper is organized as follows: In Sect. II, tations of the test and the model point sets [24], [25], [32].
we present a review on the object registration. Section III Rusu et al. align two noisy point clouds of real scenes by
demonstratesthecomputationprocedureoftheHoCPfeatures finding correct point-to-point correspondences between the
as a scale-variant part representation, their combination with PointFeatureHistograms(PFH)andfeedthisalignmenttoan
the Iterative Hough Forest (IHF), and the registration process. ICP algorithm for fine tuning [23]. The cluttered and partially
Experimental results are provided in Sect. IV and finally, occludedobjects’posesarehypothesizedbyaccumulatingthe
the paper is concluded in Sect. V with several remarks, and votesofthematchingfeaturesin[24].Choietal.[25]propose
discussions. point-pair features for both RGB and depth channels that are
II. RELATEDWORK conducted in a voting scheme to hypothesize the rotation and
Alargenumberofstudieshavebeenproposedfortheobject translation parameters of the objects in the cluttered scenes.
registration,rangingfromthepoint-wisecorrespondencebased Thefeaturesproposedin[5]makeuseofthevisibilitycontext
3
of the scene to tackle the registration. Despite achieving good tiondegradesthegeneralizationanditisoneoftheremaining
registration results, these techniques underperform when the challenges that should be addressed, apart from occlusion,
scenes are under heavy occlusion and clutter, and the objects’ clutter and/or similar looking distractors. Beyond object pose
geometry are indistinguishable from background. estimation [14], [15], there are several part-based solutions
Learning-based methods have good generalization across proposed for different tasks such as human pose recognition
severe occlusion and clutter [6], [8], [26], [27]. The method [16],[17],3Dfaceanalysis[18],orhandposeestimation[19]
presented in [6] formulates the recognition problem globally to name a few. They experience different part sizes and select
and derives occlusion aware features. A set of principal the one that performs best, however, none of these solutions
curvatureratiosarecomputedforallpixelsindepthimagesto investigatehowextractingvariablesizepartscanbeutilizedin
extract the edgelets. In [8], the contextual information of the a single framework. In this study, we investigate the effect of
objects is encoded with simple depth and RGB pixels. This this size variation and show that the simultaneous utilization
technique improves the confidence of a pose hypothesis using of the parts of varying size can improve 6D object pose
a Ransac-like algorithm. Cabrera et al. [27] back project the estimation, especially in heavily occluded and cluttered depth
parts inside the initially found coarse bounding box to the maps, supplying a rich source of discriminative information.
image and pass down the forest again. The parts with the
lowest contributions are penalized in such a way that finer III. OURREGISTRATIONAPPROACH
registration is produced.
In this section we detail our registration approach by firstly
The state-of-the-art accuracy on registration is acquired
describing the computation procedure of the HoCP features
by the iterative random forest algorithms. The part-based
as a scale-variant part representation. We then present how to
strategy, Latent-Class Hough Forest [15], refines the initially
encode the discriminative information of these scale-variant
hypothesized object pose by iteratively updating the object
featuresintotheforest.Finally,wedemonstratehowtoexploit
class distributions in the leaf nodes during testing. Iterative
the learnt shape information in a coarse-to-fine fashion to
Multi Output Random forest (iMORF) [9] jointly predicts the
refine the pose hypotheses.
head pose, the facial expression and the landmark positions.
The relations between these tasks are modelled so that their
A. Scale-variant Part Representation: Histogram of Control
performances are iteratively improved with the extraction of
Points
moreinformativefeatures.Theideas,iterativeposerefinement
during testing and iterative extraction of more discriminative Given a positive depth image we initially normalize it into
features, form a basis for our Iterative Hough Forest (IHF) a unit cube and then new point clouds at different scales are
architecture: during training, we encode discriminative shape sampled as follows:
information of the HoCP features into the forest. Despite X −X¯
that the skeleton of our training procedure is similar to the {X } = n×3 n×3 +0.5, i=0,1,2,...,m (1)
N i s ∗α
methodologyin[15],ourforestlearnsthediscriminativeshape i
informationthatwillbeiterativelyexploitedattesttime.Inthe with
 
courseofinference,unlike[15],weupdatethetestimageitself max(X)-min(X)
 
and the hypotheses confidence by a noise removal process α=max max(Y)-min(Y) ,h =max(Z )−min(Z )
i Ni Ni
that allows us to extract more informative features from the  max(Z)-min(Z) 
(2)
test images. Whilst these approaches [9], [15] rely on the
where X = [X,Y,Z] is the world coordinate vector of
scale-invariant features to improve the confidence of a pose
the original foreground point cloud, X¯ is the mean of X,
hypothesis, Novatnack et al. [10], [11] introduce a framework
X = [X ,Y ,Z ] is the normalized foreground pixels,
that registers the range images in a coarse-to-fine fashion N N N N
m is the number of the scales, α is the scale factor and h
by utilizing the detailed information provided by the scale
is the scale. The constant s takes real numbers to generate
variation. The shape descriptors with the coarsest scale are i
the point clouds at different scales, starting from s = 1 that
matched initially and a rough alignment is achieved since 0
corresponds to the initial normalization. A training image and
fewer features are extracted in coarser scales. The descriptor
its samples at different scales are shown in Fig. 2 (a).
matchingathigherscalesproducesimprovedpredictionsofthe
Oncewegenerateasetofscale-spaceimages,werepresent
pose. Inspired by [10], we design a “scale-variant” implicit
these point clouds first globally with the control points of Im-
volumetric part description, “Histogram of Control Points
plicitB-Splines(IBS).IBSisdefinedthroughthecombination
(HoCP)”, and associate it with the random forest framework.
of B-Spline tensor products:
Selecting the part size is important since larger parts tend
N
to match the disadvantages of a holistic template while (cid:88)
f(x)= n B (x)B (y)B (z) (3)
smaller ones are prone to noise [15]. In heavily occluded i,j,k i j k
i,j,k=1
and cluttered scenes, relatively smaller parts perform well
whilstthelargeronesaremoreconvenientinocclusion/clutter- where {ni,j,k} are the coefficients defining a control lattice of
free scenarios. Discriminative information encoded into small sizeN×N×N andBi(x),Bj(y),Bk(z)arethesplinebasis
sized parts might not be fully exploited by larger parts, most functions.Thisdefinitioncanbereformulatedasthefollowing
particularly when the object representation is scale-variant. inner product:
Hence,thisapplication-specific/task-dependentpartsizeselec- f(x)=nTe(x)=e(x)Tn (4)
4
Normalized point clouds at different scales Variable size part extraction Histogram of Control Points (HoCP) representation
The training depth image is
normalized into a unit cube and increase in the scale
then new point clouds at
different scales are sampled
(rotated for better visualization,
az: 113, el: -22). . . .
. . .
training depth image
viewpoint: viewpoint:
(a) (b) az : 39, el: -68 (c) az : 76, el: -18 (d)
Fig.2. ComputationprocedureoftheHoCPfeaturesasascale-variantpartrepresentation:(a)initialnormalization(s0=1)ofthetrainingdepthimageis
theoutmostpointcloudandtheinneronesaresampledbydifferentsi values.(b)globalrepresentationofthescale-spaceimages.(c)extractionprocessof
thevariablesizeparts(centersofthesepartsarethesame).(d)HoCPrepresentationofthepartsextractedin(c).
where the coefficient vector n includes the control values control points to show the increase in their discriminative
{n } and the basis vector e(x) depends on the given power as the scale gets higher. Note that IBS resolution
i,j,k
data point since it sorts the spline basis function products N determines the complexity of the representation (level of
{B (x)B (y)B (z)}.ThebasisvectorsinEq.4arecomputed detail) in a unit cube, whilst the scale indicates the relative
i j k
for the whole point cloud and the coefficient vector n is size of the object with respect to the unit cube dimensions.
calculated based on the 3L algorithm [31]. Rouhani et al. In our architecture, despite sampling point clouds at different
[13] construct the spline basis functions B (x), B (y), B (z) scales h , (e.g. i = 0,1,...,8), we fix the complexity of the
i j k i
through the following blending functions: representation, (e.g. N =50).
IBS is the combination of the locally controlled functions
b (u)=(1−u3)/6, b (u)=(3u3−6u2+4)/6
0 1 (5) and allows one to propose effective part-based solutions for
b2(u)=(−3u3+3u2+3u+1)/6, b3(u)=u3/6 object registration. We benefit from such a property and
partition the globally represented scale-space depth maps into
and reformulate Eq. 3 in order to determine the control point
parts. We express the part size g in image pixels which also
vector n of the point clouds that are normalized into the unit
cube [0 1]3: depictstheratiobetweenthesizesoftheextractedpartandthe
boundingboxoftheglobalpointcloud.Inourpriorwork[14],
3
f(x)= (cid:88) n b (u)b (v)b (w) (6) wehaveextractedandrepresentedthepartsthathavethesame
i+l,j+m,k+p l m p
size, that is, the parts growing around every individual pixel
l,m,p=0
at each scale occupy the same area in image pixels. We now
where
extend the work extracting the parts that are different in size.
A 3D bounding box defined in metric coordinates is traversed
i=(cid:100)x/∆(cid:101), j =(cid:100)y/∆(cid:101), k =(cid:100)z/∆(cid:101)
x x y y z z in the unit cube of each scale-space image and the parts are
u= −(cid:98) (cid:99), v = −(cid:98) (cid:99), w = −(cid:98) (cid:99) extractedaroundnon-zeropixels.Thetotalnumberofthedata
∆ ∆ ∆ ∆ ∆ ∆
points in this 3D bounding box varies for the point clouds at
∆=1/(N −3).
different scales, and consequently, the size of the extracted
Thus,theunitcubeissplitintoanN×N×N voxelgridwhere parts differs. Figure 2 (c) shows an example of variable size
N istheIBSresolution.Eachcontrolpointinnisdefinedwith partextractionprocessinwhichtheredpartsaregrownaround
an index-weight pair: the index number indicates the vertex thesamedatapoint.Notehowthepartsizedecreaseswhenthe
of this grid at which the related control point is located. The scale of the normalized object point cloud gets higher, since
weightinformsthedescriptorsignificanceaboutthecontrolof less number of data points are deployed in the 3D bounding
thegeometrytoberepresented.Thescale-spaceimagesinFig. box. Each part has its own implicit volumetric representation,
2 (a) are globally represented in Fig. 2 (b) with the control formed by the closest control points to the part center, the
point descriptors. We use all control points to represent the ones lying inside the 3D bounding box along depth direction.
structures, but one can sort these descriptors based on their Suchapartdescriptioncharacterizesthelocalityinacascaded
weights and utilize the ones higher than a threshold. In Fig. fashion,growingregionswithdifferentcharacteristicsarounda
2 (b), the point clouds are lastly reconstructed by using the
5
point.Weencodethisinformationintohistogramsinspherical and clutter [7]. These perturbations most likely occur on
coordinates. Each of the part centers is coincided with the patches extracted along depth discontinuities such as the
centerofasphere.Thecontrolpointsofthepartaredescribed contoursoftheobjects.Theyforceatestpatchtodiverge(oc-
by the log of the radius t , the cosine of the inclination t cluded/cluttered)fromitspositivecorrespondencebychanging
r θ
and the azimuth t . Then, the sphere is divided into the bins its representation, r of the sphere, and the histogram
φ max
and the relation between the bin numbers ν ,ν ,ν and the coordinates consequently.
r θ φ
histogram coordinates t ,t ,t is given as follows [20]: A group of candidate split functions are produced at each
r θ φ
ν r
t = r log( ) node by using a set of randomly assigned patches {Ti} and
r log(rrmmainx) rmin thresholds{τi}.Theonethatbestoptimizetheoffsetandpose
t =ν z (7) regression entropy [21] is selected as the split function. Each
θ θr tree is grown by repeating this process recursively until the
ν y
t = φtan−1( ) forest termination criteria are satisfied. When the termination
φ 2π x
conditions are met, the leaf nodes are formed and they store
where rmin and rmax are the radii of the nested spheres votes for both the object center ∆x=(∆x,∆y,∆z) and the
with the minimum and the maximum volumes, x,y,z are the object rotation θ =(θ ,θ ,θ ).
r p y
Cartesian coordinates of each descriptor with radius r. rmax Depending on the part extraction approach, all parts in P
equalstothedistancebetweenthepatchcenterandthefarthest (see Eq. 8) can either be of the same size or of the variable
descriptor of the related patch. The numbers of the control size. From now on, we will refer to the forests trained on
points in each bin are counted and stored in a d=νr∗νθ∗νφ variable size parts as the IHF-variable size, and to the ones
dimensionalfeaturevectorf.Figure2(d)illustratestheHoCP learnt by using fixed size parts as the IHF-fixed size.
representationsofthepartsextractedinFig.2(c).Notethatthe
controlpointscomputedathigherscalescapturemoredetailed
C. 6D Object Pose Estimation
part geometry.
Onceweencodethediscriminativeinformationofthescale-
variantHoCPfeaturesintotheforest,wenextdemonstrate6D
B. The Combination of HoCP and Iterative Hough Forest
pose estimation of objects considering that the learnt forest is
TheproposedIHFisthecombinationofrandomizedbinary IHF-variable size.
decision trees. It is trained only on foreground synthetically Theproposedarchitectureregistersobjectsintwosteps:the
rendereddepthimagesoftheobjectofinterest.Wegeneratea initialregistrationandtheiterativeposerefinement.Theinitial
set of scale-space images from each training point cloud and registration roughly aligns the test object and this alignment
sample a set of parts {Pi} as explained in subsection III-A is further improved by the iterative pose refinement.
and annotate those as follows: Consider an object that was detected by a coarse bounding
box, I , as shown in the leftmost image of Fig. 3 (a). At an
P ={P }={(c ,∆x ,θ ,f ,D )} (8) b
i i i i i i
iteration instant k, the following quantities are defined:
where ci = (cxi,cyi) is the part center in pixels, ∆xi = • ∆x0:k = {∆x0,∆x1,...,∆xk} = {∆x0,∆x1:k}: the
(∆x ,∆y ,∆z ) is the 3D offset between the centers of
i i i history of the object position predictions.
the part and the object, θi = (θri,θpi,θyi) is the rotation • θ0:k = {θ0,θ1,...,θk} = {θ0,θ1:k}: the history of the
parameters of the point cloud from which the part P is
i object rotation estimations.
extracted and Di is the depth map of the part. • V1:k = {v1,v2,...,vk} : the history of the inputs (noise
EachtreeisconstructedbyusingasubsetS oftheannotated
removals) applied to the test image.
training parts S ⊂ P. We randomly select a template patch • M0:k = {M0,M1,...,Mk} = {M0,M1:k}: the his-
T from S and assign it to the root node. We measure the tory of the set of the feature vectors where Mk ={f }.
i
similarity between T and each patch Si in S as follows: • hk: the object scale (the scale of the foreground pixels)
• Depth check: The depth values of the descriptors Sin in the unit cube at iteration k (see Eq. 2).
and Tn that represent the parts Si and T are checked, • gk: the size of the parts extracted at iteration k.
and the spatially inconsistent ones in Sn are removed as
i We formulate the initial registration as follows:
in [7], generating Ω that includes the spatially consistent
descriptors of the patch Si. (∆x0,θ0)=arg∆mx0a,xθ0p(∆x0,θ0|Ib,M0,h0,g0). (10)
• Similarity measure: Using Ω, the feature vector fΩ is
We find the best parameters that maximize the joint posterior
generated and the L norm between this vector and f is
2 T densityoftheinitialobjectposition∆x0 andtheinitialobject
measured:
rotation θ0. The initial registration process is illustrated in
F(S ,T)=(cid:107)f −f (cid:107) (9)
i Ω T 2 Fig. 3 (a). The test image is firstly normalized into a unit
• Similarity score comparison: Each patch is passed cube. Unlike training, this is a “single” scale normalization
either to the left or the right child nodes according to the that corresponds to s =1 (see Eq. 1). The patches extracted
0
split function that compares the score of the similarity from the globally represented point cloud are described with
measure F(Si,T) and a randomly chosen threshold τ. the HoCP features and are passed down all the trees. At this
The main reason why we apply a depth check to the patches stage, we extract the coarsest patches from the test image,
is to remove the structural perturbations, due to occlusion i.e., the ones occupying the largest area in image pixels. We
6
Initial registration
(a) . . . ...
Iterative pose refinement
n
o
(b) tsitare1t i . . . ...
. . . .
. . . .
. . . .
n
o
ita Fig.4. Datasetgeneration:theinputoftheproposedframeworkisthedepth
(c) reti ts . . . ... ivmisaugaelizoaftiothne).coarsely detected objects (RGB correspondence is for better
a
l
Fig.3. Theobjectofinterestisfirstroughlyalignedbyextractingthecoarsest iteratively performed until the maximum iteration is reached
parts and this alignment is then iteratively refined based on finer (smaller) (see Fig. 3 (c)):
ones.
(∆xk,θk)=arg max p(∆xk,θk |M1:k,V1:k,x0,hk,gk) (13)
determine the effect that all patches have on the object pose ∆xk,θk
by accumulating the votes stored in the leaf nodes as in [7] We approximate the registration hypothesis at each iteration
and approximate the initial registration given in Eq. 10. Once by using the stored information in the leaf nodes as we do
the initial hypothesis x0 = (∆x0,θ0) is obtained, the set of in the initial registration. If we would demonstrate the 6D
pixels that belong to the background/foreground clutter {p } object pose estimation considering that the learnt forest is the
j
are removed from I according to the following criterion: IHF-fixed size, the only difference in the formulation would
b
be the part extraction viewpoint. Instead of traversing 3D
(cid:40)
I (p )=D (p ), γψ <D (pk)<βψ bounding box in the unit cube, we would extract the parts
vk = b j Ib j 1 Ib j 2 (11) with a predefined size in pixels, and at an iteration instant k,
I (p )=0, otherwise
b j gk would remain the same as g0. In the next section, we will
compare the registration performances of the forests that are
with
separately trained on fixed and variable size parts.
γ =min(Dk ), β =max(Dk ) (12)
H H
IV. EXPERIMENTALRESULTS
where Dk and D are the depth maps of the hypothesis There are several publicly available datasets [15], [30] to
H Ib
H at iteration k, and of the I , ψ and ψ are the scaling test the performances of the object registration methods. For
b 1 2
coefficients. The efficacy of vk is illustrated in Fig. 3. In the eachobjecttypeinthesedatasets,asetofRGB-Dtestimages
rightmost image of Fig. 3 (a), the test image and the initial are provided with ground truth object pose parameters. We
hypothesisaresuperimposed.Thishypothesisisexploitedand have analysed these images and have found that “Coffee
the test image is updated by v1 as in Eq. 11. The updated Cup”,“Camera”and“Shampoo”(includedintheICVLdataset
image is shown in Fig. 3 (b) and assigned as input for the [15]) are some of the best demonstrable objects to test and
1st iteration. It is normalized and represented globally. The to compare our registration architecture with the state-of-
object “scale” (h1) in the unit cube is relatively increased the-art methods since they are located in highly occluded
(comparewiththeinitialregistration)andmorediscriminative and cluttered scenes. We further process the test images of
control point descriptors n are computed. This is mainly these objects to generate a new test dataset according to the
because of the fact that the standard deviation of the input following criteria:
image decreased since we removed foreground/background • Since the HoCP features are scale-variant, the depth
clutter. As a follow up step, we traverse the 3D bounding valuesofthetrainingandthetestimagesshouldbeclose
box in the unit cube during part extraction, while the increase to each other up to a certain degree. In this study, we
in the normalized object scale gives rise to extract patches train the forests at a single depth value, f mm, and test
d
whose size are smaller (finer) than the ones extracted during with the images at the range of [f ∓50] mm.
d
the initial registration. The resulted hypothesis of the 1st • The test object instances located at the range of [fd ∓
iteration is shown on the right. The extraction of finer parts 50] mm are assumed as detected by coarse bounding
representedwithmorediscriminativecontrolpointdescriptors boxes (see Fig. 4). The image regions included in these
along with the noise removal process result in more accurate bounding boxes are cropped and the new test dataset is
and confident hypothesis. This pose refinement process is generated.
7
TABLEI
F1SCORESOFTHEINITIALREGISTRATIONSDETERMINEDFORDIFFERENT
PARTSIZES(G),IBSRESOLUTION(N)&FEATUREDIMENSION(D)AND
NUMBEROFITERATION
Part F1 N&d F1 # F1
Size,g Score Score iter Score
1 0.5966 80&128 0.7068 0 0.7510
5
1 0.6096 80&256 0.7368 1 0.7742
41 0.6532 80&512 0.7425 3 0.7745
13 0.7068 100&128 0.6870 5 0.7932
2
(a) 23 0.6341 100&256 0.7510
3 0.6539 100&512 0.7438
4
ground truth and estimated poses of the test object. The
registration hypothesis that ensures the following inequality
is considered as correct:
ω ≤z Φ (14)
ω
where Φ is the diameter of the 3D model of the test object
and z is a constant that determines the coarseness of an
ω
hypothesis that is assigned as correct. We set z to 0.08
(b) ω
when we refine the parameters of the proposed architecture
and the effect of various z values is separately examined in
ω
the comparative study.
A. Parameter Optimization
The parameters of the proposed architecture are optimized
only by training several IHFs based on fixed size parts. These
experiments are performed on the “Coffee Cup” dataset.
1) Thesizeofthepartsextractedduringtheinitialregistra-
tion: Theinitialregistrationhypothesisisusedbytheiterative
poserefinementinordertoimprovethealignment’sconfidence
(c)
(seeEq.13),andhence,g0 isoneoftheimportantparameters
Fig.5. Precision-Recallcurvesforparameteroptimization:(a)comparesthe that directly affects the success of the complete architecture.
performancesoftheforestswithdifferentpatchsizes.(b)illustratestheregis- IHF-fixed size uses the parts that are of the same size during
trationperformancesfordifferentIBSresolutionsN andfeaturedimensions
both the initial registration and the iterative pose refinement.
d. (c) shows the effect of the iteration number. For the corresponding F1
scoresseeTableI. IHF-variable size roughly aligns the object of interest during
theinitialregistrationextractingcoarsestparts,theonesoccu-
pyingthelargestareainimagepixels.Ititerativelyrefinesthis
The generated dataset includes 276 “Coffee Cup”, 360 “Cam- alignment based on the automatically extracted finer (smaller)
era” and 200 “Shampoo” images each of which is at the parts, that is, it works in a size range. Thus, evaluating the
range of [750∓50] mm since we train the forests used in all performancesoftheinitialregistrationsfordifferentpartsizes
experiments with the positive samples at f =750 mm depth. is a crucial experiment that determines not only the optimum
d
The maximum depth is 25 and the number of the maximum g0 values, but also the range of the part sizes at which the
samples at each leaf node is 15 for each tree. Every forest is IHF-variable size works in the most feasible way. The effect
the ensemble of 3 trees with these termination criteria. of the part size is examined by setting the IBS resolution N
Our experiments are two folds: parameter optimization to 80, the HoCP feature dimension d to 128 in addition to
and comparative study. The architecture parameters have an the previously defined forest parameters. We change the part
important impact upon the registration and include the size of size g0: 0.20, 0.25, 0.33, 0.50, 0.66 and 0.75 times of the
the parts extracted during the initial registration g0, the IBS objectboundingboxandforeach,wetrainseparateIHF-fixed
resolution N, the HoCP feature dimension d (the number of size. The resultant Precision-Recall (PR) curves of the initial
the bins or quantization parameter) and the iteration number. registrationsareshowninFig.5(a)andthecorrespondingF1
Once the best parameters are acquired, we compare the per- scores are demonstrated in Table I. According to this figure
formance of our architecture with the state-of-the-art methods andtheircorrespondingF1scores,wecanchooseanypartsize
in the comparative study. apartfromtheonessmallerthan 1 timesoftheboundingbox.
5
Both experiments use the metric proposed in [30] to deter- Consideringboththecomputationalloadandtheaccuracy,we
mine whether a registration hypothesis is correct. This metric choose 1 astheoptimalpartsizefortheIHF-fixedsize.Onthe
2
outputs a score ω that calculates the distance between the other hand, IHF-variable size uses the parts at various sizes,
8
Image id: Image id: Image id:
650 979 1494
Initial Initial Initial
registration Iterative pose refinement registration Iterative pose refinement registration Iterative pose refinement
no tu
itca pni
r
t
xe trap e trapnoitcartxe
z
is
d- tu
e p
xiF tuo
Initial reg. Iterative pose refinement Initial reg. Iterative pose refinement Initial reg. Iterative pose refinement
n
oitc tup
a n
r i
t
x
e
trap ezis trapnoitcartxe
e-
lb tu
a p
ir tu
a o
V
Fig.6. Comparisonbetweenthevariable-sizeandthefixed-sizepartextractionprocessesandtheireffectontheregistrationresults.Iterativeposerefinement
modulesillustratethe1st,3rd and5th iterationfromlefttoright(RGBcorrespondencesofthetestobjectsareforbettervisualization).
beginningwiththecoarsest(largest)onesextractedduringthe the memory consumption and the accuracy, and agree on the
initial registration, and ending with the finest (smallest) ones values of N =100 & d=256.
extracted at the last iteration of the iterative pose refinement. 3) The effect of the iteration: The last parameter we op-
We reanalyse the F1 scores presented in Table I taking into timize is the iteration number. We test several IHFs-fixed
account this size variation. When we increase the part size size [14] each of which has k = 0,1,3, and 5 iterations,
from 1 to 3, the F1 score ranges between 0.6 and 0.7. respectively. Their PR curves are shown in Fig. 5 (c). As
5 4
Despite the significant variation of the part size, the deviation expected,theforeststhatusegreaternumberofiterationsshow
in the F1 scores is negligible. We choose 3 as the size of better performances (see Table I) since more discriminative
4
the parts extracted during the initial registration phase of the features are extracted due to the noise removal process.
IHF-variable size. One might suggest us to train both IHF- Figure 6 demonstrates the registration results of several test
fixedandIHF-variablesizeseparatelyinordertofindthebest objects comparing the IHFs that are trained on both the fixed-
corresponding g0 values, however, it does not make sense. size and the variable-size parts. The RGB correspondences of
Because, the positive impact of the training on variable size the test objects are shown at the top, and each “iterative pose
parts is much observed during the iterative pose refinement refinement” module illustrates the 1st, 3rd and 5th iteration
phase. Hence, training only on the fixed size parts and the at its 1st, 2nd and the 3rd columns, respectively. The part
examination through the resultant F1 scores are reasonable to samplesshowninthe“partextraction”rowsaregrownaround
infer the best g0 for each approach. the same data point. We first discuss the “image id: 650”.
2) IBS resolution and HoCP feature dimension: We next The object is initially aligned by extracting the parts that
tune the IBS resolution N and the HoCP feature dimension are of size 1 for the fixed-size and 3 for the variable-size
2 4
d by setting the part size to 1 . We test the combinations of approach. By using the initial registration output, background
2
N = 80,100 and d = 128,256,512, the ones that are the clutter is removed from the test image. The amount of the
mostapplicableN−dpairstorepresent 1 patchsize.ThePR reduction is approximately the same for both approach. After
2
curvesofthesecombinationsaredepictedinFig.5(b)andthe reduction, the test image is updated and is assigned as the
corresponding F1 scores are illustrated in Table I. According input for the 1st iteration. IHF-fixed size keeps on extracting
to these results, we infer that the combinations composed by the parts that are of size 1 till the last iteration, whilst the
2
d = 128 relatively underperform whilst the remaining have IHF-variable size grows finer (smaller) regions in proportion
approximately the same F1 scores. We take into account both to the removed foreground/background clutter. One can infer
9
z = 0.05 z = 0.07 z = 0.09
ω ω ω
z = 0.11 z = 0.13 z = 0.15
ω ω ω
Fig.7. Precision-Recallcurvesofthe’CoffeeCup’dataset:EachimagecomparestheIHF-fixedsize(initialregistration),theIHF-fixedsize(initialregistration
+iterativeposerefinement)andtheIHF-variablesize(initialregistration+iterativeposerefinement)withtheLCHFs[15]trainedseparatelyonDepth,and
onRGB-Dchannels.Greatervaluesofzω resulthigherprecisionandrecallvalues.F1scoresarepresentedinTableII.
TABLEII
F1SCORESOFTHE’COFFEECUP’DATASETARESHOWNATDIFFERENTzωVALUES.
zω LCHF-D LCHF-RGBD IHF-fixedsize(fs) IHF-fixedsize(fs) IHF-variablesize(vs)
value [15] [15] (InitReg) (InitReg+Ref) (InitReg+Ref)
F1scores
0.05 0.4867 0.3818 0.4375 0.5297 0.5095
0.07 0.7202 0.6639 0.6985 0.7595 0.7891
0.09 0.7984 0.7683 0.7633 0.7975 0.8150
0.11 0.8344 0.8199 0.8000 0.8312 0.8565
0.13 0.8548 0.8554 0.8163 0.8481 0.8773
0.15 0.8589 0.8595 0.8353 0.8608 0.8940
that the variable size approach registers the object of interest Fig. 7 for several z values and their corresponding F1 scores
ω
slightlybetterthantheIHF-fixedsize.Forthe“imageid:979” are presented in Table II. A short analysis on the images
and the “image id: 1494”, the same regions of the test images of Fig. 7 reveals that the increase in z value generates
ω
are removed as the iteration progresses. However, the IHF- higher F1 scores for each approach. According to Table II,
variable size demonstrate better results for both objects. This the LCHF trained on the color gradient + surface normal fea-
comparison also verifies that we have selected the optimum tures underperforms the LCHF trained merely on the surface
g0 value for each approach. As the iteration progresses, we normals. The main reason of this underperformance is the
observe smooth transitions between the estimated translation distortion along the object borders arising from the occlusion
and rotation parameters. and the clutter, that is, the distortion of the color gradient
information in the test process. The initial registration of
B. Comparative Study
the IHF-fixed size performs better than any versions of the
These experiments are conducted on the “Coffee Cup”, LCHF. This shows that the HoCP features represent the parts
“Camera” and “Shampoo” datasets to compare our approach better than the surface normals and color gradients providing
with the state-of-the-art methods including the Latent-Class robustness across occlusion, clutter and missing depth pixels.
Hough forests (LCHF) [5] trained separately on the surface When this initial registration (see the 3th column of Table
normal (LCHF-Depth (D) channel) and the color gradient + II) acquired from the IHF-fixed size is iteratively refined,
the surface normal (LCHF-RGBD channel) features. In order more accurate registrations are resulted (see the 4th column
to make a fair comparison between methods, we train and of Table II). Because, the iterative pose refinement module
test the LCHFs by using the authors’ software. The forest of the IHF-fixed size reduces the amount of the noise in the
parameters of all approaches are the same. For both datasets testdepthmapsremovingforeground/backgroundclutter.This
we generate PR curves at various zω values, beginning with removal process also enables IHF-fixed size to compute more
thevaluethatstrictlylimitsthedeviationsbetweentheground discriminative control points for better shape representation.
truthandtheestimatedposeparameters,0.05,goingupin0.01 The IHF-variable size with initial registration + iterative pose
increments, and ending with the value that accepts relatively refinement outperforms other approaches. The main reason of
rough estimations as correct, 0.15. this high performance is the utilization of the parts that are
The PR curves of the coffee cup dataset are depicted in
10
z = 0.05 z = 0.07 z = 0.09
ω ω ω
z = 0.11 z = 0.13 z = 0.15
ω ω ω
Fig.8. Precision-Recallcurvesofthe’Camera’dataset:EachimagecomparestheIHF-fixedsize(initialregistration),theIHF-fixedsize(initialregistration
+iterativeposerefinement)andtheIHF-variablesize(initialregistration+iterativeposerefinement)withtheLCHFs[15]trainedseparatelyonDepth,and
onRGB-Dchannels.Greatervaluesofzω resulthigherprecisionandrecallvalues.F1scoresarepresentedinTableIII.
TABLEIII
F1SCORESOFTHE’CAMERA’DATASETARESHOWNATDIFFERENTzωVALUES.
zω LCHF-D LCHF-RGBD IHF-fixedsize(fs) IHF-fixedsize(fs) IHF-variablesize(vs)
value [15] [15] (InitReg) (InitReg+Ref) (InitReg+Ref)
F1scores
0.05 0.1003 0.0736 0.2071 0.1538 0.3963
0.07 0.2696 0.2240 0.3954 0.3878 0.6706
0.09 0.3723 0.4121 0.4761 0.5047 0.7680
0.11 0.3991 0.4674 0.5140 0.5710 0.8035
0.13 0.4304 0.5246 0.5494 0.6091 0.8242
0.15 0.4551 0.5500 0.5731 0.6388 0.8596
z = 0.05 z = 0.07 z = 0.09
ω ω ω
z = 0.11 z = 0.13 z = 0.15
ω ω ω
Fig.9. Precision-Recallcurvesofthe’Shampoo’dataset:EachimagecomparestheIHF-fixedsize(initialregistration),theIHF-fixedsize(initialregistration
+iterativeposerefinement)andtheIHF-variablesize(initialregistration+iterativeposerefinement)withtheLCHFs[15]trainedseparatelyonDepth,and
onRGB-Dchannels.Greatervaluesofzω resulthigherprecisionandrecallvalues.F1scoresarepresentedinTableIV.
different in size. The cascaded representation of the locality template.Despitethefactthatweapplyadepthcheckinorder
increases the robustness across clutter, occlusion, missing to get rid of those noise during testing, it is highly naive.
depth pixels and/or similar looking distractors. The object On the other hand, the proposed framework can get rid of
of interest is first roughly aligned by extracting the coarsest those structural perturbations by growing smaller regions as
parts. It is highly possible that these initially extracted parts the iteration progresses. Apart from that, the control point
include the portions belonging to the background/foreground descriptors computed at later iterations allows the complete
clutter since they are the coarsest and are close to a holistic framework to represent the shapes in a more discriminative