Table Of Content1
Quasi-homography Warps in Image Stitching
Nan Li, Yifang Xu, and Chao Wang
Abstract—Naturalnessofwarpingisgainingextensiveattention
in image stitching. Recent warps such as SPHP, AANAP and
GSP, use a global similarity to effectively mitigate projective
distortion (which enlarges regions), however, they necessarily
bring in perspective distortion (which generates inconsistency).
In this paper, we propose a quasi-homography warp, which
balances perspective distortion against projective distortion in
the non-overlapping region, to create natural-looking mosaics.
7
Our approach formulates the warp as a solution of a system of
1
bivariate equations, where perspective distortion and projective
0
distortion are characterized as slope preservation and scale
2
linearization respectively. Our proposed warp only relies on a
n globalhomographythusistotallyparameter-free.Acomprehen-
a siveexperimentshowsthatquasi-homographyoutperformssome
J state-of-the-art warps in urban scenes, including homography,
7 AutoStitch and SPHP. A user study demonstrates that quasi-
2 homography wins most users’ favor as well, comparing to
homography and SPHP.
]
V Index Terms—Image stitching, image warping, projective dis-
tortion, perspective distortion.
C
.
s I. INTRODUCTION
c
[ IMAGEstitchingplaysanimportantroleinmanymultime-
dia applications, such as panoramic videos [1]–[3], virtual
1
reality [4]–[6], etc. Basically, it is a process of combining
v
6 multiple images with overlapping fields of views to produce a
0 wide-viewpanorama[7],wherethefirststageisdetermininga
0 warp for each image to transform it in a common coordinate
8
system, then the warped images are composed [8]–[12] and
0
blended [13]–[15] into a final mosaic. Evaluation of warping
.
1 includes alignment quality in the overlapping region and
0
naturalness quality in the non-overlapping region.
7
Earlywarpsfocusonimprovingalignmentquality.Aglobal
1
: warp such as similarity or projective [16], aims to minimize
v
alignment errors between overlapping pixels by a uniform
i
X transformation. Homography is the most frequently used one, Fig.1. Adistortioncomparisonamongdifferentwarpsinthenon-overlapping
region,whichusesthesameglobalhomographyalignmentintheoverlapping
r because it is the farthest planar transformation where straight region. (a) Reference and target images. (b) Warped result of homography,
a
lines are preserved in the warped image. However, it usually whichintroducesprojectivedistortioninpeople.(c)WarpedresultofSPHP,
suffers from projective distortion (which enlarges regions) whichgeneratesperspectivedistortioninbuildings.(d)Warpedresultofquasi-
homography,thatbalancesprojectivedistortionagainstperspectivedistortion.
in the non-overlapping region (see Fig. 1(b)). Some warps
address alignment issues due to parallax in the overlapping
region, by using multiple local transformations instead of a region by using a global similarity. Though they effectively
singleglobalone[17]–[20],orbycombiningimagealignment mitigate projective distortion, they necessarily bring in per-
with seam-cutting approaches [21]–[23]. spective distortion (which generates inconsistency), because
Recently, some warps such as shape-preserving half- similarity transformations keep original perspectives individu-
projective (SPHP) warp [24], adaptive as-natural-as-possible ally (see Fig. 1(c)). It motivates us to propose a novel warp,
(AANAP) warp [25], and warp with global similarity prior thatovercomesdrawbacksofhomographyandsimilaritytrans-
(GSP) [26], address naturalness issues in the non-overlapping formations, to mitigate projective distortion and perspective
distortion simultaneously in the non-overlapping region (see
N. Li is with the Center for Applied Mathematics, Tianjin University,
Tianjin300072,China.E-mail:[email protected]. Fig. 1(d)).
Y. Xu is with the Center for Combinatorics, Nankai University, Tianjin In this paper, we propose a quasi-homography warp, which
300071,China.Email:[email protected].
balances projective distortion against perspective distortion in
C. Wang is with the Department of Software, Nankai University, Tianjin
300071,China.Email:[email protected]. the non-overlapping region, to create natural-looking mosaics
2
theoverlappingregion[16].Globalwarpsarerobustbutoften
not flexible enough to provide accurate alignment. Gao et al.
[17]proposedadual-homographywarptoaddresssceneswith
twodominantplanesbyaweightedsumoftwohomographies.
Lin et al. [18] proposed a smoothly varying affine warp
to replace a global affine transformation with a smoothly
affine stitching field, which is more flexible and maintains
much of the motion generalization properties of affine or
homography. Zaragoza et al. [19] proposed an as-projective-
as-possible warp in a moving DLT framework, which is able
to accurately align images that differ by more than a pure
rotation.Louetal.[20]proposedalocalalignmentmethodby
piecewise planar region matching, which approximates image
regionswithplanesbyincorporatingpiecewiselocalgeometric
models.
Othermethodscombineimagealignmentwithseam-cutting
approaches [27]–[30], to find a locally registered area instead
of aligning the overlapping region globally. Gao et al. [21]
proposed a seam-driven stitching framework, which searches
a homography with minimal seam costs instead of minimal
Fig.2. Adistortioncomparisonamongdifferentwarpsinthenon-overlapping alignment errors. Zhang and Liu [22] proposed a parallax-
region,whichemploysthesamehomographyalignmentandthesameseam- tolerant warp, which combines homography and content-
cuttingcomposition[27]intheoverlappingregion.Infinalstitchingresults,
preserving warps to locally align images. Lin et al. [23]
weusebluerectanglestohighlightacomparisonofprojectivedistortion,and
redrectanglestohighlightacomparisonofperspectivedistortion. proposed a seam-guided local alignment method , which
iteratively improves warping by adaptive feature weighting
according to their distances to current seams.
(see Fig. 2). Our proposed warp only relies on a global
Thesemethodsfocusonimprovingalignmentqualityinthe
homography thus is totally parameter-free. The rest of the
overlappingregion,sotheysometimessufferfromnaturalness
paper is organized as follows. Section II describes some of
issues in the non-overlapping region (see Fig. 2).
recentrelatedworks.SectionIIIprovidesadetailedderivation
of our proposed warp. In particular, Section III-A presents an
analysis of homography and SPHP warps, where perspective
distortion is addressed by preserving corresponding slopes of
B. Warps for Better Naturalness
horizontal lines and vertical lines, and projective distortion is
describedintermsoflinearizingascalefunction.SectionIII-B Many efforts have been devoted to mitigate distortion in
reformulateshomographyasasolutionofasystemofbivariate the non-overlapping region to create natural-looking mosaics.
equations, where slope preservation and scale linearization A pioneering work [31] uses spherical or cylindrical warps
are characterized respectively. In Section III-C, we linearize to provide multi-perspective results to address this problem,
the scale function to a piece-wise C1 continuous function, however, it necessarily curves straight lines.
and define our quasi-homgoraphy warp as a solution of a
Recently, some methods take the advantage of global sim-
modifiedsystem.Someimplementationdetails(includingtwo-
ilarity (preserve original perspective) to mitigate projective
image stitching and multiple-image stitching) and variations distortion. Chang et al. [24] proposed a SPHP warp, which
(including orthogonal rectification and partition refinement)
spatially combines a projective and a similarity, where the
are proposed in Section IV. Section V presents a comparison
projective maintains good alignment in the overlapping re-
experiment and a user study, which demonstrate that our
gion while the similarity keeps original perspective in the
proposed warp not only outperforms some state-of-the-art
non-overlapping region. Lin and Pankanti [25] proposed an
warpsinurbanscenes,butalsowinsmostusersfavor.Finally,
AANAPwarp,whichcombinesalinearizedhomographyanda
conclusions are drawn in Section VI.
globalsimilaritywithsmallestrotationangletocreatenatural-
looking mosaics. Chen et al. [26] proposed a GSP warp to
II. RELATEDWORK
stitch multiple images with a global similarity prior in the
In the following, we review recent works of image warping objective function, which constrains the warp resembles a
inaspectsofalignmentandnaturalnessrespectively.Formore similarity as a whole.
fundamental concepts about image stitching, please refer to a
Thesemethodsuseaglobalsimilaritytoeffectivelymitigate
comprehensive survey [7] by Szeliski.
projective distortion in the non-overlapping region, assuming
thattheoriginalimagesaremostnaturaltousers.However,the
A. Warps for Better Alignment
assumption is not consistent with human perception in urban
Conventional stitching methods usually use global warps scenes,duetotheperspectivedistortiongeneratedbydifferent
such as affine, similarity and homography, to align images in perspectives between images (see Fig. 2).
3
Fig.3. Perspectivedistortionv.s.slopepreservation.(a)Targetimage.(b)Warpedresultofhomography.(c)WarpedresultofSPHP.(d)Ourwarpedresult.
Original Homography SPHP Our warp
500 500 500 500
400 400 400 400
x‘300 x‘300 x‘ 300 x‘ 300
200 200 200 200
100 100 100 100
p p1 p2 p
0 0 0 0
0 100 200 x 300 400 500 0 100 200 x 300 400 500 0 100 200 x 300 400 500 0 100 200 x 300 400 500
（a） (b） （c） （d）
Fig.4. Projectivedistortionv.s.scalelinearization.Itdemonstratesscalefunctionsofdifferentwarpsonaspecialhorizontalline(markedredinFig.3).
III. PROPOSEDWARPS andfx,fy,gx,gy denotepartialderivativesoff andg.Infact,
In this section, we first present an analysis of homography (2) is a sufficient and necessary condition of a line-preserving
and SPHP warps, to show their advantages and disadvantages warp. In order to characterize perspective distortion, we draw
in aspect of perspective distortion or projective distortion (see a grid-to-grid map of lines with slopes
Fig.3and4),thenhomographyisreformulatedasasolutionof g (x,y)
slope(x,y,0)= x , (5)
a system of bivariate equations, where projective distortion is
f (x,y)
x
quantifiedbyanonlinearscalefunctiononaspecialhorizontal
g (x,y)
line(seeFig.5(b)),finallyquasi-homographyisproposedasa slope(x,y,∞)= y , (6)
f (x,y)
solutionofamodifiedsystem,bylinearizingthescalefunction y
to a piece-wise C1 continuous function (see Fig. 5(c)). which are corresponding to horizontal lines and vertical lines
in the target image. Consequently, any point (x,y) ∈ I can
A. Analysis of Homography and SPHP Warps be regarded as the intersection point of a horizontal line and
LetI andI(cid:48)denotethetargetimageandthereferenceimage a vertical line, which is corresponding to a point (x(cid:48),y(cid:48))∈I(cid:48)
respectively. A warp H is a planar transformation [16], which astheintersectionpointoflineswithslopes(5,6)(seeFig.3).
relates pixel coordinates (x,y)∈I to (x(cid:48),y(cid:48))∈I(cid:48), where It is easy to check that a homography warp H
0
(cid:26) x(cid:48) =f(x,y) . (1) f (x,y)= h1x+h2y+h3, (7)
y(cid:48) =g(x,y) 0 h x+h y+1
7 8
h x+h y+h
We say a warp is line-preserving, if any straight line in the g (x,y)= 4 5 6, (8)
0 h x+h y+1
target image is transformed into a straight line in the warped 7 8
result. If a line-preserving warp aligns the target image to the satisfiescondition(2),thereforehomographyisline-preserving
reference image, then the perspective of the warped result is (seeFig.3(b)).However,SPHPisnotline-preserving,because
consistent with the perspective of the reference image. Given projective’s perspective in the overlapping region is inconsis-
a line-preserving warp H as (1) and a line with slope k tentwithsimilarity’sperspectiveinthenon-overlappingregion
crossing (x,y) ∈ I, then it is corresponding to a line with (see Fig. 3(c)).
slope slope(x,y,k) crossing (x(cid:48),y(cid:48))∈I(cid:48), where
In horizontal image stitching, there always exists a special
slope(x,y,k)= g(x+z,y+kz)−g(x,y) (∀z ∈R) (2) horizontal line y =y∗, that preserves horizontal under a line-
f(x+z,y+kz)−f(x,y) preserving warp, where y∗ is determined by solving
g(x+z,y+kz)−g(x,y) g (x,y )
= lim (3) slope(x,y ,0)= x ∗ =0. (9)
z→0f(x+z,y+kz)−f(x,y) ∗ fx(x,y∗)
g (x,y)+kg (x,y)
= x y , (4) In fact, f(x,y∗) is a scale function on the special horizontal
f (x,y)+kf (x,y)
x y line,whichquantifiesprojectivedistortioninhorizontalimage
4
Fig.5. Quasi-homographyv.s.homography.(a)Targetimage.(b)Reformulationofhomography.(c)Derivationofquasi-homography.Inthenon-overlapping
region,quasi-homographynotonlypreservescorrespondingslopesofgridsashomography,butalsopossessesalinearscalefunctiononthespecialhorizontal
lineassimilarity.
stitching.Inordertocharacterizethefunctionmoreexplicitly, where(x ,y)and(x,y )areprojectionsof(x,y)ontox=x
p ∗ p
we draw a map x(cid:55)→f(x,y ) (see Fig. 4). and y =y respectively (see Fig. 5(a)), and
∗ ∗
Given a homography warp H as (7,8), we can obtain
0 (h h −h h )y−h h +h
slope(x,y,0)= 4 8 5 7 6 7 4, (16)
y = h6h7−h4 , (10) (h1h8−h2h7)y−h3h7+h1
∗ h4h8−h5h7 slope(x,y,∞)= (h4h8−h5h7)x+h6h8−h5, (17)
(h h −h h )(h x+h )+(h h −h )h (h h −h h )x+h h −h
f (x,y )= 4 8 5 7 1 3 6 7 4 2. 1 8 2 7 3 8 2
0 ∗ (h4h8−h5h7)h7x+(h6h8−h5)h7 which are independent of x and y respectively.
(11)
It is easy to check that, if we regard f(x,y) and g(x,y) as
Homographynecessarilygeneratesprojectivedistortion,be-
unknowns, then H as (7,8) is the unique solution of (14,15)
cause the derivative f(cid:48)(x,y ) constantly increases in the non- 0
0 ∗ satisfying (10,13,16,17). Comparing to the formulation (7,8),
overlappingregion(seeFig.4(b)).However,SPHPpossessesa
our reformulation characterizes perspective distortion as slope
piece-wiseC1 continuousscalefunction,whichislinearinthe
preservation(slope(x,y,0)andslope(x,y,∞))andprojective
non-overlapping region (see Fig. 4(c)). This difference leads
distortionasscalelinearization(f (x,y ))respectively.Infact,
to less projective distortion of SPHP than homography. 0 ∗
slope(x,y,0)andslope(x,y,∞)constrainshapesofthegrid,
while f (x,y ) determines its density (see Fig. 5(b)).
B. Reformulation of Homography Warps 0 ∗
Suppose I and I(cid:48) are well-aligned by a global homography
H as(7,8)intheoverlappingregion,ourgoalistoproposea
0
warp H that combines the advantages of homography (line-
∗ C. Quasi-homography Warps
preserving) and similarity (a linear scale function) in the non-
overlapping region. In the following, we give a reformulation
OurreformulationofhomographyandouranalysisofSPHP
ofhomography,thatcharacterizesslopepreservationandscale motivateustomodifyf (x,y )toapiece-wiseC1 continuous
0 ∗
linearization respectively.
function, which is linear in the non-overlapping region.
First,wedefinex=x asthepreimageoftheborderofthe
p First, the special horizontal line y = y (13) intersects the
overlappingregionandthenon-overlappingregion(markedin ∗
preimage x=x (10) at
blue in Fig. 5(a)), which satisfies p
(h h −h h h )p+h (h h −h ) h
f(x ,y)=p, (12) x = 5 7 6 7 8 2 6 7 4 + 3 .
p ∗ (h h −h h )(ph −h ) ph −h
4 8 5 7 7 1 7 1
where p=0 if assigning the left image as the target image I, (18)
otherwise p equals to the width of the reference image I(cid:48). Then, the truncation of Taylor’s expansion of f (x,y ),
0 ∗
For a homography H , the preimage
0 (h h −h h )y +(h −h h )
f (x,y )=p+ 1 8 2 7 ∗ 1 3 7 (x−x )
x = (ph8−h2)y−h3+p. (13) ∗ ∗ (h7x∗+h8y∗+1)2 ∗
p h −ph (19)
1 7
is linear in the non-overlapping region and is C1 connected
Then,inordertobenefitfromtheanalysisofhomographyand
to f (x,y ) in the overlapping region. Let O denote the
SPHP, we reformulate homography as a solution of 0 ∗
overlapping region, we modify the scale function to
g(x,y)−g0(xp,y) =slope(x,y,0), (14) (cid:40)f (x,y ), if (x,y )∈O,
f(x,y)−f0(xp,y) f†(x,y∗)= 0 ∗ ∗ (20)
f (x,y ), otherwise,
g(x,y)−g (x,y ) ∗ ∗
0 ∗ =slope(x,y,∞), (15)
f(x,y)−f0(x,y∗) which is piece-wise C1 continuous. If we regard f(x,y) and
5
g(x,y) as unknowns, then the system of bivariate equations Algorithm 1 Image stitching using quasi-homography.
Input: two images taken from a camera rotated vertically.
g(x,y)−g (x ,y)
0 p =slope(x,y,0), (21) Output: one horizontally stitched image.
f(x,y)−f (x ,y)
0 p 1) Use SIFT [32] to extract and match features;
g(x,y)−g (x,y )
0 ∗ =slope(x,y,∞), (22) 2) UseRANSAC[33]toestimateaglobalhomographyH0;
f(x,y)−f (x,y )
† ∗ 3) Calculate a quasi-homography warp:
satisfying (10,13,16,17,20) possess a unique solution H† a) Calculate a forward map H† (23,24) to get the canvas;
(cid:40) b) Calculate a backward map H−1 (27) to fill the canvas
f (x,y), if (x,y )∈O, †
f (x,y)= 0 ∗ (23) by bilinear interpolations;
†
f (x,y), otherwise,
∗ 4) Use seam-cutting [27] to blend the overlapping region.
(cid:40)
g (x,y), if (x,y )∈O,
g (x,y)= 0 ∗ (24)
†
g (x,y), otherwise,
∗
where
w x2y+w x2+w xy+w x+w y+w
f (x,y)= 1 2 3 4 5 6,
∗ h x+h y+1
7 8
(25)
w x2y+w x2+w xy+w x+w y+w
g (x,y)= 7 8 9 10 11 12,
∗ h x+h y+1
7 8
(26)
define a warp H in the non-overlapping region and Fig.6. Stitchingmultipleimages.Coordinatesofotherimagesaretransformed
w ,...,w are pol∗ynomials in p,h ,...,h . tothecoordinatesystemofthereferenceimageI1.
1 12 1 8
In fact, the warp H just crowds the grid of homography in
†
thehorizontaldirectionbutwithoutvaryingitsshape(seeFig.
B. Multiple-image Stitching
5(c)).Inthissense,wecallH aquasi-homographywarpcor-
†
responding to a homography warp H . A quasi-homography Given a sequence of multiple images, which are captured
0
H maintainsgoodalignmentintheoverlappingregionasH from a camera rotated vertically, our warping method consists
† 0
whilemitigatesperspectivedistortionandprojectivedistortion of three stages. In the first stage, we determine a reference
simultaneously as H in the non-overlapping region. imageasthestandardperspective.Then,weestimateahomog-
∗
IfahomographywarpH (7,8)isaninjectionintheimage raphy for every two images with overlapping fields of views,
0
domain, then its corresponding quasi-homography warp H is and calculate its corresponding quasi-homography. Finally,
†
also an injection. Therefore, given (x(cid:48),y(cid:48))∈I(cid:48), then (x,y)∈ we concatenate all target images in the image plane of the
I is determined by an inverse map reference image.
(cid:40) Fig. 6 illustrates the concatenation procedure of stitching
H−1, if (x,y )∈O,
H†−1 = H0−1, otherwi∗se, (27) four images. First, we set I1 as the reference image, so
∗ perspectives of other target images should agree with its per-
where H−1 spective. Then, we calculate quasi-homography warps H†0→1,
∗ H2→1 and H3→2. Finally, we concatenate I to I by
† † 3 1
x=RootOf(m x2+m x+m ), (28)
1 2 3
H3→1 =H2→1◦H3→2. (30)
h x(cid:48)−h h x(cid:48)−h y(cid:48)+h h y(cid:48)−h h +h h † † 0
y =− 4 6 7 1 3 7 3 4 1 6 ,
h h x(cid:48)−h h x(cid:48)−h h y(cid:48)+h h y(cid:48)−h h +h h For stitching more images, we obtain the concatenation maps
4 8 5 7 1 8 2 7 2 4 1 5
(29) recursively.
and m ,...,m are polynomials in x(cid:48),y(cid:48),p,h ,...,h .
1 3 1 8
IV. IMPLEMENTION C. Orthogonal Rectification
Inthissection,wefirstpresentmoreimplementationdetails
Inurbanscenes,usersaccustomtotakepicturesbyrotating
ofourquasi-homographyintwo-imagestitchingandmultiple-
cameras vertically, hence any vertical line in the target image
imagestitching,andthenweproposetwovariationsincluding
isexpectedtotransformtoaverticallineinthewarpedresult.
orthogonal rectification and partition refinement.
However, it will inevitably sacrifice alignment quality in the
overlapping region.
A. Two-image Stitching
In order to achieve orthogonal rectification, we incorpo-
Giventwoimagescapturedfromacamerarotatedvertically,
rate an extra constraint in homography estimation, which
if a homography H is estimated, then a quasi-homography
0 constrains the external vertical boundary of the target image
H can be calculated, which smoothly extrapolates H in the
† 0 preserves vertical in the warped result. Then, it should satisfy
overlapping region to H in the non-overlapping region. A
∗
brief algorithm is given in Algorithm 1. f(p,0)=f(p,h), (31)
6
is more precise when (x ,y) is more locally aligned, thus we
∗
modifythepartitionlinex astheexternalborderoftheseam
p
for further refinement (see Fig. 8).
V. EXPERIMENTS
Weexperimentedourproposedmethodonarangeofimages
captured from rear and front cameras in urban scenes. In
our experiments, we use SIFT [32] to extract and match
features,RANSAC[33]toestimateaglobalhomography,and
seam-cutting [27] to blend the overlapping region. Codes are
implemented in OpenCV 2.4.9 and generally take 1s-2s on
a desktop PC with Intel i5 3.0GHz CPU and 8GB memory
to stitch two images with 800×600 resolution by Algorithm
IV-A, where the calculation of quasi-homography only takes
0.1s (including forward map and backward map).
Fig.7. Anexampleoforthogonalrectification.
A. Comparisons to State-of-the-art Warps
We compared our quasi-homography warp to state-of-the-
art warps in urban scenes, including homography, SPHP and
AutoStitch. In order to highlight comparisons on natural-
ness quality in the non-overlapping region, for homography,
SPHP and quasi-homography, we use the same homography
alignment and the same seam-cutting composition in the
overlapping region. More results and original input images
are available in the supplementary material.
Fig.9illustratesanexampleinselfie-stitching.Homography
provides a line-preserving stitching result but generates pro-
jective distortion in buildings and trees (highlighted in blue
rectangles). SPHP mitigates projective distortion significantly
but brings in apparent perspective distortion in the top of
buildings (highlighted in red rectangles). Quasi-homography
Fig.8. Anexampleofpartitionrefinement.
creates a more natural-looking stitching result in both aspects
of projective distortion and perspective distortion.
Fig. 10 illustrates an example in multiple-image stitching,
wherehistheheightofthetargetimageandp=0ifassigning
where stitching results are cropped for the sake of layout.
theleftimageastarget,otherwisepequalstothewidthofthe
Homography enlarges trees on the right (highlighted in blue
target image. For a homography warp H as (7,8), we obtain
0
rectangles)butpreservesallstraightlines.SPHPcurvesbuild-
h (h p+h )
h = 2 7 9 . (32) ings and grounds (highlighted in red rectangles) but without
8 h1p+h3 enlarging the region of trees. AutoStitch produces a fisheye-
Then, a global homography is estimated by solving like mosaic due to spherical warps. Quasi-homography bal-
ances projective distortion against perspective distortion by
N
min(cid:88)(cid:107)a h(cid:107)2 s.t. (cid:107)h(cid:107)=1,h = h2(h7p+h9). (33) preserving horizontal lines and vertical lines, simultaneously
i 8 h p+h possessing a piece-wise C1 continuous scale function.
1 3
i=1
Since quasi-homography just crowds the grid of homography
B. User Study
in the horizontal direction but without varying its shape, so
the external vertical boundary still preserves vertical (see Fig. In order to investigate whether quasi-homography is pre-
7). ferred by users in urban scenes, we conduct a user study to
compare our result to homography and SPHP. We invite 17
participantstorank20unannotatedgroupsofstitchingresults,
D. Partition Refinement
including 5 groups from front camera and 15 groups from
In (12), we simply set the partition line x as the border of rear camera. In each group, we use the same homography
p
the overlapping region and the non-overlapping region, which alignment and the same seam-cutting composition, and all
directly determines the linear scale function (19). In fact, the parametersaresettoproducetheoptimalresults.Inourstudy,
scale ratio each participant ranks three unannotated stitching results in
(h h −h h )y +(h −h h ) each group, and a score is counted by assigning weights 4, 2
ρ= 1 8 2 7 ∗ 1 3 7 (34)
(h x +h y +1)2 and 1 to Rank 1, Rank 2 and Rank 3.
7 ∗ 8 ∗
7
Fig.9. Adistortioncomparisonamongdifferentwarpsinthenon-overlappingregioninselfie-stitching,whichemploysthesamehomographyalignmentand
the same seam-cutting composition [27] in the overlapping region. In final stitching results, we use blue rectangles to highlight a comparison of projective
distortion,andredrectanglestohighlightacomparisonofperspectivedistortion.
Table I shows a summary of rank vote and total score for
three warps, and the histogram of the score is shown in Fig.
11inthreeaspects.Thisuserstudydemonstratesthatstitching
results of quasi-homography win most users’ favor in urban
scenes.
VI. CONCLUSION
In this paper, we propose a quasi-homography warp, which
balances perspective distortion against projective distortion in
thenon-overlappingregion,tocreatenatural-lookingmosaics.
Experiments show that quasi-homography outperforms some
state-of-the-art warps in urban scenes, including homography,
Fig.11. Scoreofcomparedmethodsinfrontandrearcamerasetting.
AutoStitchandSPHP.Auserstudydemonstratesthatstitching
8
Fig.10. Acomparisonofdifferentwarpsinstitchingfourimagesofteachingbuildingexample.
9
TABLEI [13] P.J.BurtandE.H.Adelson,“Amultiresolutionsplinewithapplication
THECOMPARISONRESULTSFORTHREEMETHODS. to image mosaics,” ACM Trans. Graphics, vol. 2, no. 4, pp. 217–236,
1983.
Methods Results [14] P. Pe´rez, M. Gangnet, and A. Blake, “Poisson image editing,” ACM
Trans.Graphics,vol.22,no.3,pp.313–318,2003.
Rank 1 Rank 2 Rank 3 Total score
[15] A.Levin,A.Zomet,S.Peleg,andY.Weiss,“Seamlessimagestitching
Homography 69 210 61 757
in the gradient domain,” in Proc. Eur. Conf. Comput. Vis., May 2004,
SPHP [24] 19 56 265 453 pp.377–389.
Quasi-homography 252 74 14 1170 [16] R.HartleyandA.Zisserman,Multipleviewgeometryincomputervision.
Cambridgeuniv.press,2003.
[17] J. Gao, S. J. Kim, and M. S. Brown, “Constructing image panoramas
using dual-homography warping,” in Proc. IEEE Conf. Comput. Vis.
results of quasi-homography win most users’ favor as well, PatternRecog.,Jun.2011,pp.49–56.
comparing to homography and SPHP. It should be noted [18] W.-Y.Lin,S.Liu,Y.Matsushita,T.-T.Ng,andL.-F.Cheong,“Smoothly
varying affine stitching,” in Proc. IEEE Conf. Comput. Vis. Pattern
that, though quasi-homography preserves horizontal lines and
Recog.,Jun.2011,pp.345–352.
verticallines,itmaycurvesomediagonallinesinthestitching [19] J. Zaragoza, T.-J. Chin, M. S. Brown, and D. Suter, “As-projective-
result. Fortunately, users accustom to take pictures of orthog- as-possible image stitching with moving DLT,” in Proc. IEEE Conf.
Comput.Vis.PatternRecog.,Jun.2013,pp.2339–2346.
onal compositions in urban scenes. Future works may include
[20] Z. Lou and T. Gevers, “Image alignment by piecewise planar region
generalizing quasi-homography warps in the spatially-varying matching,”IEEETransactionsonMultimedia,vol.16,no.7,pp.2052–
warping framework to improve alignment quality. 2061,2014.
[21] J. Gao, Y. Li, T.-J. Chin, and M. S. Brown, “Seam-driven image
stitching,”Eurographics,pp.45–48,2013.
REFERENCES [22] F.ZhangandF.Liu,“Parallax-tolerantimagestitching,”inProc.IEEE
[1] S.TzavidasandA.K.Katsaggelos,“Amulticamerasetupforgenerating Conf.Comput.Vis.PatternRecog.,May.2014,pp.3262–3269.
stereopanoramicvideo,”IEEETransactionsonMultimedia,vol.7,no.5, [23] K. Lin, N. Jiang, L.-F. Cheong, M. Do, and J. Lu, “Seagull: Seam-
pp.880–890,2005. guided local alignment for parallax-tolerant image stitching,” in Proc.
[2] X.Sun,J.Foote,D.Kimber,andB.S.Manjunath,“Regionofinterest Eur.Conf.Comput.Vis.,Oct.2016.
extraction and virtual camera control based on panoramic video cap- [24] C.-H. Chang, Y. Sato, and Y.-Y. Chuang, “Shape-preserving half-
turing,”IEEETransactionsonMultimedia,vol.7,no.5,pp.981–990, projectivewarpsforimagestitching,”inProc.IEEEConf.Comput.Vis.
2005. PatternRecog.,May2014,pp.3254–3261.
[3] V.R.Gaddam,M.Riegler,R.Eg,andP.Halvorsen,“Tilingininteractive [25] C.-C. Lin, S. U. Pankanti, K. N. Ramamurthy, and A. Y. Aravkin,
panoramic video: Approaches and evaluation,” IEEE Transactions on “Adaptive as-natural-as-possible image stitching,” in Proc. IEEE Conf.
Multimedia,vol.18,no.9,pp.1–1,2016. Comput.Vis.PatternRecog.,Jun.2015,pp.1155–1163.
[4] H.Y.Shum,K.T.Ng,andS.C.Chan,“Avirtualrealitysystemusing [26] N.I.S.withtheGlobalSimilarityPrior,“Yu-shengchenandyung-yu
the concentric mosaic: construction, rendering, and data compression,” chuang,”inProc.Eur.Conf.Comput.Vis.,Oct.2016,pp.186–201.
IEEETransactionsonMultimedia,vol.7,no.1,pp.85–95,2005. [27] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy min-
[5] W. K. Tang, T. T. Wong, and P. A. Heng, “A system for real-time imization via graph cuts,” IEEE Trans. Pattern Anal. Mach. Intell.,
panoramagenerationanddisplayintele-immersiveapplications,”IEEE vol.23,no.11,pp.1222–1239,Nov.2001.
TransactionsonMultimedia,vol.7,no.2,pp.280–292,2005. [28] A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn,
[6] Q. Zhao, L. Wan, W. Feng, and J. Zhang, “Cube2video: Navigate be- B. Curless, D. Salesin, and M. Cohen, “Interactive digital photomon-
tweencubicpanoramasinreal-time,”IEEETransactionsonMultimedia, tage,”ACMTrans.Graphics,vol.23,no.3,pp.294–302,2004.
vol.15,no.8,pp.1745–1754,2013. [29] V. Kwatra, A. Scho¨dl, I. Essa, G. Turk, and A. Bobick, “Graphcut
[7] R.Szeliski,“Imagealignmentandstitching:Atutorial,”Found.Trends textures: image and video synthesis using graph cuts,” ACM Trans.
Comput.Graph.Vis.,vol.2,no.1,pp.1–104,2006. Graphics,vol.22,no.3,pp.277–286,2003.
[8] S. Peleg, “Elimination of seams from photomosaics,” Comput. Graph. [30] A. Eden, M. Uyttendaele, and R. Szeliski, “Seamless image stitching
ImageProcess.,vol.16,no.1,pp.90–94,1981. ofsceneswithlargemotionsandexposuredifferences,”inProc.IEEE
[9] M.-L. Duplaquet, “Building large image mosaics with invisible seam Conf.Comput.Vis.PatternRecog.,vol.2,Jun.2006,pp.2498–2505.
lines,”inProc.SPIEVisualInformationProcessingVII,1998,pp.369– [31] M.BrownandD.G.Lowe,“Automaticpanoramicimagestitchingusing
377. invariantfeatures,”Int.J.Comput.Vis.,vol.74,no.1,pp.59–73,2007.
[10] J.Davis,“Mosaicsofsceneswithmovingobjects,”inProc.IEEEConf. [32] D.G.Lowe,“Distinctiveimagefeaturesfromscale-invariantkeypoints,”
Comput.Vis.PatternRecog.,June.1998,pp.354–360. Int.J.Comput.Vis.,vol.60,no.2,pp.91–110,2004.
[11] A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis [33] M.A.FischlerandR.C.Bolles,“Randomsampleconsensus:aparadigm
andtransfer,”inProc.ACMSIGGRAPH,2001,pp.341–346. for model fitting with applications to image analysis and automated
[12] A.MillsandG.Dudek,“Imagestitchingwithdynamicelements,”Image cartography,”Commun.ACM,vol.24,no.6,pp.381–395,1981.
Vis.Comput.,vol.27,no.10,pp.1593–1602,2009.
