Improving Feature Tracking with Robust Statistics

24 Pages·2012·0.39 MB·English

Improving Feature Tracking
with Robust Statistics
y
a b a a
Andrea Fusiello , Emanuele Trucco , Tiziano Tommasini , Vito Roberto
a
Dipartimento di Matematica e Informatica
Universit(cid:18)a di Udine,
Via delle Scienze 206, I-33100 Udine, IT
E-mail: ffusiello,tommasin,[email protected]
b
Department of Computing and Electrical Engineering
Heriot-Watt University
Riccarton, Edinburgh, EH14 4AS, UK
E-mail: [email protected]
y
Corresponding author
Improving Feature Tracking with Robust Statistics
Abstract
Thispaperaddressesrobustfeaturetracking. Weextendthewell-knownShi-Tomasi-
Kanade tracker by introducing an automatic scheme for rejecting spurious features.
We employa simpleande(cid:14)cient outlierrejectionrule, calledX84, and prove that its
theoretical assumptions are satis(cid:12)ed in the feature tracking scenario. Experiments
withrealandsynthetic imagescon(cid:12)rmthat ouralgorithmmakes goodfeatures track
better; we show a quantitative example of the bene(cid:12)ts introduced by the algorithm
for the case of fundamental matrix estimation. The complete code of the robust
tracker is available via ftp.
Key words: Motion Analysis, Feature Tracking, Robust Statistics, Optical (cid:13)ow,
Registration, X84
1
Improving Feature Tracking with Robust Statistics
1 Introduction
Much workonstructure frommotion[12]hasassumed thatcorrespondences through
a sequence of images could be recovered. Feature tracking (cid:12)nds matches by selecting
image features and tracks these as they move from frame to frame. It can be seen
as an instance of the general problem of computing the optical (cid:13)ow, that is, the
vector's (cid:12)eld that describes how the imageischangingwith time, atrelativelysparse
image positions [15, 2, 5]. The methods based on the detection of two dimensional
features (such as corners) have the advantage that the full optical (cid:13)ow is known at
every measurement position, because they do not su(cid:11)er from the aperture problem
e(cid:11)ect (a discussion on this subject can be found in [24]). Works on tracking of two
dimensional features include [13, 1, 6, 18, 26].
Robust tracking means detecting automatically unreliable matches, or outliers, over
an image sequence (see [14] for a survey of robust methods in computer vision). Re-
cent examples of such robust algorithms include [23], which identi(cid:12)es tracking out-
lierswhileestimatingthefundamentalmatrix, and[22], which adoptsa RANSAC[8]
approach toeliminateoutliersforestimatingthe trifocaltensor. Such approaches in-
crease the computationalcost of tracking signi(cid:12)cantly, as they are based on iterative
algorithms.
This paper concentrates on the well-known Shi-Tomasi-Kanade tracker, and pro-
poses a robust version based on an e(cid:14)cient outlier rejection scheme. Building on
results from[13], TomasiandKanade[20]introduced a featuretracker based on SSD
matching and assuming translational frame-to-frame displacements. Subsequently,
Shi and Tomasi [19] proposed an a(cid:14)ne model, which proved adequate for region
matching over longer time spans. Their system classi(cid:12)ed a tracked feature as good
(reliable) or bad (unreliable) according to the residual of the match between the
associated image region in the (cid:12)rst and current frames; if the residual exceeded a
user-de(cid:12)ned threshold, the feature was rejected. Visual inspection of results demon-
strated good discriminationbetween good and bad features, but the authors did not
specify how to reject bad features automatically.
1
Improving Feature Tracking with Robust Statistics
This is the problem that our method solves. We extend the Shi-Tomasi-Kanade
tracker (Section 2) by introducing an automatic scheme for rejecting spurious fea-
tures. We employ a simple, e(cid:14)cient, model-free outlier rejection rule, called X84,
and prove that its assumptions are satis(cid:12)ed in the feature tracking scenario (Section
3). Our RobustTracking algorithm is summarized in Section 4. Experiments
with real and synthetic images con(cid:12)rm that our algorithm makes good features to
track better, in the sense that outliers are located reliably (Section 5). We illustrate
quantitatively the bene(cid:12)ts introduced by the algorithm with the example of funda-
mental matrix estimation. Image sequences with results and the source code of the
robust tracker are availableon line (http://www.dimi.uniud.it/~fusiello/demo-rtr/).
2 The Shi-Tomasi-Kanade tracker
In this section the Shi-Tomasi-Kanade tracker [19, 20] will be brie(cid:13)y described.
>
Consider an image sequence I(x;t), where x = [u;v] are the coordinates of an
image point. If the time sampling frequency (that is, the frame rate) is su(cid:14)ciently
high, we can assume that small image regions undergo a geometric transformation,
but their intensities remain unchanged:
I(x;t) = I((cid:14)(x);t+(cid:28)); (1)
where (cid:14)((cid:1)) is the motion (cid:12)eld, specifying the warping that isappliedto imagepoints.
The fast-sampling hypothesis allows us to approximate the motion with a transla-
tion, that is,
(cid:14)(x) = x+d; (2)
where d is a displacement vector. The tracker's task is to compute d for a number
of automatically selected point features for each pair of successive frames in the
sequence. As the image motion model is not perfect, and because of image noise,
2
Improving Feature Tracking with Robust Statistics
(1) is not satis(cid:12)ed exactly. The problem is then (cid:12)nding the displacement d which
minimizes the SSD residual
X(cid:2) (cid:3)2
(cid:15) = I(x+d;t+(cid:28))(cid:0)I(x;t) ; (3)
W
where W is a given feature window centered on the point x. In the following we will
solve this problem by means of a Newton-Raphson iterative search.
Thanks to the fast-sampling assumption, we can approximate I(x+d;t+(cid:28)) with
its (cid:12)rst-order Taylor expansion:
>
I(x+d;t+(cid:28)) (cid:25) I(x;t)+rI(x;t) d+It(x;t)(cid:28); (4)
>
where rI = [Iu;Iv] = [@I=@u;@I=@v] and It = @I=@t: We can then rewrite the
residual (3) as
X
> 2
(cid:15) (cid:25) (rI(x;t) d+It(x;t)(cid:28)) : (5)
W
To minimize the residual (5), we di(cid:11)erentiate it with respect to the unknown dis-
placement d and set the result to zero, obtaining the linear system:
Cd = g; (6)
where
2 3
X Iu2 IuIv
C = 4 5 (7)
2
W IuIv Iv
X
>
g = (cid:0)(cid:28) It[Iu Iv] : (8)
W
(cid:0)1
If dk = C g is the displacement estimate at iteration k, and assuming a unit time
3
Improving Feature Tracking with Robust Statistics
interval between frames, the algorithm for minimizing (5) is the following:
8
>
< d0 = 0
h i
P :
>: dk+1 = dk +C(cid:0)1 (I(x;t)(cid:0)I(x+dk;t+1))rI(x;t)
W
2.1 Feature extraction
A feature is de(cid:12)ned as a region that can be easily tracked from one frame to the
other. In this framework, a feature can be tracked reliably if a numerically stable
solutionto(6)canbe found, which requires thatCiswell-conditionedanditsentries
are well above the noise level. In practice, since the larger eigenvalue is bound by
the maximum allowable pixel value, the requirement is that the smaller eigenvalue
must be su(cid:14)ciently large. Calling (cid:21)1 and (cid:21)2 the eigenvalues of C, we accept the
corresponding feature if
min((cid:21)1;(cid:21)2) > (cid:21)t (9)
where (cid:21)t is a user-de(cid:12)ned threshold [19].
Figure 1 near here
This algebraic characterization of \trackable" features has an interesting interpre-
tation, as they turns out to be corners, that is image features characterized by an
intensity discontinuity in two directions. Since the motion of an image feature can
be measured only in its projection on the brightness gradient (aperture problem),
corners are the features whose motion can be measured.
Discontinuitycanbedetected, forinstance,usingnormalizedcross-correlation,which
measures how well an image patch matches other portions of the image as it is
shifted from its original location. A patch which has a well-de(cid:12)ned peak in its
auto-correlation function can be classi(cid:12)ed as a corner. Let us compute the change
in intensity, as the sum of squared di(cid:11)erences, in the direction h for a patch W
centered in x = (u;v):
4
Improving Feature Tracking with Robust Statistics
X
2
Eh(x) = (I(x+d)(cid:0)I(x+d+h)) (10)
d2W
Using the Taylor series expansion truncated to the linear term:
X (cid:0) > (cid:1)2
Eh(x) (cid:25) rI(x+d) h
d2W
X
> >
= h (rI(x+d))(rI(x+d)) h
d2W
0 1
= X h>@ Iu2 Iu Iv Ah (11)
2
d2W Iu Iv Iv
0 2 31
= h>@X 4 Iu2 IuIv5Ah:
2
d2W IuIv Iv
The change in intensity around x is therefore given by
>
Eh(x) = h C h (12)
where C is just the matrix de(cid:12)ned in (7). Elementary eigenvector theory tells us
that, since jjhjj = 1, then
(cid:21)1 < Eh(x) < (cid:21)2; (13)
where (cid:21)1 and (cid:21)2 are the eigenvalues of C. So, if we try every possible orientation
h, the maximum change in intensity we will (cid:12)nd is (cid:21)2, and the minimum value
is (cid:21)1. We can therefore classify the structure around each pixel by looking at the
eigenvalues of C:
(cid:15) no structure: (cid:21)1 (cid:25) (cid:21)2 (cid:25) 0;
(cid:15) edge: (cid:21)1 (cid:25) 0, (cid:21)2 (cid:29) 0;
(cid:15) corner: (cid:21)1 e (cid:21)2 both large and distinct.
5
Improving Feature Tracking with Robust Statistics
Hence, the features selected according to criterion criterion (9) are to be interpreted
as corners. Indeed, this method is very closely related to some classical corner
detectors, such as [16, 17, 11].
Figure 1 shows the value of the minimum eigenvalue for the (cid:12)rst frame of the \Ar-
tichoke" sequence (see Section 5).
2.2 A(cid:14)ne model
The translational model cannot account for certain transformations of the feature
window, for instance rotation, scaling, and shear. An a(cid:14)ne motion (cid:12)eld is a more
accurate model [19], that is,
(cid:14)(x) = Mx+d; (14)
where d is the displacement, and M is a 2(cid:2)2 matrix accounting for a(cid:14)ne warping,
and can be written as M = 1 + D, with D = [dij] a deformation matrix and 1
the identity matrix. Similarly to the translational case, one estimates the motion
parameters, D and d, by minimizing the residual
X(cid:2) (cid:3)2
(cid:15) = I(Mx+d;t+(cid:28))(cid:0)I(x;t) : (15)
W
By plugging the (cid:12)rst-order Taylor expansion of I(Mx + d;t + (cid:28)) into (15), and
imposing that the derivatives with respect to D and d are zero, we obtain the linear
system
Bz = f; (16)
>
in which z = [d11 d12 d21 d22 d1 d2] contains the unknown motion parameters, and
6
Improving Feature Tracking with Robust Statistics
X
>
f = (cid:0)(cid:28) It[uIu uIv vIu vIv Iu Iv] ;
W
with
2 3
X U V
B = 4 5;
>
W V C
2 3
2 2 2 2
u Iu u IuIv uvIu uvIuIv
6 7
6 7
2 2 2 2
6 u IuIv u Iv uvIuIv uvIv 7
U = 6 7;
6 7
2 2 2 2
6 uvIu uvIuIv v Iu v IuIv 7
4 5
2 2 2 2
uvIuIv uvIv v IuIv v Iv
2 3
2 2
V> = 4 uIu uIuIv vIu vIuIv 5:
2 2
uIuIv uIv vIuIv vIv
Again, (15) is solved for z using a Newton-Raphson iterative scheme.
If frame-to-frame a(cid:14)ne deformations are negligible, the pure translation model is
preferable (the matrix M is assumed to be the identity). The a(cid:14)ne model is used
for comparing features between frames separated by signi(cid:12)cant time intervals to
monitor the quality of tracking.
3 Robust monitoring
In order to monitor the quality of the features tracked, the tracker checks the resid-
uals between the (cid:12)rst and the current frame: high residuals indicate bad features
which must be rejected. Following [19], we adopt the a(cid:14)ne model, as a pure trans-
lational model would not work well with long sequences: too many good features
7
Improving Feature Tracking with Robust Statistics
are likely to undergo signi(cid:12)cant rotation, scaling or shearing, and would be incor-
rectly discarded. Non-a(cid:14)ne warping, which will yield high residuals, is caused by
occlusions, perspective distortions and strong intensity changes (e.g. specular re(cid:13)ec-
tions). This section introduces our method for selecting a robust rejection threshold
automatically.
3.1 Distribution of the residuals
We begin by establishing which distributionis to be expected for the residuals when
comparinggoodfeatures, i.e., almostidenticalregions. We assume thattheintensity
I((cid:14)(x);t) of each pixel in the current-frame region is equal to the intensity of the
y
corresponding pixel in the (cid:12)rst frame I(x;0) plus some Gaussian noise n (cid:17) (cid:17)(0;1) .
Hence
I((cid:14)(x);t)(cid:0)I(x;0) (cid:17) (cid:17)(0;1):
Since the square of a Gaussian random variable has a chi-square distribution, we
obtain
(cid:2) (cid:3)2 2
I((cid:14)(x);t)(cid:0)I(x;0) (cid:17) (cid:31) (1):
The sum of n chi-square random variables with one degree of freedom is distributed
as a chi-square with n degrees of freedom (as it is easy to see by considering the
moment-generating functions). Therefore, the residual computed according to (3)
2
over a N(cid:2)N window W is distributed as a chi-square with N degrees of freedom:
X(cid:2) (cid:3)2 2 2
(cid:15) = I((cid:14)(x);t)(cid:0)I(x;0) (cid:17) (cid:31) (N ): (17)
W
Figure 2 near here
y
(cid:17) means that the variableto the left has the probabilitydistribution speci(cid:12)ed to the right.
8