Table Of ContentSpringer Series in Information Sciences 28
Editor: Thomas S. Huang
Springer Series in Information Sciences
Editors: Thomas S. Huang Teuvo Kohonen Manfred R. Schroeder
Managing Editor: H. K. V. Lotsch
Content-Addressable Memories 16 Radon and Projection 1ransform
By T. Kohonen 2nd Edition Based Computer Vision
Algorithms, A Pipeline Architecture, and
2 Fast Fourier 1ransform and
Industrial Applications By J. L. C. Sanz,
Convolution Algorithms
E. B. Hinkle, and A. K. Jain
By H. J. Nussbaumer 2nd Edition
17 Kalman Filtering
3 Pitch Determination of Speech Signals
Algorithms and Devices By W. Hess with Real-Time Applications
By C. K. Chui and G. Chen 2nd Edition
4 Pattern Analysis and Understanding
By H. Niemann 2nd Edition 18 Linear Systems and Optimal Control
By C. K. Chui and G. Chen
5 Image Sequence Analysis
Editor: T. S. Huang 19 Harmony: A Psychoacoustical Approach
By R. Parncutt
6 Picture Engineering
Editors: King-sun Fu and T. L. Kunii 20 Group-Theoretical Methods
in Image Understanding
7 Number Theory in Science
By Ken-ichi Kanatani
and Communication With Applications
in Cryptography, Physics, Digital 21 Linear Prediction Theory
Information, Computing, and Self A Mathematical Basis
Similarity By M. R. Schroeder for Adaptive Systems
2nd Edition By P. Strobach
8 Self-Organization 22 Psychoacoustics Facts and Models
and Associative Memory By E. Zwicker and H. Fast!
By T. Kohonen 3rd Edition 23 Digital Image Restoration
9 Digital Picture Processing Editor: A. K. Katsaggelos
An Introduction By L. P. Yaroslavsky 24 Parallel Algorithms
10 Probability, Statistical Optics, in Computational Science
and Data Testing By D. W. Heermann and A. N. Burkitt
A Problem Solving Approach 25 Radar Array Processing
By B. R. Frieden 2nd Edition Editors: S. Haykin, J. Litva,
11 Physical and Biological Processing and T. J. Shepherd
oflmages Editors: O. J. Braddick 26 Signal Processing and Systems Theory
and A. C. Sleigh Selected Topics
12 Multiresolution Image Processing By C. K. Chui and G. Chen
and Analysis Editor: A. Rosenfeld 27 3D Dynamic Scene Analysis
13 VLSI for Pattern Recognition and A Stereo Based Approach
Image Processing Editor: King-sun Fu By Z. Zhang and O. Faugeras
14 Mathematics of Kalman-Buey Filtering 28 Theory of Reconstruction
By P. A. Ruymgaart and T. T. Soong from Image Motion
2nd Edition By S. Maybank
15 Fundamentals 29 Motion and Structure
of Electronic Imaging Systems from Image Sequences
Some Aspects of Image Processing ByJ. Weng, T.S. Huang,
By W. F. Schreiber 3rd Edition andN. Ahuja
Stephen Maybank
Theory of Reconstruction
from Image Motion
With 29 Figures
Springer-Verlag
Berlin Heidelberg New York
London Paris Tokyo
Hong Kong Barcelona
Budapest
Dr. Stephen Maybank
GEC-Marconi Limited. Hirst Research Centre,
East Lane, Wembley, Middlesex, HA9 7PP, UK
Series Editors:
Professor Thomas S. Huang
Department of Electrical Engineering and Coordinated Science Laboratory,
University of Illinois, Urbana, IL 61801, USA
Professor Teuvo Kohonen
Laboratory of Computer and Information Sciences, Helsinki University of Technology,
SF-02150 Espoo 15, Finland
Professor Dr. Manfred R. Schroeder
Drittes Physikalisches Institut, Universitat Gottingen, Biirgerstrasse 42-44,
W-3400 Gottingen, Fed. Rep. of Germany
Managing Editor: Dr.-Ing. Helmut K. V. Lotsch
Springer-Verlag, Tiergartenstrasse 17,
W-6900 Heidelberg, Fed. Rep. of Germany
ISBN- 13: 978-3-642-77559-8 e-ISBN -13: 978-3-642-77557-4
DOl: 10.1007/978-3-642-77557-4
Library of Congress Cataloging-in-Publication Data. Maybank, Stephen, 1954-. Theory of reconstruction from
image motion / Stephen Maybank. p.cm. - (Springer series in information sciences; 28). Includes bibliographical
references and index. ISBN-13:978-3-642-77559-8 1. Computer vision. 2. Image processing. I. Title. II. Series.
TA1632.M3625 1993 006.3'7-dc 92-17979
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned,
specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on
microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is per
mitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and
permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the
German Copyright Law.
© Springer-Verlag Berlin Heidelberg 1993
Softcoverreprint of the hardcover 1st edition 1993
The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even
in the absence of a specific statement, that such names are exempt from the relevant protective laws and regula
tions and therefore free for general use.
Typesetting: Data conversion by Springer-Verlag
Production Editor: P. Treiber
54/3140-5432 1 0 - Printed on acid-free paper
Preface
The image taken by a moving camera changes with time. These image motions
contain information about the motion of the camera and about the shapes of the
objects in the field of view. There are two main types of image motion, finite
displacements and image velocities. Finite displacements are described by the
point correspondences between two images of the same scene taken from different
positions. Image velocities are the velocities of the points in the image as they
move over the projection surface. Reconstruction is the task of obtaining from
the image-motions information about the camera motion or about the shapes of
objects in the field of view. In this book the theory underlying reconstruction is
described.
Reconstruction from image motion is the subject matter of two different sci
entific disciplines, photogrammetry and computer vision. In photogrammetry
the accuracy of reconstruction is emphasised; in computer vision the emphasis
is on methods for obtaining information from images in real time in order to
guide a mechanical device such as a robot arm or an automatic vehicle. This
book arises from recent work carried out in computer vision. Computer vision is
a young field but it is developing rapidly. The earliest papers on reconstruction
in the computer vision literature date back only to the mid 1970s. As computer
vision develops, the mathematical techniques applied to the analysis of recon
struction become more appropriate and more powerful. The advances in the
theory are matched by the advances in the speed and capacity of the electronic
equipment available for computer vision. It is now possible to implement in real
time methods which were of only theoretical interest ten years ago.
The book is divided into six chapters. Chapter 1 is an introduction. Chap
ters 2 and 3 describe reconstruction from image correspondences, with particular
emphasis on the ambiguous case, in which the same set of image correspondences
yields two or more different reconstructions. Chapter 4 describes reconstruction
from image velocities, again with an emphasis on the ambiguous case. Chap
ter 5 describes cases in which the data for reconstruction are just sufficient to
ensure that only a finite number of reconstructions are possible. In Chap. 6 four
algorithms for reconstruction from image motions are described. Two of the al
gorithms require image correspondences, and the remaining two require image
velocities.
VI Preface
The chapters are divided into sections and the sections are divided into sub
sections. For example Sect. 2.1.1 is Subsection 1 of Section 1 of Chapter 2.
Most of the results are stated either as theorems or as propositions. A proposi
tion is usually less important or easier to prove than a theorem. The theorems,
propositions and definitions are labelled in sequence, chapter by chapter. The
numbering of the sections and subsections within a chapter is independent of the
numbering of the theorems. The end of a proof of a theorem or a proposition is
marked by a box, 0, at the right-hand margin of the final line. The references
are placed at the end of each chapter.
To return to the contents of the chapters in more detail: Chapter 1 describes
the background to reconstruction. Certain useful mathematical concepts and
notation are introduced. Two different but mathematically equivalent frame
works for reconstruction from point correspondences are described in Chap. 2,
namely the Euclidean framework and the projective geometric framework. The
Euclidean framework emphasises the description of the image in terms of coor
dinates, whilst the projective geometric framework emphasises the description
of the image in terms of the lines in space passing through the optical centre
of the camera. The ambiguous case of reconstruction is discussed within both
frameworks. The properties of certain 3 x 3 matrices known as essential matri
ces are described. These matrices arise naturally in the Euclidean framework.
They summarise the information about the camera displacement contained in a
set of image correspondences. Reconstruction up to a collineation is discussed
briefly at the end of Chap. 2. In this approach the full camera calibration is not
required, but the reconstruction has a higher degree of ambiguity.
The geometry of the ambiguous case of reconstruction from image correspon
dences is developed in Chap. 3. Ambiguity is only possible if the points giving
rise to the image correspondences lie on certain surfaces of degree two known as
critical surfaces. There are at least two critical surfaces involved in each case
of ambiguity, one for each possible reconstruction. The intersection of two crit
ical surfaces contains a curve known as a horopter curve. Many properties of
critical surfaces are directly related to the properties of horopter curves. The
critical surfaces for the ambiguous case of reconstruction up to a collineation are
discussed.
Chapter 4 describes the reconstruction of camera velocity from image veloc
ities. It is shown that the ambiguous case is similar in many respects to the
ambiguous case of reconstruction from image correspondences. The constraints
on the camera velocity arising from four image velocity vectors are examined in
detail. The translational velocity is subject to a quartic polynomial constraint.
If the image velocity field is irregular then the leading order terms of the quartic
split into two linear factors and a quadratic factor.
Chapter 5 discusses three cases of reconstruction in which the data obtained
from the images are just sufficient to ensure that only a finite number of re
constructions are possible. The three cases are reconstruction from point cor
respondences, reconstruction from image velocities and reconstruction up to a
Preface VII
collineation. In these minimal cases the algebraic nature of reconstruction is
particularly apparent. Reconstruction is formulated as the problem of finding
the common zeros of a particular set of multivariate polynomials. The degree of
a reconstruction problem is defined to be the number of reconstructions obtained
in the minimal case. If the degree is high then reconstruction is likely to be in
trinsically difficult, regardless of the particular algorithm employed. The degree
of reconstruction from point correspondences and from image velocities is ten in
both cases. The degree of reconstruction up to a collineation is three.
In Chapter 6, four different reconstruction algorithms are described. Two are
for reconstruction from image correspondences, and two are for reconstruction
from image velocities. In all four algorithms it is assumed that the camera
calibration is known. The algorithms for reconstruction from image velocities
illustrate the importance of irregularities in the image velocity field. If the moving
surface has a large change in depth or orientation over a small part of the field
of view then reconstruction can be carried out using algorithms that are simpler
than those designed for the general case. The results obtained are also more
stable in the presence of noise.
The thesis on which part of this book is based began with a quotation from
Machado suggested by Mari Carmen, 'Caminante no hay camino. Se hace camino
al andar'. (Traveller there are no paths, Paths are made by walking.) Like a
path, science is constructed, but at the same time it is not made arbitrarily. It
has to go somewhere.
Sophia Antipolis S.J. Maybank
January 1992
Acknowledgements
Chapter 4 and Sect. 6.2 of this book are adapted from a Ph.D. thesis written whilst I was at
Marconi Command and Control Systems Ltd., Frimley, UK (MCCS). The thesis is based on
work carried out as part of a contract between the Royal Signals and Radar Establishment,
Malvern, UK (RSRE) and MCCS. I thank MCCS and RSRE for the opportunity to write the
thesis, and for permission to incorporate material into this book. The presentation, clarity and
coherence of the thesis owe much to the untiring efforts of my supervisor, George Loizou, of
Birkbeck College, University of London. Birkbeck is unique amongst British colleges in that the
majority of the students are part time. I enrolled as a graduate student, and at the same time
continued in employment at MCCS. The thesis was completed with the support of ESPRIT
Project P940 (DMA). The revision of the thesis necessary for this book and the addition of
new material was made possible by the support of ESPRIT Project P2502 (Voila). I thank the
GEe-Marconi Hirst Research Centre, Wembley, for their support and encouragement during
the writing of this book after my transfer there in April 1989. The final editing was carried
out during a visit to INRIA, Sophia Antipolis, France.
I thank the many people that I have been fortunate enough to meet in the course of my work
in computer vision. In particular, I am grateful to Bernard Buxton, David Castelow, David
Corrall, Olivier Faugeras, Pavel Grossmann, Richard Johnson, Christopher Longuet-Higgins,
George Loizou and Zhengyou Zhang. The book was typeset in Jb.TE», using macros written
by Zhengyou Zhang. I thank Robert Maybank for reading a preliminary draft and noting the
many places where improvement was needed. Thanks are also due to Tom Huang for suggesting
that the thesis should be published as a book, and to Dr. Lotsch of Springer-Verlag for making
publication possible.
Contents
1 Introduction ... 1
1.1 Background.. 1
1.2 Reconstruction 3
1.3 Conventions About the Image . 6
1.4 Mathematical Background . . . 7
1.4.1 Terminology ..... 8
1.4.2 Euclidean Space and Projective Space 9
1.4.3 Algebraic Curves 12
References. . . . . . . . . . . . . . . . . . . . . . . 14
2 Reconstruction from Image Correspondences. 17
2.1 Euclidean Framework for Reconstruction. 18
2.1.1 Euclidean Treatment of Ambiguity .. 22
2.1.2 Ambiguity and Instability ....... . 27
2.1.3 The Maximum Number of Reconstructions. 28
2.2 Essential Matrices ... . . . . . . . . . . . . . 36
2.2.1 A Characterisation of Essential Matrices 38
2.2.2 The Singular Value Decomposition 44
2.2.3 Symmetric and Antisymmetric Parts 45
2.2.4 Ambiguity.............. 50
2.3 Projective Framework for Reconstruction. 55
2.3.1 The Epipolar Transformation . . . 56
2.3.2 Ambiguity.............. 58
2.3.3 The Intersection of a Critical Surface Pair 61
2.4 Reconstruction up to a Collineation. . . . . . . 62
2.4.1 Reconstruction Based on the Epipolar Transformation 63
2.4.2 Ambiguity... 66
2.4.3 Critical Surfaces 68
References. . . . . . . . . . 70
3 Critical Surfaces and Horopter Curves 73
3.1 The Absolute Conic ......... . 74
X Contents
3.1.1 The Absolute Conic and Camera Calibration 79
3.1.2 Involutions and the Absolute Conic ..... . 81
3.2 Rectangular Quadrics ................. 84
3.2.1 Algebraic Characterisations of Rectangular Quadrics 87
3.2.2 Rigid Involutions of Rectangular Quadrics 91
3.3 Horopter Curves . . . . . . . . . . . . . . . . 94
3.3.1 Characterisations of Horopter Curves. 94
3.3.2 Rigid Involutions of Horopter Curves 97
3.3.3 The Centre of a Horopter Curve ... 98
3.3.4 Examples................ 100
3.3.5 Horopter Curves on Rectangular Quadrics 102
3.4 Horopter Curves and Reconstruction . . . . . . . 106
3.4.1 A Formula for r,p ...........•.. 108
3.4.2 Two Cubic Constraints on Critical Surfaces 110
3.4.3 An Example ........ . 112
3.5 Reconstruction up to a Collineation . 114
References. . . . . . . . . . . . . . . . . . 118
4 Reconstruction from Image Velocities 119
4.1 Framework ........ . 120
4.2 Ambiguity.......... 123
4.2.1 Preliminary Results 124
4.2.2 Critical Surfaces . . 126
4.2.3 Singular Critical Surfaces 128
4.2.4 Critical Surface Pairs. . . 129
4.2.5 Cubic Polynomial Constraints on Critical Surfaces 132
4.2.6 The Maximum Number of Reconstructions. . 133
4.3 Algebraic Properties of Four Image Velocity Vectors 136
4.3.1 The Quartic Polynomial Constraint. 137
4.3.2 Irregular Image Velocity Fields . . . . . . . . 138
4.3.3 The Effects of Small Perturbations . . . . . . 142
4.3.4 Symmetric Arrangements of Base Points: The Square. 144
4.3.5 Symmetric Arrangements of Base Points: The Tripod. 150
4.4 The Linear System of Quartics ....... 153
4.4.1 The Dimension of the Linear System 154
4.4.2 Singular Points of Quartics ..... 157
4.4.3 Base Points of the Linear System . . 158
4.5 The Derivatives of the Image Velocity Field 162
4.5.1 The Derivatives to Second Order . . 162
4.5.2 Polynomial Constraints on the Translational Velocity . 166
4.5.3 Time to Contact 169
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . " 170
