Motion and Structure from Image Sequences

457 Pages·1993·14.172 MB·English

by  Professor Juyang Weng, Professor Thomas S. Huang, Professor Narendra Ahuja (auth.)

Springer Series in Information Sciences 29
Editor: Thomas S. Huang
Springer Series in Information Sciences
Editors: Thomas S. Huang Teuvo Kohonen Manfred R. Schroeder
Managing Editor: H. K. V. Lotsch
Content-Addressable Memories 16 Radon and Projection Transform-
By T. Kohonen 2nd Edition Based Computer Vision
2 Fast Fourier Transform and Algorithms, A Pipeline Architecture, and
Convolution Algorithms Industrial Applications By J. L. C. Sanz,
By H. J. Nussbaumer 2nd Edition E. B. Hinkle, and A. K. Jain
3 Pitch Determination of Speech Signals 17 Kalman Filtering
Algorithms and Devices By W. Hess with Real-TIme Applications
By C. K. Chui and G. Chen 2nd Edition
4 Pattern Analysis and Understanding
By H. Niemann 2nd Edition 18 Linear Systems and Optimal Control
By C. K. Chui and G. Chen
5 Image Sequence Analysis
Editor: T. S. Huang 19 Harmony: A Psychoacoustical Approach
By R. Parncutt
6 Picture Engineering
Editors: King-sun Fu and T. L. Kunii 20 Group-Theoretical Methods
in Image Understanding
7 Number Theory in Science
By Ken-ichi Kanatani
and Communication With Applications
in Cryptography, Physics, Digital 21 Linear Prediction Theory
Information, Computing, and Self- A Mathematical Basis
Similarity By M. R. Schroeder for Adaptive Systems
2nd Edition By P. Strobach
8 Self-Organization 22 Psychoacoustics Facts and Models
and Associative Memory By E. Zwicker and H. Fastl
By T. Kohonen 3rd Edition 23 Digital Image Restoration
9 Digital Picture Processing Editor: A. K. Katsaggelos
An Introduction By L. P. Yaroslavsky 24 Parallel Algorithms
10 Probability, Statistical Optics, in Computational Science
and Data Testing By D. W. Heermann and A. N. Burkitt
A Problem Solving Approach 25 Radar Array Processing
By B. R. Frieden 2nd Edition Editors: S. Haykin, J. Litva,
11 Physical and Biological Processing and T. J. Shepherd
ofimages Editors: O. J. Braddick 26 Signal Processing and Systems Theory
and A. C. Sleigh Selected Topics
12 Multiresolution Image Processing By C. K. Chui and G. Chen
and Analysis Editor: A. Rosenfeld 27 3D Dynamic Scene Analysis
13 VLSI for Pattern Recognition and A Stereo Based Approach
Image Processing Editor: King-sun Fu By Z. Zhang and O. Faugeras
14 Mathematics of Kalman-Bucy Filtering 28 Theory of Reconstmction
By P. A. Ruymgaart and T. T. Soong from Image Motion
2nd Edition By S. Maybank
15 Fundamentals 29 Motion and Stmcture
of Electronic Imaging Systems from Image Sequences
Some Aspects of Image Processing By J. Weng, T. S. Huang,
By W. F. Schreiber 3rd Edition andN.Ahuja
Juyang Weng Thomas S. Huang
Narendra Ahuja
Motion and Structure
from Image Sequences
With 119 Figures
Springer -Verlag
Berlin Heidelberg New York
London Paris Tokyo
Hong Kong Barcelona
Budapest
Professor Juyang Weng
Professor Thomas S. Huang
Professor Narendra Ahuja
University of Illinois at Urbana-Champaign, Beckman Institute,
405 North Mathews Ave., Urbana, IL 61801, USA
Series Editors:
Professor Thomas S. Huang
Department of Electrical Engineering and Coordinated Science Laboratory,
University of Illinois, Urbana, IL 61801, USA
Professor Teuvo Kohonen
Laboratory of Computer and Information Sciences, Helsinki University of Technology,
SF-02150 Espoo 15, Finland
Professor Dr. Manfred R. Schroeder
Drittes Physikalisches Institut, Universitlit Gottingen, Biirgerstrasse 42-44,
W-3400 Gottingen, Fed. Rep. of Germany
Managing Editor: Dr.-Ing. Helmut K. V. Lotsch
Springer-Verlag, TIergartenstrasse 17,
W-6900 Heidelberg, Fed. Rep. of Germany
ISBN-13:978-3-642-77645-8 e-ISBN-13:978-3-642-77643-4
DOl: 10.1007/978-3-642-77643-4
Library of Congress Cataloging-in-Publication Data. Weng, Juyang, 1957-Motion and structure from image se
quences I Juyang Weng, Thomas S. Huang, Narendra Ahuja. p. cm. -(Springer series in information sciences; 29)
Includes bibliographical references and index. ISBN-13:978-3-642-77645-8
1. Image processing. 2. Computer vision. I. Huang, Thomas S., 1936-. II. Ahuja, Narendra, 1950-. III. Title.
IY. Series. TA 1632.W46 1993 006.4'2-dc 92-28268
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned,
specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on
microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is per
mitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and
permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the
German Copyright Law.
© Springer-Verlag Berlin Heidelberg 1993
Softcover reprint of the hardcover lst edition 1993
The use of general descriptive names, registered names, trademarks, etc. in this pUblication does not imply, even
in the absence of a specific statement, that such names are exempt from the relevant protective laws and regula
tions and therefore free for general use.
Typesetting: Camera-ready by authors
54/3140-5 4 3 2 1 0 -Printed on acid-free paper
Preface
This book has grown out of our research activities in the area of
deriving geometric and motion descriptions of dynamic scenes from
visual images. It should prove useful for researchers who are
interested in a comprehensive treatment of this topic and practitioners
who wish to find algorithms they can use.
The results presented here are mostly very recent, some of them
being published here for the first time. A rich collection of practical
algorithms is presented, applicable to monocular or stereo views.
Effort has been made to present a balanced treatment of the
theoretical and practical issues. Theoretical issues of broad and prac
tical significance have been explored in detail, while other more
specific ones are not covered. The topics covered in detail include
image matching and motion estimation, error analysis, error estima
tion, stability, optimality, and experiments with simulated and real
image data, with the objective being to present general techniques that
have a mathematical foundation and engineering applications.
This book is meant for graduate students and practicing engineers
who are interested in computer vision. The organization of the
material is intended to facilitate extraction and use of the algorithms
without necessitating study of the rest of the material.
Chapter 1 introduces the problem of motion analysis and briefly
reviews past work. Chapter 2 is concerned with image matching, a
step which is prerequisite for all further discussions in the book.
Chapter 3 presents a closed-form solution for motion and structure
parameters from two views of a rigid object. In the presence of noise,
this solution is required as an initial condition for the optimization
approach investigated in Chapter 4. The principles covered in Chapter
4 are used in later chapters whenever optimization is discussed.
Chapters 5 and 6 address two separate issues, planar surfaces and line
features for motion analysis, each of which can be skipped without
VI Preface
loss of continuity. While Chapters 3 to 6 all assume monocular
views, Chapter 7 is devoted to the analysis of image pairs acquired
using stereo cameras. The techniques of robust statistics introduced in
Chapter 7 can also be applied to monocular images. The subject of
analyzing long image sequences, instead of only two or three views, is
also treated in Chapter 7. Modeling and prediction of motion are
covered in Chapter 8. Although both Chapter 7 and Chapter 8 con
sider long image sequences, they are different. Chapter 7 does not
impose any restriction on changes in motion with time, and thus inter
frame motions can be very different from one to the next. In Chapter
8, however, the motion is assumed to be smooth in the sense that it
follows certain specified, although not very restrictive, object dynam
ics. The method in Chapter 7 is useful when motion is unpredictable
and quite irregular, while that in Chapter 8 can be used when motion
is known to be smooth and therefore predictable. The following flow
chart indicates the interdependence among the material in different
chapters.
Chapter 5
~
a..,.",
0_ 1 - 2 _ Ch,p"" 3 - <lapltt 4 ( Ch,,,,,, 7 - ,
Chapter 6
Most of the work presented in this book was done as part of the
Ph.D. program of one of the authors (J. W.) at the University of llli
nois during 1985-1988, while the other two authors were his research
advisors. A small portion of the work was done during 1989-1990
when one of the authors (J. W.) was with Centre de Recherche Infor
matique de Montreal and Ecole Poly technique, University of Montreal,
Montreal, Canada. Many colleagues have had valuable influence on
this book. First and foremost, we are indebted to Roger Y. Tsai and
Bob Yen, former Ph.D. students of one of the authors (T. S. H.), for
their pioneering work in motion estimation using points and lines,
respectively. During 1986, intensive discussions with Xinhua Zhuang
significantly helped us in developing the algorithm presented in
Chapter 3. Collaborations with J. K. Aggarwal greatly influenced our
work on optimization. Discussions with Rama Chellappa made us
investigate the inherent stability of motion problems more critically.
The conjecture of a sufficient condition for the decomposability of the
Preface vn
E matrix came to our attention during our interaction with Olivier
Faugeras. Discussions with John Aloimonos and Minas Spetsakis trig
gered our investigation on the equivalence between two methods of
linear motion estimation. Interaction with colleagues Bill Hoff, Steve
Blostein, Homer Chen, Christian Debrunner, Yuncai Liu, Dorothea
Blostein, Edward Altman, Toshifumi Tsukiyama, Gary Silverman,
Lynn Abbott, Dmitry Goldgof, Paul Cohen, Ning Cui, and Nicola
Rebibo had a significant impact on the work presented here.
Much of our work in motion estimation has been supported by the
National Science Foundation. We gratefully acknowledge the encou
ragement and support of Drs. E. Barrett, S. S. Chen, Y. T. Chien, K.
Laws, and H. Moraff.
Juyang Weng
Urbana, lllinois Thomas S. Huang
December 1991 Narendra Ahuja
Contents
1. Introduction ............................................................................... 1
1.1 Fonnulation of the Problem ............................................... 2
1.2 Overview of Recent Progress ............................................. 4
1.3 Bibliographical Notes ......................................................... 6
2. Image Matching .......................... .......... .................................... 7
2.1 Approaches to Establishing Correspondences ................... 7
2.2 An Approach to Image Matching ...................................... 10
2.2.1 Image Attributes ....................................................... 12
2.2.2 Relationships of the Attributes ................................ 14
2.2.3 Intra-Regional Smoothness and Occlusion ............. 14
2.2.4 Multi-Resolution Multi-Grid Structure ................... 17
2.2.5 Limitations ....................................... ......................... 19
2.3 Algorithm .... ........................... ........................ ..................... 20
2.3.1 Locally Rigid Motion .............................................. 20
2.3.2 3-D Motion and Image Plane Motion ..................... 21
2.3.3 Motion Insensitive Image Attributes ........... ............ 24
2.3.4 Edgeness and Comemess .... .................................... 26
2.3.5 Smoothness ...... ........... ..................... ......................... 29
2.3.6 Minimizing Residuals .............................................. 31
2.3.7 Recursive Blurring ............ ....................................... 34
2.3.8 Preprocessing and Nonnalization ............................ 35
2.3.9 Outline of the Matching Algorithm ........................ 37
2.4 Motion and Structure Computation ................................... 38
2.5 Refinements ................ .................... ..................................... 39
X Contents
2.5.1 Bottom-Up and Top-Down ...................................... 39
2.5.2 Refinement Based on Rigidity................................. 40
2.6 Exrunples ............................................................................. 41
2.7 Summary ............................................................................. 59
2.8 Bibliographical Notes ......................................................... 61
Appendix ............................................................................. 62
3. Two-View Analysis ................................................................... 65
3.1 Some Basic Issues .............................................................. 65
3.2 An Algorithm ...................................................................... 67
3.2.1 Problem Statement ............ ....................................... 68
3.2.2 Algorithm .. ..................... .......................................... 70
3.2.3 Justification of the Algorithm .................................. 73
3.3 Error Estimation ................................................................. 78
3.3.1 Algorithm and Perturbation ..................................... 79
3.3.2 Perturbation of Eigenvalues and Eigenvectors ....... 81
3.3.3 Error Estimation for the Algorithm ......................... 83
3.4 Error Analysis ..................................................................... 90
3.4.1 Structure of the Scene .... .......................................... 90
3.4.2 Motion Parruneters ................................................... 91
3.4.3 System Parameters ..... ................... ........................... 97
3.5 Performance ........................................................................ 98
3.5.1 Performance .............................................................. 99
3.5.2 Error Estimation ....................................................... 100
3.5.3 Error versus Motion and System Parameters .......... 106
3.5.4 Real Images .............................................................. 115
3.6 Summary ............................................................................. 117
3.7 Bibliographical Notes ......................................................... 118
Appendix 3.A Perturbation of Eigenspace ....................... 119
Appendix 3.B Quaternions ................................................ 122
Appendix 3.C Alternative Fitting ..................................... 125
4. Optimization .............................................................................. 127
4.1 Motivations ......................................................................... 127
4.2 Stability of Linear Algorithms ........................................... 129
Contents XI
4.2.1 The Linear Algorithms ............................................ 131
4.2.2 The Epipolar Constraint ........................................... 132
4.2.3 Using the Constraint in Matrix E ............................ 134
4.2.4 A Type of Motion .................................................... 135
4.2.5 Beyond the Epipolar Constraint .............................. 140
4.3 Maximum Likelihood Estimation ...................................... 141
4.3.1 Gaussian Distribution ............................................... 143
4.3.2 Uncertainty Polyhedron Distribution ....................... 146
4.4 A Two-Step Approach and Computation .......................... 149
4.5 Minimum Variance Estimation .......................................... 154
4.6 Error Estimation and Error Bounds ................................... 157
4.6.1 Error Estimation ....................................................... 157
4.6.2 Error Bounds ............................................................ 158
4.7 Batch and Sequential Methods ........................................... 162
4.7.1 Linear Systems ......................................................... 163
4.7.2 Nonlinear Systems ................................................... 166
4.7.3 Recursive-Batch Solution ........................................ 172
4.8 Numerical Examples ........................................................... 174
4.8.1 Whether a Good Initial Guess Is Necessary ........... 174
4.8.2 Epipolar Improvement versus Optimization ........... 175
4.8.3 Sequential versus Batch Solutions .......................... 176
4.8.4 Uncertainty Polyhedron Model................................ 185
4.8.5 Error Estimation and Error Bound .......................... 188
4.8.6 Inherent Limitation of Optical Flow....................... 193
4.8.7 Real World Images .................................................. 195
4.9 Summary ............................................................................. 195
4.10 Bibliographical Notes ......................................................... 197
Appendix 4.A Decomposability........................................ 197
Appendix 4.B Weights ...................................................... 199
Appendix 4.C Triangulation with Noise .......................... 200
Appendix 4.D Matrix Derivatives ........ ............................ 202
Appendix 4.E Cramer-Rao Bound .................................... 203
5. Planar Scenes ................................................................ ............ 205
5.1 Planar Scene as a Degenerate Case ................................... 205