Table Of ContentSpringer Series in Information Sciences 33
Editor: Thomas S. Huang
Springer
Berlin
Heidelberg
New York
Barcelona
Budapest
Hong Kong
London
Milan
Paris
Santa Clara
Singapore
Tokyo
Springer Series in Information Sciences
Editors: Thomas S. Huang Teuvo Kohonen Manfred R. Schroeder
Managing Editor: H. K. V Lotsch
30 Self-Organizing Maps
By T. Kohonen 2nd Edition
31 Music and Schema Theory
Cognitive Foundations
of Systematic Musicology
By M. Leman
32 The Maximum Entropy Method
By N. Wu
33 A Few Steps Towards 3D Active Vision
By T. Vieville
34 Calibration and Orientation of Cameras in Computer Vision
Editors: A. Griin and T. S. Huang
35 Speech Processing: Fundamentals and Applications
By B. S. Atal and M. R. Schroeder
Volumes 1-29 are listed at the end of the book.
Thierry Vie ville
A Few Steps
Towards
3D Active Vision
With 58 Figures
Springer
Professor Thierry Vieville
INRIA, Projet Robotvis
Unite de Recherche Sophia Antipolis
2004, route des Lucioles, BP. 93
F-06902 Sophia Antipolis Cedex, France
Series Editors:
Professor Thomas S. Huang
Department of Electrical Engineering and Coordinated Science Laboratory,
University of Illinois, Urbana, IL 61801, USA
Professor Teuvo Kohonen
Helsinki University of Technology, Neural Networks Research Centre,
Rakentajanaukio 2C, FIN-02IS0 Espoo, Finland
Professor Dr. Manfred R. Schroeder
Drittes Physikalisches Institut, Universitat Giittingen, Biirgerstrasse 42-44,
0-37073 Giittingen, Germany
Managing Editor:
Dr.-Ing. Helmut K. V. Lotsch
Springer-Verlag, Tiergartenstrasse 17,
0-69121 Heidelberg, Germany
Library of Congress Cataloging-in-Publication Data
Vieville, Thierry, 1959-
A few steps towards 3D active vision 1 Thierry Vieville. p. cm. - (Springer series in information sciences; 33)
Includes bibliographical references and index.
ISBN-13: 978-3-642-64580-8
I. Robot vision. 2. Computer vision. I. Title. II. Series.
TJ211.3. V54 1997 629.8'92637-dc21 97-19673 CIP
ISBN-13: 978-3-642-64580-8 e-ISBN-13: 978-3-642-64580-8
001.10.1007/978-3-642-64580-8
Springer-Verlag Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned,
specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on
microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is
permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version,
and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under
the German Copyright Law.
© Springer-Verlag Berlin Heidelberg 1997
Softcover reprint of the hardcover 1st edition 1997
The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even
in the absence ofa specific statement, that such names are exempt from the relevant protective laws and regulations
and therefore free for general use.
Typesetting: Camera ready by the author using a Springer TEX macro package
Cover design: design & production GmbH, Heidelberg
SPIN: 10507842 54/3144 -5432 I 0 -Printed on acid-free paper
To Alice and Adele
Preface
This book aims to analyse a specific problem in the field of active vision: how
suitable is it to explicitly use 3D visual cues in an reactive visual task? In
order to answer this question, we have collected a set of studies on this subject
and have used these experimental and theoretical developments to propose a
synthetic view on this problem, completed by some specific experimentations.
We first propose, through a short survey of the active vision problem, a
precise definition of "3D active vision" and analyse, from a general point of
view, the basic requirements for a reactive system to integrate 3D visual cues,
and on the reserve, how can a reactive paradigm help to obtain 3D visual
information. This discussion is intentionally very general and does not make
any assumption about the particular architecture of the visual system, but is
based on experimental data found in the literature, obtained on many other
systems.
The next step is to verify whether these general ideas can easily be imple
mented, considering an effective active visual system. This is the goal of the
second chapter in which we describe a mechanical system which has been de
signed to facilitate the fixation and tracking of 3D objects. The mechanics are
described first and a framework to deal with eye-neck control is proposed.
The visual mechanisms able to detect visual targets and effectively realise
their 3D observations are described in the second part of this chapter. An
important issue is that all these mechanisms are adaptive in the sense that
strategies to automatically adapt their parameters are proposed.
A step further, the main problem when considering 3D vision on a reac
tive visual system is the problem of auto-calibration. The third chapter deals
with this problem, in the case of a robotic head. The specificity of the ap
proach is that deep involvements in difficult concepts of projective Geometry
are avoided. Moreover, the auto-calibration strategy has been optimized con
sidering a robotic head and the uncertainty about the calibration parameters
is derived.
As soon as the mount is calibrated, we can - as in a biological visual sys
tem - make the visual layers of the system collaborate with other perceptual
cues. The most relevant odometric cue, in this case, is inertial cue. This is
studied in details in the fourth chapter, and shows how the usual structure
of the motion paradigm can be simplified and enhanced when performed in
VIII Preface
cooperation with inertial cues. This also leads to high-level mechanisms of
image stabilization, in which rotational and translational components of the
system self-motion are treated separately.
As far as 3D visual perception is concerned, the retinal motion field is a
very informative cue. However, we must consider that the camera calibration
is not always known with precision, and that several objects with different
motions are likely to be observed at the same time. This level of complexity
is not covered by standard tools, but this gap is filled in the last chapter,
where the motion field is analysed under these very general conditions, with
robust mechanisms of estimation and several methods to segment object, in
motion, detect planar structures, etc.
It is clear, at the end of these developments, that a reactive visual system,
providing that it has been designed with some auto-calibration capabilities,
is using inertial cues in cooperation with vision and is performing an analysis
of the retinal motion in a suitable way, can definitely have 3D active visual
perception capabilities. This has been demonstrated by both theoretical ar
guments and several experimentations. Does it mean that "the problem is
solved"? Not at all. On the contrary, the gate is open now to realize much
more complex reactive behaviors, based on 3D visual cues. This is briefly
described in the conclusion.
Olivier Faugeras is gratefully acknowledged for some powerful ideas which
are at the origin of this work.
A lot of thanks to Emmanuelle Clergue, Reyes Enciso, Bernard Giai
Checa, Herve Mathieu, Luc Robert and Cyril Zeller which have highly con
tributed to some aspects of this work.
A big thank-you to Philippa Hook and Veit Schenk for their corrections.
Sophia-Antipolis, July 1997 Thierry Vieville
Contents
1. From 2D to 3D Active Vision. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 The Concept of Active Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Can Reactive Vision Be "Better"
Than Passive Vision? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 A Blink at the State of the Art in Active Vision. . . . . 4
1.2 A Short Review of Existing Active Visual Systems. . . . . . . . . 5
1.2.1 Active Visual Sensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.2 Control for Active Vision. . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.3 Auto-Calibration of an Active Visual System ... .... 8
1.2.4 Perception of 3D Parameters Using Active Vision ... 9
1.3 Architecture of an Active Visual System. . . . . . . . . . . . . . . . . . 10
1.3.1 The Three Main Functions
of an Active Visual System. . . . . . . . . . . . . . . . . . . . . . . 10
1.3.2 Basic Requirements of a Visual System. . . . . . . . . . . . . 12
1.4 2D Versus 3D Vision in an Active Visual System. . . . . . . . . . 13
1.4.1 Active Vision and 2D Visual Servoing . . . . . . . . . . . . . . 13
1.4.2 Introduction of 3D Vision in Visual Loops. . . . . . . . . . 15
1.4.3 Basic Modules for a 3D Active Visual System. . . . . . . 16
1.5 Gaze Control in 3D Active Visual Systems. . . . . . . . . . . . . . . . 18
2. 3D Active Vision on a Robotic Head ....... .. ..... ...... 23
2.1 A One-to-One 3D Gaze Controller. . . . . . . . . . . . . . . . . . . . . . . 23
2.1.1 Technical Data on the Robotic Head. . . . . . . . . . . . . . . 24
2.1.2 Computing the Inverse Kinematic for 3D Fixation. . . 25
2.2 Active Observation of a 3D Visual Target .. .............. 28
2.2.1 A ID Model of the Eye-Neck Tracking. . . . . . . . . . . . . 29
2.2.2 A Linearized Adaptive ID Model
of the Eye-Neck Tracking. . . . . . . . . . . . . . . . . . . . . . . . 30
2.2.3 Statistical Filtering of the Linearized Model .. ... .. . 31
2.2.4 Controlling the Neck and Eye Positions ...... .. .... 34
2.2.5 Automatic Tuning from the System Residual Error. . 35
2.2.6 Estimating VR ... . .. . .. .. ..... . .... . .. .. . . .. .... 36
2.2.7 Estimating a .... ... ... .... .. .. ... .. .. .. ... ..... 36
2.2.8 Estimating I .... ... .. .. .... ..... ... .. ....... ... 37
X Contents
2.2.9 Simulation of the Combined Behavior . . . . . . . . . . . . . . 37
2.3 Detection of Visual Targets for 3D Visual Perception. . . . . . . 40
2.4 Computing the 3D Parameters of a Target. . . . . . . . . . . . . . . . 42
2.4.1 A Unique Framework to Integrate Depth
from Vergence, Motion and Zoom .... ..... ... ..... 43
2.4.2 Using Second-Order Focus Variations
to Compute Depth ................... ... .... .... 43
2.4.3 Multi-Model Concurrency in 3D Tracking .. ... ..... 45
2.4.4 Considering Further 3D Information. . . . . . . . . . . . . . . 45
2.5 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.5.1 Head Intrinsic Calibration. . . . . . . . . . . . . . . . . . . . . . . . 46
2.5.2 Looking at a 3D Point ........ .............. ... , . 48
2.5.3 Where to Look Next Experiment. . . . . . . . . . . . . . . . . . 48
2.5.4 Reconstruction of a Coarse 3D Map ...... ....... .. 49
2.5.5 Tracking of a 3D Target: Simulation Experiments ... 50
2.5.6 Tracking of a 3D Target: Real Object Experiments .. 51
2.5.7 Conclusion... ............... .. ..... ............ 52
3. Auto-Calibration of a Robotic Head . . . . . . . . . . . . . . . . . . . . . 55
3.1 Introduction...... ... ..... .... ......... ....... ........ 55
3.2 Reviewing the Problem of Visual Sensor Calibration. . . . . . . 57
3.3 Equations for the Tracking of a Stationary Point ..... .... . 58
3.4 Recovering the Parameters of the Trajectory. . . . . . . . . . . . . . 61
3.4.1 An Initial Estimate of the Coefficients .. .. ......... 61
3.4.2 Minimizing the Nonlinear Criterion . . . . . . . . . . . . . . . . 63
3.5 Computing Calibration Parameters . . . . . . . . . . . . . . . . . . . . . . 65
3.5.1 Equations for the Intrinsic Calibration Parameters. . . 66
3.5.2 Extrinsic Parameters Computation . . . . . . . . . . . . . . . . 68
3.5.3 Calibration Algorithm ..... .... ... .... ........... 69
3.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.6.1 How Stable are Calibration Parameters
When Zooming? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.6.2 Experiment 1: Parameter Estimation
with Synthetic Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.6.3 Experiment 2: Trajectory Identification
Using Real Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.6.4 Experiment 3: Parameters Estimation
Using Real Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.7 Conclusion ...... ... .. ........ ..... .. .. .... .... .. .... . 77
3.8 Comparison with the Case of Known Translations .. . . . . . . . 78
3.9 Application to the Case of a Binocular Head. . . . . . . . . . . . . . 79
3.10 Instantaneous Equations for Calibration . . . . . . . . . . . . . . . . . . 80
3.10.1 Reviewing the Definition of the Fundamental Matrix. 80
3.10.2 Characterizing the Essential Matrix
for Fixed Axis Rotations. . . . . . . . . . . . . . . . . . . . . . . . . 81
