Revisiting Normalized Cross-Correlation for Accurate Camera Pose Estimation and Accurate Real ...

271 Pages·2015·37.24 MB·English

Revisiting Normalized Cross-Correlation for Accurate Camera Pose
Estimation and Accurate Real-Time Multiple View Stereo
by
Eduardo de Brito Almeida
B. S., Universidade Federal do Ceará, 2004
Sc. M. in Engineering, Brown University, 2010
Sc. M. in Applied Mathematics, Brown University, 2010
Submitted in partial fulfillment of the requirements
for the Degree of Doctor of Philosophy in the
School of Engineering at Brown University
Providence, Rhode Island
May 2015
© Copyright 2015 by Eduardo de Brito Almeida
This dissertation by Eduardo de Brito Almeida is accepted in its present form by
the School of Engineering as satisfying the dissertation requirement
for the degree of Doctor of Philosophy.
Date
David B. Cooper, Advisor
School of Engineering
Recommended to the Graduate Council
Date
Joseph L. Mundy, Reader
Date
Pedro F. Felzenszwalb, Reader
Approved by the Graduate Council
Date
Peter M. Weber
Dean of the Graduate School
iii
Curriculum Vitae
Eduardo de Brito Almeida was born in Stony Brook, New York on July 12th, 1981. He graduated
from Federal University of Ceará in Fortaleza, Ceará, Brazil in December 2004 with a Bachelor of
Science degree in Electrical Engineering. In the Summer of 2006, Eduardo began his graduate studies
at Brown University in Providence, Rhode Island. Since then he has been at the Laboratory for
Engineering Man-Machine Systems (LEMS) at Brown University working as a research assistant
in the field of computer vision under the supervision of Prof. David B. Cooper. Eduardo earned
Master of Science degrees in Engineering and in Applied Mathematics from Brown University in
2010. He is a NASA Rhode Island Space Grant Fellow and a NASA Harriett G. Jenkins Pre-Doctoral
Fellow. In 2009, 2010 and 2012, Eduardo interned at NASA’s Jet Propulsion Laboratory (JPL), a
division of California Institute of Technology located in Pasadena, California, where he worked on
projects related to image-based 3-d reconstruction under the direction of Dr. Curtis W. Padgett.
He received a Space Act Award from NASA Inventions and Contributions Board in recognition of
his development of significant scientific contribution that has been determined to be of value in the
conduct of aeronautical activities of NASA. His research interests include computer vision, pattern
recognition, applied probability and statistics, change detection, image-based 3D reconstruction,
multiple view geometry, structure from motion and GPU acceleration.
iv
Acknowledgements
I owe many people a debt of gratitude for supporting me and making this dissertation possible. In
particular, I would like to express my sincere appreciation to my advisor Prof. David B. Cooper for
the essential guidance, advices and support he gave me during the time I was working on this thesis
and throughout all of my graduate studies. His great experience has always positively influenced
my professional career in countless aspects. He has a great sense of humor and he also has been a
wonderful friend, who, for instance, has introduced me to the excitement of sailing out of Boston
harbor.
I am deeply thankful to Joseph L. Mundy and Pedro F. Felzenszwalb for serving in my thesis
committee and providing many insightful comments and suggestions in this work. In addition, I am
also very honored to have interacted with Prof. Gabriel Taubin, Prof. Benjamin Kimia and other
professors at Brown University who have taught me and guided my academic work.
Particular thanks goes to professors Fátima Sombra, Basilis Gidas and Rui Seara whose encour-
agement inspired me to go after my dream of pursuing a doctoral degree in the United States.
This dissertation is a culmination of several years of research sponsored by Brown University and
its School of Engineering; by the Mobility and Robotics Section at the Jet Propulsion Laboratory
(JPL), California Institute of Technology; by the National Aeronautics and Space Administration
(NASA); by the NASA Harriett G. Jenkins Pre-doctoral Fellowship Program (JPFP); by the Rhode
Island Space Grant Consortium (RISG); and by the student programs from JPL Education Office.
I gratefully acknowledge my JPL mentor Dr. Curtis W. Padgett, the RISG director Dr. Peter H.
Schultz, and all JPFP program members. Their invaluable support in awarding me a total of two
NASA fellowships and three exciting internships have greatly enhanced my academic experience. I
v
have tremendously benefited by collaborating with Dr. Padgett and his group at JPL, where the
ideas for the foundations of this thesis were consolidated.
There are several friends who I would like to thank. Gilson Lima, John Raiti, Brandon Mayer,
Ricardo Fabbri, Kilho Son, Isabel Restrepo and David Borton for the amazing friendship and all the
countless advices. Ming-Ching, Nhon, Osman, Vishal, Fatih, Amir, Anil, Shubao, Maruthi, Daniel
Crispell, Ozge, DanielMoreno, Matthew, Krishna, Ibrahim andall other labmates fortheir sympathy
and for providing a pleasant work environment at LEMS.
I would like to give very special thanks to my father, my mother, my brother and my sister-in-law,
for their constant support and endless love that always encouraged me to reach my goals and pursue
my graduate school ambitions. This thesis is dedicated to them. I would also like to thank all
my relatives for always being so kind and supportive. Their presence is constantly felt despite the
physical distance. I am also grateful to Juliana, Carol, Vanessa, Kárida, Sílvia and Wallace Casaca
for the memorable companionship during my last years at Brown, and Fr. Bodah for the friendship.
Above all else, I acknowledge the blessings of God, to whom I owe everything.
vi
Contents
List of Tables xiii
List of Figures xv
1 Introduction 1
1.1 Goals and requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Contibutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Related work 13
2.1 Feature detection and matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Dense reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.1 Real-time 3-d reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Disambiguation of NCC correspondences . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.1 Disambiguating dense repeating features . . . . . . . . . . . . . . . . . . . . . 25
3 Review of basic algorithms 31
3.1 Rectification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 Weighted normalized cross-correlation . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2.1 Size relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2.2 Why is weighting important? . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2.3 Why is normalization important? . . . . . . . . . . . . . . . . . . . . . . . . . 37
vii
3.3 Delaunay triangulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 Camera estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.4.1 Bundle adjustment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.4.2 Reprojection error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4 Near Real-time Camera Pose Estimation 43
4.1 Matching overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.1.1 Displaying image matches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 Interest points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.3 Matching process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.3.1 Correlation peaks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.3.2 Weak corner removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.3.3 Collection of sets of match candidates . . . . . . . . . . . . . . . . . . . . . . 61
4.3.4 Initial low-resolution matching . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3.5 Native resolution matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.4 Disambiguation process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.4.2 Fundamental matrix estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.4.3 Ratio-inequalities test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.4.4 Highest scores ratio module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4.5 Epipolar constraint module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4.6 Epipolar sectors module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4.7 Multi-view disambiguation modules . . . . . . . . . . . . . . . . . . . . . . . 70
4.4.8 Indirect matches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.4.9 Affine transfer modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.4.10 Match topology module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.4.11 Epipolar transfer module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
viii
5 Planar grid matching 81
5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.1.1 Motivation for modeling planar surfaces . . . . . . . . . . . . . . . . . . . . . 87
5.2 Grid points detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.2.1 Finding grid seeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.2.2 Augmented clones set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.2.3 Lattice model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.2.4 Iterative outlier removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.2.5 Defining grid main directions . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.3 Grid lattice estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.3.1 Connecting vertices into line segments . . . . . . . . . . . . . . . . . . . . . . 102
5.3.2 Defining line segments neighborhood . . . . . . . . . . . . . . . . . . . . . . . 103
5.3.3 Merging line segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.3.4 Organizing lines into a 2-d lattice . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.4 Grid matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.4.1 Selection of grid points to match . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.4.2 Match grid points independently . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.4.3 Reduce ambiguities by matching grid lines . . . . . . . . . . . . . . . . . . . . 109
5.4.4 Topology-preserving incremental line matching . . . . . . . . . . . . . . . . . 116
5.5 Wide-baseline matching via lattice tracking . . . . . . . . . . . . . . . . . . . . . . . 121
5.5.1 Tracking lattice planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6 Real-time Depth Estimation 126
6.1 Algorithm description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.2 The ray grid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.3 Rectification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.4 Multiple view stereo matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.4.1 Similarity score along a ray . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
ix
6.4.2 Peak representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
6.4.3 Score mapping from 2-d to 3-d . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.4.4 Background rays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.4.5 Score fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.4.6 Depth estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.4.7 Peak refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.4.8 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.6 Volume far sides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.6.1 Far side of a voxel grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.6.2 Far side of a rectangular volume . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.6.3 Far side visibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
6.7 Surface reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
6.7.1 Energy minimization using graph cuts . . . . . . . . . . . . . . . . . . . . . . 145
6.7.2 Graph construction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.7.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7 GPU Accelerated Template Matching 151
7.1 General Purpose Graphics Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.1.1 Typical GPU Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.1.2 GPU memory model and cache . . . . . . . . . . . . . . . . . . . . . . . . . . 154
7.1.3 Parallel synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
7.1.4 Data-parallel instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
7.1.5 Memory latency and performance. . . . . . . . . . . . . . . . . . . . . . . . . 160
7.1.6 Active threads, warps and blocks.. . . . . . . . . . . . . . . . . . . . . . . . . 161
7.1.7 Register spills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.1.8 Parallel functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.1.9 Summary of performance requirements . . . . . . . . . . . . . . . . . . . . . . 162
7.2 Normalized cross-correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
x