Vision-based Autonomous Navigation and Active Sensing with Micro Aerial Vehicles

130 Pages·2017·4.99 MB·English

Vision-based Autonomous Navigation and
Active Sensing with Micro Aerial Vehicles
by
Rui Huang
B.Eng., Hong Kong Polytechnic University, 2011
Dissertation Submitted in Partial Fulfillment of the
Requirements for the Degree of
Doctor of Philosophy
in the
School of Computing Science
Faculty of Applied Sciences
(cid:13)c Rui Huang 2017
SIMON FRASER UNIVERSITY
Summer 2017
All rights reserved.
However, in accordance with the Copyright Act of Canada, this work may be
reproduced without authorization under the conditions for “Fair Dealing.”
Therefore, limited reproduction of this work for the purposes of private study,
research, education, satire, parody, criticism, review and news reporting is likely
to be in accordance with the law, particularly if cited appropriately.
Approval
Name: Rui Huang
Degree: Doctor of Philosophy (Computing Science)
Title: Vision-based Autonomous Navigation and Active
Sensing with Micro Aerial Vehicles
Examining Committee: Chair: Dr. Mark Drew
Professor
Dr. Ping Tan
Senior Supervisor
Associate Professor
Dr. Richard Vaughan
Supervisor
Associate Professor
Dr. Greg Mori
Internal Examiner
Professor
Dr. David Hsu
External Examiner
Professor
Department of Computer Science
National University of Singapore
Date Defended: August 21, 2017
ii
Abstract
Microaerialvehicles(MAVs)equippedwithcamerasprovideanewperspectiveontheworld.
As MAVs have found important roles in industrial and recreational applications, they are
actively studied in the research community. The main focus has been the autonomy level.
In other words, the MAV should be able to perform certain tasks autonomously to free
human from laborious and risky work. At the basis of an autonomous system, there is
the problem of autonomous navigation. In outdoor spaces, MAVs usually use GPS signals
for self-localization. In indoor GPS-denied environments, autonomous navigation of MAVs
is still an open research question. At the high level of an autonomous system, there is
the problem of active sensing. An autonomous platform needs to actively optimize its
navigation based on its current own state and the explored environment. The constraints of
vision sensors and complex environments pose challenges to researchers. In this thesis, we
propose our solutions to these two problems: vision based navigation and active sensing.
In the first half of the thesis, we propose solutions to vision based navigation on two plat-
forms. First, we present an ultra-light and -small MAV platform which is able to perform
autonomous navigation in an unknown indoor environment. The hardware design and a
visual odometry algorithm is introduced. This is the first 50 g quadrotor performing vision
based autonomous navigation. Secondly, we aim at a consumer grade MAV. The equipped
forward-looking camera is used as the major sensor while we make use of the measurements
from the inertial sensors as well. Accurate path following is achieved by only the built-in
sensors of the platform and a ground control station.
In the second half, we address the active sensing problem in two interesting applications.
Wefirstinvestigatetheactivetargetsensingandfollowingusingamulti-robotcollaborative
system. The only sensors are cameras and the constraints of vision algorithms are taken
into consideration while we design the controller for the robots. Lastly, we demonstrate an
active image data acquisition system in the image based mapping application. The camera
placement is optimized in the loop and online feedback is provided for the sensor planning.
Weshowthefirstfullyautonomousactiveimagebasedmodelingsysteminsimulated,indoor
and outdoor environments.
iii
Keywords: Unmmanned aerial vehicle; Micro aerial vehicle; Vision based Navigation;
Visual Odometry; Image-based Modeling; Active Vision; Next Best Views
iv
Dedication
To my teachers, mentors, friends and family.
v
Acknowledgements
First of all, I would like to thank my supervisor Prof. Ping Tan for his continuous guidance,
support and patience. He teaches me not only the knowledge but also a positive attitude to
workandresearch. IfeelblessedthatI’velearnedalotfromPingandsomegreatcolleagues
including Danping Zou, Nianjuan Jiang, Zhenglong Zhou, Shuaicheng Liu, Zhe Wu, Yinda
Zhang and Zhaopeng Cui.
I would like to express my gratitude to Prof. Richard Vaughan and his autonomy lab
at Simon Fraser University. Richard and his team have given me a lot of help, support and
suggestions. Jacob Perron, Jack Thomas, Lingkang Zhang and me shared wonderful time
on a collaborative project. Mani Monajjemi, Shokoofeh Pourmehr and Seyed Abbas Sadat
have given me lots of advices and I’ve learned a lot from them.
I am very grateful to Prof. Danping Zou and his UAV team at Shanghai Jiaotong
University. Without their support and company, I could not be able to finish this thesis.
I want to thank Zhantao Deng, Rongzhi Wang, Xushen Han, Zheng Gong, Qiang Zhou,
RuihangMiaoandGuiyuLiu. Thefreezingafternoonswespenttogetherontheexperiments
have become the precious memory to me.
I am thankful for the help and support from Prof.Ben.M.Chen and his unmanned sys-
temsresearchgroupatNationalUniversityofSingapore. IwasluckytoworkatProf.Chen’s
team and share all the good memories with them. I would like to thank Kun Li, KangLi
Wang, Yijie Ke, Kevin, Limiao Bai, Fei Wang, Jinqiang Cui, Xiangxu Dong for this won-
derful experience.
I would like to thank my friend Guang-tong Zhou, Shuyang Sun, Rui Ma, Xi Ao,
Chengyang Zhu, Renjiao Yi, Zhaopeng Cui for their help and company at my tough time.
Thanks to my friends at Vancouver for sharing their friendship with me.
Lastly, I want to give my greatest gratitude to my family, my mother, father and
grandma. They are always the reason I stand up and carry on.
vi
Table of Contents
Approval ii
Abstract iii
Dedication v
Acknowledgements vi
Table of Contents vii
List of Tables viii
List of Figures ix
1 Introduction 1
1.1 Challenges and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Background 8
2.1 Vision based Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.1 Filter-based vs. Keyframe-based Approaches . . . . . . . . . . . . . 9
2.1.2 Feature-based vs. Direct Approaches . . . . . . . . . . . . . . . . . . 9
2.2 Vision based Autonomous Navigation with MAVs . . . . . . . . . . . . . . . 10
2.3 Active Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.1 Active Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.2 Image based Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.3 Active Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3 Visual Odometry on an Ultralight Quadrotor 15
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3 Platform Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3.1 Camera System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3.2 Ultrasonic Sonar Sensor . . . . . . . . . . . . . . . . . . . . . . . . . 18
vii
3.3.3 PCTx Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3.4 Analog-Digital (A/D) converter . . . . . . . . . . . . . . . . . . . . . 20
3.4 Motion Estimation Using Optical Flow . . . . . . . . . . . . . . . . . . . . . 21
3.5 Feature Detection and Homography Estimation . . . . . . . . . . . . . . . . 22
3.6 Self Motion Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.7 Linear Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.8 Control System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.9 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4 Vision-Based Navigation on a Toy Quadrotor 30
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.3 Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3.2 Platform Modeling and Controller Design . . . . . . . . . . . . . . . 33
4.4 Monocular SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.4.1 Map initialization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.4.2 Feature Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.4.3 Feature Propagation and Camera Pose Estimation . . . . . . . . . . 38
4.4.4 Keyframe Insertion and Map Generation . . . . . . . . . . . . . . . . 39
4.4.5 Re-localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.5 Sensor Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.5.1 Our EKF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.5.2 Filter Update Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.6.1 SLAM system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.6.2 EKF based Sensor Fusion and Scale Estimation . . . . . . . . . . . . 47
4.6.3 System Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5 Target Following with Multi-Robot Collaborative Visual SLAM 51
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.2.1 Localization using Multiple Cameras . . . . . . . . . . . . . . . . . . 52
5.2.2 Target Following . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.3 System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.3.1 CoSLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.3.2 Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
viii
5.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.4.1 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.4.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6 Active Image-based Modeling with a Toy Drone 64
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2.1 Active 3D Modeling System . . . . . . . . . . . . . . . . . . . . . . . 65
6.2.2 Multiple-View Stereo (MVS) . . . . . . . . . . . . . . . . . . . . . . 66
6.2.3 Next-Best-View (NBV) Planning . . . . . . . . . . . . . . . . . . . . 66
6.3 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.4 Fast 3D Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.4.1 Sparse Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.4.2 Linear Dense MVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.5 Active Image Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.5.1 Coverage Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.5.2 NBV Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.6.1 Simulated Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.6.2 Indoor Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.6.3 Outdoor Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.6.4 Running Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.6.5 Quantitative Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.7 Geometry-based NBV Selection . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7 Conclusions 96
7.1 Contributions and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Bibliography 100
Appendix A Homography Given the Plane 111
Appendix B Depth Interpolation 112
ix
List of Tables
Table 3.1 Specifications of video transceiver shown in Figure 3.3b . . . . . . . . 19
Table 3.2 Specifications of the camera system. . . . . . . . . . . . . . . . . . . . 20
Table 4.1 Parameters of our PID controllers . . . . . . . . . . . . . . . . . . . . 35
Table 4.2 Path following evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 49
Table 5.1 Rules for adjusting robot velocity. . . . . . . . . . . . . . . . . . . . . 60
Table 6.1 Running time of all examples in detail. . . . . . . . . . . . . . . . . . 86
Table 6.2 Comparison of model accuracy between different plans. . . . . . . . . 89
Table 6.3 Comparison of model completeness between different plans. . . . . . . 89
x