An Intelligent Portable Aerial Surveillance System

87 Pages·2013·4.81 MB·English

An Intelligent Portable Aerial Surveillance System:
Modeling and Image Stitching
by
Ruixiang Du
A Thesis
Submitted to the Faculty
of the
WORCESTER POLYTECHNIC INSTITUTE
in partial fulfillment of the requirements for the
Degree of Master of Science
in
Robotics Engineering
by
MAY 2013
APPROVED:
Professor Taskin Padir, Advisor
Professor Michael A. Gennert, Committee Member
Professor Xinming Huang, Committee Member
Abstract
Unmanned Aerial Vehicles (UAVs) have been widely used in modern warfare for surveil-
lance, reconnaissance and even attack missions. They can provide valuable battlefield in-
formation and accomplish dangerous tasks with minimal risk of loss of lives and personal
injuries. However, existingUAVsystemsarefarfromperfecttomeetallpossiblesituations.
One of the most notable situations is the support for individual troops. Besides the incapa-
bility to always provide images in desired resolution, currently available systems are either
too expensive for large-scale deployment or too heavy and complex for a single solder. In-
telligent Portable Aerial Surveillance System (IPASS), sponsored by the Air Force Research
Laboratory(AFRL),isaimedatdevelopingalow-cost,light-weightunmannedaerialvehicle
that can provide sufficient battlefield intelligence for individual troops. The main contri-
butions of this thesis are two-fold (1) the development and verification of a model-based
flight simulation for the aircraft, (2) comparison of image stitching techniques to provide
a comprehensive aerial surveillance information from multiple vision. To assist with the
design and control of the aircraft, dynamical models are established at different complexity
levels. Simulations with these models are implemented in Matlab to study the dynamical
characteristics of the aircraft. Aerial images acquired from the three onboard cameras are
processed after getting the flying platform built. How a particular image is formed from
a camera and the general pipeline of the feature-based image stitching method are first
introduced in the thesis. To better satisfy the needs of this application, a homography-
based stitching method is studied. This method can greatly reduce computation time with
very little compromise in the quality of the panorama, which makes real-time video display
of the surroundings on the ground station possible. By implementing both of the meth-
ods for image stitching using OpenCV, a quantitative comparison in the performance is
accomplished.
iii
Acknowledgements
I would like to express my deepest appreciation to all those who provided me the pos-
sibility to complete this thesis.
To start I would like to thank my advisor Professor Taskin Padir. Without his great
patience, valuable guidance and persistent help, I would never have been able to finish this
project and learn so much in this process. I consider it an honor to work with him in his
Robotics and Intelligent Vehicles Research Laboratory.
I would like to thank my committee members, Professor Michael A. Gennert and Pro-
fessor Xinming Huang, for their insightful comments and suggestions.
I would also like to thank the IPASS team, Adam Blumenau, Alec Ishak, Brett Limone,
Zachary Mintz, Corey Russell and Adrian Sudol. Their works for the IPASS project made
itpossibleformetotestandverifypartofmyresearch. Ourdiscussionandideaexchanging
inspired me a lot to delve further into this project.
Further more, I would like to thank my friends, Gang Li and Xianchao Long. They
provided kindly help when I got in trouble during my works.
Finally, I would like to thank my parents, Xintian Du and Wenxia Yu. Without their
support, I would never have a chance to study at Worcester Polytechnic Institute in the
United States. Their selfless love and encouragement always keeps me moving forward in
my life and career.
iv
Contents
List of Figures vi
List of Tables viii
1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 WASP III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 RQ-11 Raven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3 RQ-16 T-Hawk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Objectives of the IPASS Project . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.1 Needs Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.2 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 System Design of IPASS 11
2.1 Mechanical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Electronic Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Software Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.1 Embedded Software . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.2 Software for Ground Station . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Final Product of the IPASS Project . . . . . . . . . . . . . . . . . . . . . . 17
2.5 Chapter Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 Modeling and Simulation 19
3.1 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.1.1 Basic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.1.2 2D Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.1.3 3D Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2 Matlab Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2.1 Simulation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2.2 3D Animation for the Simulation . . . . . . . . . . . . . . . . . . . . 25
v
3.2.3 Simulation with the Basic Model . . . . . . . . . . . . . . . . . . . . 27
3.2.4 Simulation with the 2D Model . . . . . . . . . . . . . . . . . . . . . 28
3.2.5 Simulation with the 3D Model . . . . . . . . . . . . . . . . . . . . . 30
3.3 Chapter Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4 Image Stitching 35
4.1 Image Formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.2 Motion Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.3 Feature-based Image Stitching . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3.1 Feature-based Registration . . . . . . . . . . . . . . . . . . . . . . . 40
4.3.2 Compositing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.4 Homography Based Stitching . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.4.1 Homography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.4.2 Plane induced parallax . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.4.3 Infinite Homography . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.4.4 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.5 Comparison of the Two Image Stitching Methods . . . . . . . . . . . . . . . 55
5 Conclusion and Future Work 61
Bibliography 63
A Appendix A: Matlab Simulation Code for the 3D Model 65
A.1 simulation.m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
A.2 ipass.m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
B Appendix B: C++ Code for Homography-based Image Stitching Method 74
B.1 homostitch.h . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
B.2 homostitch.cpp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
vi
List of Figures
1.1 WASP III [1] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 RQ-11 Raven [2] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 RQ16 T-Hawk [3] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1 Two Prototypes of the Chassis . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Final Design of the Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Electronic Box and Cameras. . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 System Diagram of the System . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5 Ground Station Software Developed by IPASS Team . . . . . . . . . . . . . 16
2.6 The Whole Intelligent Portable Aerial Surveillance System . . . . . . . . . 17
2.7 Image Stitching Result by IPASS team . . . . . . . . . . . . . . . . . . . . . 18
3.1 Force and Moment Analysis of the Basic Model . . . . . . . . . . . . . . . . 19
3.2 Force and Moment Analysis of the 2D Model . . . . . . . . . . . . . . . . . 21
3.3 Coordinate System of the 3D Model . . . . . . . . . . . . . . . . . . . . . . 22
3.4 Mechanical Model of the Aircraft in Sketchup . . . . . . . . . . . . . . . . . 26
3.5 Virtual Reality Scene in vrbuild2 . . . . . . . . . . . . . . . . . . . . . . . . 26
3.6 PD Control: desired value z=10, psi=0.785 (45 degrees) . . . . . . . . . . . 27
3.7 Simulation with Basic Model . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.8 Simulation with 2D Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.9 Flight Trajectory with Different values of the Offset e . . . . . . . . . . . . 29
3.10 Simulation of the Aircraft with Only Rotation about Center Axis . . . . . . 30
3.11 Simulation of the Aircraft with Only Offset of the Mass Center . . . . . . . 31
3.12 Simulation of the Aircraft when τ = 0.15 and e=0.01m . . . . . . . . . . . . 32
3.13 Simulation of the Aircraft when τ = 0.35 and e=0.01m . . . . . . . . . . . . 33
3.14 Flight Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.1 Pinhole Camera Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.2 Four Coordinates in the Image Formation . . . . . . . . . . . . . . . . . . . 37
4.3 Basic set of 2D planar transformations [4] . . . . . . . . . . . . . . . . . . . 38
4.4 Hierarchy of 2D coordinate transformations [4] . . . . . . . . . . . . . . . . 39
4.5 Pipeline of Feature-based Stitching Method from OpenCV Library . . . . . 40
4.6 Test Frame with Two Microsoft Webcams . . . . . . . . . . . . . . . . . . . 41
vii
4.7 Original Images Taken by the Two Cameras . . . . . . . . . . . . . . . . . . 42
4.8 SURF Feature Points in the Two Images . . . . . . . . . . . . . . . . . . . . 43
4.9 Matching of SURF Feature Points . . . . . . . . . . . . . . . . . . . . . . . 44
4.10 Stitching Result with a Planar Compositing Surface . . . . . . . . . . . . . 45
4.11 Stitching Result with a Spherical Compositing Surface . . . . . . . . . . . . 46
4.12 Homography between Two Image Planes [5] . . . . . . . . . . . . . . . . . . 47
4.13 Transformations of Three Cameras with Homography . . . . . . . . . . . . 48
4.14 Chessboard with Corners Marked by Color Circles and Lines . . . . . . . . 49
4.15 Image Alignment using Homography (without misalignment) . . . . . . . . 50
4.16 Image Alignment using Homography (with misalignment) . . . . . . . . . . 51
4.17 Distortion of a Camera: (a) No Distortion (b)Barrel Distortion (c) Pincush-
ion Distortion [6] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.18 Distortion Correction: (Left) distorted image (Right) undistorted image . . 54
4.19 Image Alignment with Infinite Homography . . . . . . . . . . . . . . . . . . 56
4.20 Simplified Pipelines of Feature-based Method and Homography-based Method 57
4.21 GUI of the Application for Image Stitching . . . . . . . . . . . . . . . . . . 58
4.22 ComparisonofPanoramas-top: feature-basedmethod,bottom: homography-
based method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
viii
List of Tables
1.1 General Characteristics of WASP III [1] . . . . . . . . . . . . . . . . . . . . 2
1.2 General Characteristics of RQ-11 Raven [2] . . . . . . . . . . . . . . . . . . 4
1.3 General Characteristics of RQ-16 T-Hawk (adapted from [3], [7] and [8]) . . 4
2.1 List of Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.1 Update Frequency of the Two Stitching Methods . . . . . . . . . . . . . . . 58
1
Chapter 1
Introduction
1.1 Background
UnmannedAerialVehicles(UAVs)havebeenwidelyusedinavarietyofareasinmodern
world since their first appearance in early nineties. With the help of the achievements in
technology fields such as automatic control, autonomous navigation and communication,
now UAVs can accomplish various kinds of tasks that once required a human pilot to do
onboard. Since the use of UAVs eliminates the possibilities of loss of lives and personal
injuries, UAVs are especially suitable for tasks that need to be performed in dangerous
environments.
Currently, UAVs are deployed predominantly for military applications. They are mostly
usedforreconnaissancetoprovidebattlefieldintelligenceandtheattacktoobjectsinspecial
areas. This thesis focuses on the UAVs that are designed for reconnaissance missions. A
large number of models for this purpose have been developed so far. These models are
classified by different standards, such as scale and flight time. Since the research has a
concentration on small scale UAVs for reconnaissance, several such existing models are
briefly introduced below.
2
Figure 1.1: WASP III [1]
Primary function Reconnaissance and surveillance with low-altitude operation
Contractor Aerovironment, Inc. (Increment III)
Power plant Electric motor, rechargeable lithium ion batteries
Wingspan 28.5 inches (72.3 cm)
Length 10 inches (25.4 cm)
Weight (air vehicle) 1 pound (453 grams)
Weight (total system) 14.4 pounds (6.53 kilograms)
Speed 20 - 40 mph
Operating altitude From 150 to 500+ feet above ground level (to 152+ meters)
Altitude 1,000 feet
System Cost approximately $49,000 (2006 dollars)
Payload High resolution, day/night camera
Table 1.1: General Characteristics of WASP III [1]