Object Detection and Tracking Based on Optical Flow in Unmanned Aerial Vehicles

192 Pages·2015·41.26 MB·English

by  Håkon Hagen Helgesen

Object Detection and Tracking Based on
Optical Flow in Unmanned Aerial Vehicles
Håkon Hagen Helgesen
Master of Science in Cybernetics and Robotics
Submission date: June 2015
Supervisor: Thor Inge Fossen, ITK
Co-supervisor: Tor Arne Johansen, ITK
Norwegian University of Science and Technology
Department of Engineering Cybernetics

Problem Formulation
Thesis Description:
A UAV with daylight camera can be used to observe interesting objects, such
as ships, boats, persons, vehicles, marine mammals, ice and environmental spills.
The purpose of the thesis is to investigate the effectiveness of optical flow in order
to 1) detect moving objects (moving object segmentation), 2) track the detected
objects, and 3) estimate relative velocity between the UAV and the background
(sea or terrain).
The following items should be considered:
1. Literature survey, considering general methods for detection and tracking of
objects and features from a moving platform with ego-motion using opti-
cal flow, and related applications such as automotive (pedestrians and other
vehicles), aerial ocean surveillance, UAV velocity estimation, etc.
2. Develop algorithms for computing optical flow assuming no moving objects.
UsethealgorithmtoestimatetherelativevelocitybetweentheUAVandthe
background. Use the calculated velocity in a nonlinear observer with IMU
and GNSS in order to estimate the position, velocity, attitude and gyro bias
of the UAV.
3. Developanalgorithmforcomputingopticalflowandsegmentationofmoving
objects assuming one moving object in the image frame.
4. Basedon2and3,developatrackingsystemfortrackinganarbitrarynumber
of objects (from zero to many) from a sequence of images.
5. Use the position of the UAV to compute georeferenced position and velocity
over ground estimates for tracked objects.
6. Test the algorithms through computer simulations.
Start date: 2015-01-12
Due date: 2015-06-08
Thesis performed at: Department of Engineering Cybernetics, NTNU
Supervisor: Professor Thor I. Fossen, Dept. of Eng. Cybernetics, NTNU

Abstract
InrecentyearstheUnmannedAerialVehicle(UAV)communityhasdiscoveredthe
enormousamountofinformationthatcanbeextractedfromavideocamera. Itcan
be used for collision avoidance, navigation, velocity estimation, terrain mapping,
object detection and object tracking in addition to many other applications. This
thesislooksintoseveraldifferentwaysavideocamerainthepayloadofafixed-wing
UAV can be utilized. The first part investigates a way to calculate the body-fixed
velocity of the UAV with the images captured by the camera. It can be achieved
with optical flow and measurements of the roll angle, pitch angle and the altitude
of the UAV. The calculated body-fixed velocity can be used as a measurement
in the navigation system. This thesis looks into a navigation system based on
a nonlinear observer that estimates attitude, position, velocity and gyro bias. In
ordertoestimatethesestatesthenonlinearobserverutilizesmeasurementsfroman
Inertial Measurement Unit (IMU), a Global Navigation Satellite System (GNSS)
receiver and a video camera.
The second part looks into moving object detection and tracking. An algorithm
for detection of moving objects has been developed. It utilizes the navigation
states of the UAV and optical flow in order to extract the moving objects from
the images. Furthermore a tracking system based on the moving object detection
algorithm has been proposed. The tracking system consists of the moving object
detection algorithm, a classifier that describes each object and a discrete Kalman
filter that estimates the motion of the object. The objects are tracked in the
image plane and the estimates are transformed to the North-East-Down (NED)
coordinate system.
A fixed-wing UAV experiment has been carried out to gather images captured
from an airborne UAV and collect data from an IMU and a GPS receiver. The
navigation system has been evaluated offline by computer simulations based on
the data collected at the experiment. The tracking system and the moving object
detection algorithm have also been evaluated in computer simulations. Promising
and accurate results were shown for both the navigation system and the proposed
tracking system.
Keywords: Unmanned Aerial Vehicle, Vision Based Navigation, Vision-Aided
Nonlinear Observer, Discrete Kalman Filter, Navigation System, Sensor Fusion,
Computer Vision, Optical Flow, Object Detection, Object Tracking
iii
iv
Sammendrag
De siste ti ˚arene har bruk av kamera i ubemannede fly vært et sentralt forskn-
ingstema. Ved˚a montere et kamera i et ubemannet fly kan man benytte det for
˚a unng˚a kollisjoner, navigere, estimere hastighet, lage terrengprofiler, detektere
objekter og m˚alfølge ojekter blant annet. Denne avhandlingen ser p˚a forskjellige
anvendelser hvor et kamera montert i nyttelasten til et ubemmanet fly er nyttig.
Denførstedelenundersøkerogpresentererenm˚atebildenefrakameraetkanbrukes
til˚aberegnehastighetentildetubemannendeflyet. Dettekanrealiseresmedoptisk
flyt,m˚alingavorienteringentilflyetogenhøydem˚aling. Denberegnedehastigheten
kanbrukesinavigasjonssystemettildetubemannedeflyet. Denneavhandlingenser
nærmerep˚aetnavigasjonsssytemsomerrealisertmedenulineærobserversomes-
timerer orientering, posisjon, hastighet og gyroskopbias. Den ulineære observeren
er avhengig av m˚alinger fra en IMU, et globalt posisjonerings system (GPS) og
kameraet.
Den andre delen undersøker hvordan objekter i bevegelse kan detekteres og følges
ved hjelp av kameraet. En algoritme som finner objekter i bevegelse blir foresl˚att.
Denne algoritmen benytter estimatene fra navigasjonssystemet og optisk flyt for
˚afinneobjekteribevegelseibildene. Itilleggeretm˚alfølgingssystemsombenytter
denne algoritmen foresl˚att. Dette systemet best˚ar av algoritmen som finner objek-
ter i bevegelse, en beskrivelse av alle objekter som detekteres slik at de kan skilles
fra hverandre og et diskret Kalman filter som estimerer bevegelsen til objektene.
Objektene m˚alfølges i bildeplanet og estimatene transformeres til Nord-Øst-Ned
(NED) koordinatsystemet.
En testflyvning med et ubemannet fly har blitt gjennomført for˚a samle bilder og
data fra IMU og GPS. Navigasjonssystemet har blitt testet og evaluert gjennom
simuleringer. M˚alfølgingssystemetogalgoritmenfor˚afinneobjekteribevegelsehar
ogs˚a blitt testet og evaluert gjennom simuleringer. Simuleringene viste lovende og
nøyaktige resultater for b˚ade navigasjonssystemet og m˚alfølgingssystemet.
Nøkkelord: UbemannetFly,KamerabasertNavigasjon,Kamera-AssistertUlineær
Observer, Diskret Kalman Filter, Navigasjonssystem, Sensorfusjon, Bildebehan-
dling, Optisk Flyt, Objekt Deteksjon, M˚alfølging
v
vi
Preface
The work described in this thesis is carried out in the Department of Engineering
Cybernetics, at the Norwegian University of Science and Technology, the spring of
2015. ItissubmittedinpartialfulfilmentoftherequirementsforthedegreeofMSc.
at the Norwegian University of Science and Technology. In addition to this thesis
a part of the work has been to write two papers in cooperation with fellow MSc.
student Jesper Hosen, PhD. candidate Lorenzo Fusini, Professor Thor I. Fossen
and Professor Tor A. Johansen. The first paper has been submitted and accepted
for the International Conference on Unmanned Systems 2015 (ICUAS’15). The
second paper has been submitted to the 2016 American Institute of Aeronautics
and Astronautics Science and Technology Forum and Exposition (AIAA SciTech
2016) and acceptance or rejection is expected late in August 2015.
I would like to thank Professor Thor Inge Fossen who trusted me to write this
thesis and giving me useful feedback throughout this spring. I would also like to
thankProfessorTorArneJohansenwhocreatedtheproblemformulationandhave
used a lot of time on proofreading the papers. Furthermore I would like to thank
PhD.candidateLorenzoFusiniforthecooperationandsupervision. Ihaveworked
closely with fellow MSc. student Jesper Hosen this year and he deserves a lot of
credit for a great collaboration. He has given me a lot of inspirational ideas and
useful feedback.
Finally, I would like to thank my parents, brother, sister and girlfriend Hilde for
their support.
Trondheim, June 2015
H˚akon Hagen Helgesen
vii
viii