GPU Accelerated Feature Algorithms for Mobile Devices

150 Pages·2014·2.71 MB·English

GPU Accelerated Feature Algorithms for
Mobile Devices

A thesis submitted to the Auckland University of Technology,
In fulfilment of the requirements for the degree of
Doctor of Philosophy (PhD)

Seth George Hall

School of Computing and Mathematical Sciences

March 2014

Primary Supervisor: Dr. Andrew Ensor
Secondary Supervisor: Dr. Roy Davies
Table of Contents

List of Figures ........................................................................................................................... 5
List of Tables ............................................................................................................................ 7
Attestation of Authorship ........................................................................................................ 8
Acknowledgements ................................................................................................................... 9
Abstract ................................................................................................................................... 10
Chapter 1 Introduction .......................................................................................................... 11
Chapter 2 Literature Review ................................................................................................ 22
2.1 : Augmented Reality ................................................................................................... 22
2.2 : Mobile Smartphone Platforms Overview ................................................................. 24
2.3 : Computer Vision: ..................................................................................................... 28
2.3.1 : Fiducial Markers ................................................................................................ 29
2.3.2 : Feature Detection ............................................................................................... 30
2.3.3 : Image Convolutions ........................................................................................... 31
2.3.4 : Canny Edge Detection ....................................................................................... 32
2.3.5 : FAST.................................................................................................................. 34
2.3.6 : SIFT ................................................................................................................... 35
2.3.7 : SURF ................................................................................................................. 36
2.3.8 : Lucas Kanade..................................................................................................... 36
2.4 : General Purpose Computing on the GPU: ............................................................... 38
2.4.1 : GPU programming and Shaders: ....................................................................... 38
2.4.2 GPU based feature descriptors ............................................................................. 43
2.5 : Cluster Analysis ....................................................................................................... 44
2.5.1 : DBSCAN ........................................................................................................... 45
2.5.2 : K-means Clustering ........................................................................................... 46
2.6 : Location Based Services .......................................................................................... 47
2 | P a ge
2.7 : Existing Computer Vision and MAR Applications: ................................................ 49
2.8 Significance for Mobile Augmented Reality .............................................................. 51
Chapter 3 GPU Based Canny Edge Detection ..................................................................... 53
3.1 : Image Analysis on Mobile Devices ......................................................................... 54
3.2 : Canny Shader Implementation ................................................................................. 57
3.2.1 : CPU side setup ................................................................................................... 58
3.2.2 : Gaussian Smoothing Steps ................................................................................ 59
3.2.3 : Sobel XY Steps .................................................................................................. 59
3.2.4 : Non-Maximal Suppression & Double Threshold Steps .................................... 59
3.2.5 : Weak and Strong Pixel Tests ............................................................................. 60
Chapter 4 Performance Comparison of Canny Edge Detection on Mobile Platforms ... 62
4.1 Mobile Performance Results ...................................................................................... 62
4.2 Results Discussion ...................................................................................................... 65
Chapter 5 GPU-based Feature point detection ................................................................... 67
5.1 Feature Detection and Description ............................................................................. 67
5.2 : GPU FAST implementation ..................................................................................... 69
5.3 : ColourFAST Feature Point Detection Implementation ............................................ 71
5.3.1 : CPU Side Setup and Android Camera Capture ................................................. 72
5.3.2 : Colour Conversion ............................................................................................. 73
5.3.3 : Smoothing .......................................................................................................... 74
5.3.4 : Half Bresenham and Feature Strength Calculation ........................................... 74
5.3.5 Feature Direction Calculation .............................................................................. 76
5.4 ColourFAST Results and Comparison to FAST ........................................................ 78
Chapter 6 GPU-based Feature Tracking ............................................................................. 83
6.1 : GPU-based Lucas Kanade implementation .............................................................. 84
6.2 : ColourFAST Feature Search implementation .......................................................... 85
6.2.1 Feature Point Difference Calculation .................................................................. 87
3 | P a ge
6.2.2 : Two-Step Hierarchical Approach ...................................................................... 87
6.2.3 Feature Blending .................................................................................................. 88
6.3 Results and Comparison with Lucas-Kanade ............................................................. 89
6.3.1 : Frame rate throughput tests ............................................................................... 89
6.3.2 : Tracking accuracy tests...................................................................................... 90
6.3.3 : Feature value repeatability tests......................................................................... 93
Chapter 7 Cluster Analysis & GPU-based Feature Discovery .......................................... 98
7.1 : GPU Feature Discovery Implementation ................................................................. 98
7.2 : Point Clustering ...................................................................................................... 101
7.3 : Results and Testing ................................................................................................ 105
7.4 : Future Work ........................................................................................................... 107
Chapter 8 GPU-based Object Recognition ........................................................................ 109
8.1 : Object Recognition and Feature Descriptions ........................................................ 109
8.2 : GPU-based Object Recognition version 1.............................................................. 110
8.3 : GPU Based Object Recognition Version 2 ............................................................ 114
8.4 : Results and testing .................................................................................................. 116
8.4.1 : Match Accuracy test ........................................................................................ 116
8.4.2 : Match Speed Test ............................................................................................ 122
8.5 : Future Work ........................................................................................................... 124
Chapter 9 Conclusion .......................................................................................................... 126
Appendix A: Canny Edge Detection Shaders .................................................................... 129
Appendix B: ColourFAST Feature Detection Shaders .................................................... 132
Appendix C: ColourFAST Feature Tracking Shaders .................................................... 134
Appendix D: Feature Discovery Shader ............................................................................ 137
Appendix E: ColourFAST Object Recognition Shaders .................................................. 138
References ............................................................................................................................. 140

4 | P a ge
List of Figures

Figure 1-1: Full ColourFAST GPU pipeline ........................................................................ 21
Figure 2-1: Milgrams reality-virtuality continuum [21] ...................................................... 23
Figure 2-2: 6DOF motion of a device in three-dimensional space ...................................... 23
Figure 2-3: Example types of fiducial markers .................................................................... 30
Figure 2-4: Example convolution kernels ............................................................................ 31
Figure 2-5: 16 pixel Bresenham circle around a possible FAST feature point .................... 35
Figure 2-6: Comparison of graphics pipelines ..................................................................... 40
Figure 2-7: An example of two clusters being split with the DBSCAN algorithm. ............ 46
Figure 3-1:GPU-based Canny edge detection pipeline. ..................................................... 58
Figure 3-2: Screenshots of Auckland skyline with GPU-based Canny edge detection ...... 61
Figure 5-1: GPU FAST pipeline implementation. ............................................................... 70
Figure 5-2: GPU ColourFAST feature detection pipeline.. ................................................. 72
Figure 5-3: YUV colour space using the NV21 format ....................................................... 73
Figure 5-4: Bresenham used by FAST and ColourFAST. ................................................... 75
Figure 5-5: Actual neighbourhood pixel contributions to ColourFAST. ............................. 76
Figure 5-6: Texture for feature direction vector calculations .............................................. 77
Figure 5-7: Outdoor scene with GPU FAST versus ColourFAST (lower) .......................... 80
Figure 5-8: FAST vs ColourFAST feature strengths at a 90 degree corner......................... 81
Figure 5-9: FAST vs ColourFAST feature strengths at soft 90 degree corner. ................... 81
Figure 5-10: FAST vs ColourFAST feature strengths at a 135 degree corner. .................... 82
Figure 5-11: FAST vs ColourFAST feature strengths at a 45 degree corner ....................... 82
Figure 5-12: Fast vs ColourFAST feature strengths at the end of a pixel thin line. ............ 82
Figure 6-1: Lucas-Kanade GPU pipeline ............................................................................. 85
Figure 6-2: GPU ColourFAST feature search pipeline. ....................................................... 87
Figure 6-3 Two pass feature description search ................................................................... 88
Figure 6-4: Boxplot of successful feature tracking time for up to 10 seconds motion. ...... 91
Figure 6-5: Pedestrian Tracking screenshot ......................................................................... 92
Figure 6-6: Graph showing the frequency of fluctuation for controlled environment ......... 95
Figure 6-7: Graph showing the frequency of fluctuation for uncontrolled environment ..... 96
Figure 7-1: GPU feature discovery pipeline. ....................................................................... 99
Figure 7-2: Six component Haar mask applied five times on the contour of the object. ... 100
5 | P a ge
Figure 7-3: GPU feature discovery screenshots ................................................................. 101
Figure 7-4: Average smoothed cluster movement of tracking windows ........................... 102
Figure 7-5: Screen shots of DBSCAN. .............................................................................. 104
Figure 7-6: Setup for tracking accuracy using clusters test. .............................................. 106
Figure 7-7: Graph shows tracking accuracy for clustered and non-clustered points. ...... 107
Figure 8-1: Textures generated on the CPU to describe objects to be matched. ............... 112
Figure 8-2: GPU-based object recognition pipeline version 1 ........................................... 113
Figure 8-3: GPU Object Recognition Pipeline version 2 ................................................... 114
Figure 8-4: Object recognition shader output textures ....................................................... 116
Figure 8-5: Object recognition testing screenshots ............................................................ 120
Figure 8-6: Logo dataset used for object matching ............................................................ 121
Figure 8-7: Object Recognition match accuracy for four consecutive tests. ..................... 122

6 | P a ge
List of Tables

Table 4-1: Render pass and reloading texture times with standard deviation...................... 64
Table 4-2: Frame rates and standard deviation for image capture for Canny ...................... 65
Table 5-1: Feature point throughput comparisons in frames per second (fps) .................... 79
Table 6-1: Feature tracking throughput comparisons .......................................................... 90
Table 6-2: Percentage of fluctuation of feature values for FAST vs ColourFAST. ............ 97
Table 7-1: Clustering times and standard deviation for 50 feature points ......................... 105
Table 7-2: Frame rates for tracking feature points with clustering vs non-clustering ....... 107
Table 8-1: Pipeline throughput with object recognition enabled and disabled .................. 123
Table 8-2: Average object recognition speeds for a number of feature points .................. 123

7 | P a ge
Attestation of Authorship

"I hereby declare that this submission is my own work and that, to the best of my
knowledge and belief, it contains no material previously published or written by another
person (except where explicitly defined in the acknowledgements), nor material which to a
substantial extent has been submitted to the award of any other degree or diploma of a
university or other institution of higher learning."

Signed: ______________________________

8 | P a ge
Acknowledgements

Firstly I would like to thank my primary supervisor Dr. Andrew Ensor for his total
dedication and willingness to help with the research and keeping me on track with
completing it. I really could not have finished it without his guidance, wisdom and
friendship; instead I would have leaped off the building after being driven mad from
OpenGL ES and GPU coding. I would also like to thank my secondary supervisor Dr. Roy
Davies and the rest of the AUT staff who helped me in any way. I would also like to
greatly acknowledge CoLab for granting me a generous financial scholarship so that I
could do this research without having a diet which mainly consists of two-minute noodles
and Homebrand peanut butter on budget brand toast. Also AUT’s School of Computing
and Mathematical Sciences for also providing me with fees scholarship, financial aid and
with the resources and time needed for the completion of this project. The Saeco Royal
Cappuccino Professional™ coffee making machine was put to very good use and I
probably cost AUT a fortune in coffee beans, sugar and milk alone. I would like to
dedicate this thesis to my family and friends, especially my father Danny and my amazing
Nana Carol whom have given me lots of help, advice and support over the duration of my
study. Finally thanks to my rabbit Oscar for giving me joy on those depressing days.

Good luck to the rest of the PhD students I shared the room with in the past and currently, I
enjoyed being the “go-to” guy for everything New Zealand related and I wish them all well
in their studies and lives.

World Peace!

9 | P a ge
Abstract

Mobile devices offer many new avenues for computer vision and in particular mobile
augmented reality applications that have not been feasible with desktop computers. The
motivation for this research is to improve mobile augmented reality applications so that
natural features, instead of fiducial markers or pure location knowledge, can be used as
anchor points for virtual mobile augmented reality models within the constraints imposed
by current mobile technologies. This research focuses on the feasibility of GPU-based
image analysis on current smart phone platforms. In particular it develops new GPU
accelerated natural feature algorithms for object detection and tracking techniques on
mobiles. The thesis introduces ColourFAST features which contain a compact feature
vector of colour change values and an orientation for each feature point. The feature
algorithms presented in this thesis process information in “real time”, with the objective on
high data throughputs, whilst still maintaining suitable accuracy and correctness. It
compares these new algorithms with well-known existing techniques as well as against
their modified GPU-based equivalents. The research also develops a new GPU-based
feature discovery algorithm for finding more feature points on an object, forming a cluster,
which can be collectively used to track the object and improve tracking accuracy. It looks
at clustering algorithms for tracking multiple objects and implements an elementary GPU-
based object recognition algorithm using the generated ColourFAST feature data.

10 | P a ge