Appearance-Based Gesture Recognition

124 Pages·2005·6.83 MB·English

by  Philippe Dreuw

Diplomarbeit im Fach Informatik
Appearance-Based Gesture Recognition
Rheinisch-Westfa¨lische Technische Hochschule Aachen
Lehrstuhl fu¨r Informatik VI
Prof. Dr.-Ing. H. Ney
vorgelegt von:
Philippe Dreuw
Matrikelnummer 217722
E-Mail [email protected]
Gutachter:
Prof. Dr.-Ing. H. Ney
Prof. Dr. T. Seidl
Betreuer:
Dipl.-Inform. D. Keysers
Januar 2005

Erkl¨arung
Hiermit versichere ich, dass ich die vorliegende Diplomarbeit selbstst¨andig verfasst
und keine anderen als die angegebenen Hilfsmittel verwendet habe. Alle Textauszu¨ge
und Grafiken, die sinngem¨aß oder w¨ortlich aus ver¨offentlichten Schriften entnommen
wurden, sind durch Referenzen gekennzeichnet.
Aachen, im Januar 2005
Philippe Dreuw
iii

Abstract
This diploma thesis investigates the use of appearance-based features for the recogni-
tion of gestures using video input. Previously, work in the field of gesture recognition
usually first segmented parts of the input images — for example the hand — and
then used features calculated from this segmented input. Results in the field of object
recognition in images suggest that this intermediate segmentation step is not neces-
sary and we can instead employ features directly obtained from the input images,
so-called appearance-based features. In this work, we show that using these features
andappropriatemodelsofimagevariability, wecanobtainexcellentresultsforgesture
recognitiontasks. Verygoodresultscanbeobtainedusingadownscaledimageofeach
video frame and tangent distance as a model of image variability. Also a new dynamic
tracking algorithm is introduced which makes its tracking decisions at the end of a
video sequence using the information of all frames. This tracking method allows for
tracking under very noisy circumstances. Finally, a new database with the German
fingerspelling alphabet was recorded which will be freely available for further research.
v

Acknowledgment
I would like to thank Prof. Dr.Ing. Hermann Ney for the interesting possibilities at the
Chair of Computer Science VI of the RWTH Aachen University of Technology where
I have been a member since January 2003.
I would also like to thank Prof. Dr. Thomas Seidl, who kindly accepted to co-
supervise the work.
Specially, I would like to thank Daniel Keysers, Thomas Deselaers and Morteza
Zahediforthehelpfulsuggestions,discussionsandassistancesthisworkreceived. Their
ideas and suggestions were a great help.
I would like to thank Jan Bungeroth and all volunteers for the participation in
creating the i6-Gesture database.
Also, I would like to thank the other members of the image recognition group at the
Chair of Computer Science VI – Andre Hegerath, Ilja Bezrukov, Helga Velroyen and
David Rybach – for many discussions, helpful tips and comments or proof reading the
manuscript. Also I would like to thank Alexander Zimmerman for many LATEX-tips.
Ofcoursespecialthanksgotomyparents. Theygavemethepossibilitiesdoingthese
studies. Last but not least I would like to thank my girlfriend Cindy for supporting
me while doing this work and who mostly saw me only late at night.
vii

Contents
1 Introduction 1
1.1 Gestures and Sign Language . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Stokoe Notation System . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 SignWriting Notation System . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 The HamNoSys Notation System . . . . . . . . . . . . . . . . . . . . . 5
2 State of the Art in Gesture Recognition 9
2.1 Related Work in the Context of Sign Language . . . . . . . . . . . . . 9
2.2 Related Work in the Context of Human Computer Interaction . . . . . 10
3 Features for Appearance-Based Gesture Recognition 13
3.1 Image Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1.1 Original Images . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.2 Difference Images . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.3 Skin Color Images . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1.4 Energy and History Images . . . . . . . . . . . . . . . . . . . . 20
3.2 Centroid Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4 Hidden Markov Models 25
4.1 The Theory of Hidden Markov Models . . . . . . . . . . . . . . . . . . 25
4.2 Distance Measures for the Emission Probabilities . . . . . . . . . . . . 30
4.2.1 Minkowski Distances . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2.2 Tangent Distance . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.2.3 Image Distortion Model . . . . . . . . . . . . . . . . . . . . . . 34
4.3 Training and Classification . . . . . . . . . . . . . . . . . . . . . . . . 36
5 Tracking 39
5.1 Bounding-Box Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.2 Meanshift/Camshift Tracking . . . . . . . . . . . . . . . . . . . . . . . 40
5.3 Dynamic Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.3.1 Dynamic Tracking with Fixed Size . . . . . . . . . . . . . . . . 41
5.3.2 Dynamic Tracking with Variable Size . . . . . . . . . . . . . . 48
5.3.3 Integration of Recognition and Dynamic Tracking . . . . . . . 49
ix
6 Databases 51
6.1 LTI-Gesture Database . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.2 DUISBURG-Gesture Database . . . . . . . . . . . . . . . . . . . . . . 52
6.3 i6-Gesture Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7 Experiments and Results 57
7.1 Basic HMM Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
7.1.1 Emission Distributions . . . . . . . . . . . . . . . . . . . . . . . 57
7.1.2 HMM Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
7.1.3 HMM Features Using Different Databases . . . . . . . . . . . . 64
7.2 Distance Measures Using Different Databases . . . . . . . . . . . . . . 72
7.2.1 Tangent Distance . . . . . . . . . . . . . . . . . . . . . . . . . . 72
7.2.2 IDM Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
7.2.3 Feature Size Results . . . . . . . . . . . . . . . . . . . . . . . . 78
7.3 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7.3.1 Bounding-Box Tracking . . . . . . . . . . . . . . . . . . . . . . 79
7.3.2 Camshift Tracking . . . . . . . . . . . . . . . . . . . . . . . . . 80
7.3.3 Dynamic Tracking . . . . . . . . . . . . . . . . . . . . . . . . . 82
8 Conclusion and Perspective 87
A Software Documentation 91
Bibliography 103
x