Appearance-Based Features for Automatic Continuous Sign Language Recognition

112 Pages·2006·2.59 MB·English

by  David Rybach

Diplomarbeit im Fach Informatik
Rheinisch-Westfa¨lische Technische Hochschule Aachen
Lehrstuhl fu¨r Informatik 6
Prof. Dr.-Ing. H. Ney
Appearance-Based Features for
Automatic Continuous
Sign Language Recognition
vorgelegt von:
David Rybach
Matrikelnummer 229337
Gutachter:
Prof. Dr.-Ing. H. Ney
Prof. Dr. J. Borchers
Betreuer:
Dipl.-Inform. T. Deselaers
Dipl.-Inform. P. Dreuw

Hiermit versichere ich, dass ich die vorliegende Diplomarbeit selbstst¨andig verfasst
und keine anderen als die angegebenen Hilfsmittel verwendet habe. Alle Textauszu¨ge
und Grafiken, die sinngem¨aß oder w¨ortlich aus ver¨offentlichten Schriften entnommen
wurden, sind durch Referenzen gekennzeichnet.
Aachen, im Juni 2006
David Rybach

Abstract
Thisdiplomathesisinvestigatesappearance-basedfeaturesfortheperson-independent
vision-basedrecognitionofcontinuoussignlanguage. Alargevarietyofmethodswhich
have been successfully used for automatic speech recognition is applied to this task.
Appearance-based approaches do not rely on a segmentation of the images or on pre-
defined models of the image content and use the image itself as the feature. A novel
tracking algorithm is introduced and applied to hand and head tracking. The tracked
body parts are used in order to calculate additional features to improve recognition
performance. The presented automatic sign language recognition system is evaluated
on a set of sentences in American Sign Language.
v

Acknowledgment
Iwouldliketoexpressmygratitudetoallpeoplewhosupportedmeduringtheprogress
of this work. Especially I would like to thank:
Prof. Dr.-Ing. Hermann Ney for the interesting possibilities at the Chair of Com-
puter Science 6 of the RWTH Aachen University and for many helpful discussions,
Prof. Dr. Jan Borchers who kindly accepted to co-supervise this work,
Thomas Deselaers and Philippe Dreuw for the supervision of this work and for their
countless ideas and suggestions,
Morteza Zahedi for the collection and the transcription of the RWTH-Boston-201
database,
Daniel Stein for the introduction to sign language and for the annotations of the
RWTH-Boston-104 database,
Stefan Hahn and Arne Theres for many helpful tips, comments and discussions about
speech recognition,
Daniel Schneider for proof reading the manuscript and a lot of helpful suggestions,
Julia Klubert who helped me annotating the RWTH-Boston-Hands database,
and finally my parents for supporting me in every way.
vii

Contents
1 Introduction 1
2 Sign Language 3
2.1 Manual Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Non-Manual Components . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Grammar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.4 Notation Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3 Sign Language Recognition 9
3.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2 Visual Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3 Language Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.5 Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.6 State of the Art in Sign Language Recognition . . . . . . . . . . . . . 18
4 Tracking 23
4.1 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Tracking using Dynamic Programming . . . . . . . . . . . . . . . . . . 24
4.3 Hand Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.4 Head Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5 Features for Sign Language Recognition 35
5.1 Image Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.2 Manual Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.3 Non-Manual Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.4 Feature combination . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6 Databases 45
6.1 RWTH-Boston-104 Database . . . . . . . . . . . . . . . . . . . . . . . 45
6.2 RWTH-Boston-Hands Database . . . . . . . . . . . . . . . . . . . . . . 48
ix
7 Results 51
7.1 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.2 Sign Language Recognition . . . . . . . . . . . . . . . . . . . . . . . . 60
8 Conclusion and Perspectives 79
A Additional Tables 83
List of Figures 89
List of Tables 91
Bibliography 93
x