Using Variations of Shape and Appearance in Alignment Methods for Classifying Human Actions

95 Pages·2014·5.4 MB·English

Using Variations of Shape and Appearance
in Alignment Methods for Classifying
Human Actions
by
Sultan Mohammad Almotairi
A dissertation submitted to
Florida Institute of Technology
in partial fulfillment of the
requirements for the degree of
Doctor of Philosophy
in
Computer Science
Melbourne, Florida
November, 2014
We the undersigned committee hereby approve the attached dissertation
Using Variations of Shape and Appearance in Alignment Methods for
Classifying Human Actions
by
Sultan Mohammad Almotairi
Eraldo Ribeiro, Ph.D.
Associate Professor, Computer Sciences and Cybersecurity
Committee Chair, Dissertation Advisor
Walter Bond, Ph.D.
Associate Professor, Computer Sciences and Cybersecurity
Committee Member
Philip Bernhard, Ph.D.
Associate Professor, Computer Sciences and Cybersecurity
Committee Member
Muzaffar Shaikh, Ph.D.
Professor, Engineering Systems
Outside Committee Member
Richard Ford, Ph.D.
Professor, Computer Sciences and Cybersecurity
Department Head
Abstract
Title: Using Variations of Shape and Appearance in Alignment Methods for
Classifying Human Actions
Author: Sultan Mohammad Almotairi
Committee Chair: Eraldo Ribeiro, Ph.D.
In this dissertation, we address the problem of recognizing human-action
from videos. The recognition aims at recovering action information from the
image sequences using different features such as variations of the human shape.
Approaches based on such features often use sequence-alignment methods. We
propose two novel methods for human-action recognition. We also propose an
elliptical-shaped band for the Dynamic Time Warping (DTW) that provides a good
compromise between alignment accuracy and computational speed.
First, we study the applicability of the pairwise shape-similarity measurements
for human-action recognition. Since action can be seen as a sequence of shapes
of silhouette poses, there exists similarities between actions from the same class.
Based on this observation, we propose a new method for classifying human actions.
Given two sequences of silhouettes representing an action, we measure their
iii
similarity by means of a robust sequence-alignment method. The motion cue is
implicitlyrepresentedbytheimplicitvariationsofthehuman’sshapeovertimewhile
an action is performed. We adopt the Longest Common Sub-Sequence (LCSS), a
dynamic-programming approach that calculates the minimum cost of aligning the
two sequences.
Next, we use information from inter-pose shape variations as provided by shape
descriptors for recognizing human actions. Here, in contrast to the previous
method, whereanactionisnotmodeledbyitself, wepresentamethodthatconverts
anactionintoasequencebasedonthevariationsofahuman’sshapeovertime. We
construct the sequence using the Inner-Distance Shape-Context as a measurement
of variations between shapes. Experimental results compare our method favorably
with related methods.
Finally, we develop a new global band for the Dynamic Time Warping algorithm.
In contrast with standard rectangular-shaped bands, we propose an elliptical-shaped
band that provides flexibility and a good compromise between alignment accuracy
and computational speed. The shape of the ellipse is implicitly represented by the
length of the time series. The idea of our elliptical band is to speed up DTW
and enforce a global constraint on the warping path by using a window size that
tolerates a significant amount of noise in the aligned time series.
iv
Contents
1 Introduction 1
1.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Literature Review 5
2.1 Action recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Time-series alignment . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 Action Classification 11
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Problem statement and main components . . . . . . . . . . . . . . 12
3.2.1 Data pre-processing . . . . . . . . . . . . . . . . . . . . . . 13
3.2.2 Measuring pose similarity . . . . . . . . . . . . . . . . . . . 14
3.2.3 Similarity between sequences of silhouettes . . . . . . . . . 18
3.2.4 Classification approach . . . . . . . . . . . . . . . . . . . . 20
3.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3.2 Experimental setup . . . . . . . . . . . . . . . . . . . . . . 23
3.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
v
3.4 Conclusions and future work . . . . . . . . . . . . . . . . . . . . . 30
4 Human Action Recognition Using Temporal Sequence Alignment 31
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2 Problem statement and main components . . . . . . . . . . . . . . 32
4.2.1 Data pre-processing . . . . . . . . . . . . . . . . . . . . . . 34
4.2.2 Action recognition . . . . . . . . . . . . . . . . . . . . . . 36
4.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.3.2 Experimental settings . . . . . . . . . . . . . . . . . . . . . 39
4.3.3 Results and discussion . . . . . . . . . . . . . . . . . . . . 40
4.4 Conclusions and future work . . . . . . . . . . . . . . . . . . . . . 45
5 Elliptical Band for Dynamic Time Warping 46
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.2 Problem statement and main components . . . . . . . . . . . . . . 48
5.2.1 Original dynamic time warping . . . . . . . . . . . . . . . . 48
5.2.2 Speeding up the DTW computation . . . . . . . . . . . . . 50
5.2.3 Elliptical band . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.4 Conclusions and future work . . . . . . . . . . . . . . . . . . . . . 67
6 Conclusions 68
vi
List of Figures
1.1 Examples of human actions . . . . . . . . . . . . . . . . . . . . . . 2
2.1 Examples of silhouette, optical flow, and trajectory . . . . . . . . . 7
3.1 Pairwise shape-similarity measurement for constructing the
sequence-alignment . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 The process of isolating the foreground silhouette using a simple
background subtraction operation . . . . . . . . . . . . . . . . . . 14
3.3 Building shape descriptors using Shape Context . . . . . . . . . . . 15
3.4 Building shape descriptor using Inner-Distance Shape Context . . . 17
3.5 A Comparison of shape-similarity measurement using using SC and
IDSC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.6 Hierarchical clustering of time series . . . . . . . . . . . . . . . . . 19
3.7 A simple example of aligning two sequences using LCSS . . . . . . 22
3.8 Frames from Keck-Gesture dataset . . . . . . . . . . . . . . . . . . 24
3.9 Frames from the Weizmann and ViHASi action datasets . . . . . . 25
3.10 Result on the Weizmann dataset (confusion matrix) . . . . . . . . 27
3.11 Results on the ViHASi dataset using SC+LCSS . . . . . . . . . . . 28
vii
3.12 Results on the Weizmann dataset . . . . . . . . . . . . . . . . . . 29
3.13 Results on the ViHASi dataset using graph representation . . . . . 29
3.14 ROC curves of action classification on the Weizmann dataset . . . 30
4.1 Overview of human-action recognition using temporal sequence
alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.2 Two consecutive frames representation . . . . . . . . . . . . . . . . 35
4.3 LCSS similarity measurement . . . . . . . . . . . . . . . . . . . . . 37
4.4 Sample of subjects from the ViHASi dataset . . . . . . . . . . . . . 38
4.5 Results on the Weizmann dataset (confusion matrix) . . . . . . . . 41
4.6 Results on the ViHASi dataset (confusion matrix) . . . . . . . . . . 41
4.7 Sensitivity analysis of the LCSS’ threshold (cid:15) . . . . . . . . . . . . . 43
4.8 Accuracy recognition using a different number of sample points . . 43
4.9 Examples of matching sequences . . . . . . . . . . . . . . . . . . . 44
5.1 An example of two semi-similar time series aligned using Euclidean
matching (Left) and DTW matching (Right) . . . . . . . . . . . . 47
5.2 Cost matrix with a minimum distance warp path . . . . . . . . . . 49
5.3 Samples of global bands. . . . . . . . . . . . . . . . . . . . . . . . 51
5.4 Shape of an elliptical band . . . . . . . . . . . . . . . . . . . . . . 53
5.5 The efficiency of different global bands (length vs. time) using
MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.6 The relation between the number of visited cells and the warping
window sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
viii
5.7 The relation between the speed efficiency and the warping window
sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.8 Result on speed (length vs. time) using C/C++ . . . . . . . . . . . 58
5.9 Result on speed (window size vs. time) using C/C++ . . . . . . . . 58
5.10 Result on length vs. time using error shading bar . . . . . . . . . . 60
5.11 Result on warping window size vs. time using error shading bar . . . 60
5.12 Control Chart Patterns dataset . . . . . . . . . . . . . . . . . . . . 61
5.13 The recognition accuracy using the Control Chart Patterns dataset 62
5.14 Leaf extraction process . . . . . . . . . . . . . . . . . . . . . . . . 63
5.15 The recognition accuracy using the leaf dataset . . . . . . . . . . . 64
5.16 TherecognitionaccuracyusingtheHuman-ObjectInteractiondataset 65
5.17 Different sizes and shapes of global bands . . . . . . . . . . . . . . 66
5.18 Results on the Synthetic Control Chart Time Series dataset . . . . 66
ix
List of Tables
3.1 Classification rate on the Weizmann and ViHASi datasets . . . . . 26
3.2 Results reported using the Weizmann database. . . . . . . . . . . . 26
4.1 Comparison of the accuracy result on the Weizmann and ViHASi
datasets using leave-one-out cross validation. . . . . . . . . . . . . 42
4.2 Comparison of different results reported in the literature using the
Weizmann database. . . . . . . . . . . . . . . . . . . . . . . . . . 42
x