Table Of ContentList of Figures
Figure 1.1 Relations between the five research areas. 2
Figure 1.2 A hierarchical video representation for scripted content. 5
Figure 1.3 A hierarchical video representation for unscripted content. 5
Figure 2.1 A hierarchical video representation for scripted video content. 16
Figure 2.2 An example video ToC. 27
Figure 2.3 Merging scene 1 to scene 0. 29
Figure 2.4 The Gaussian A^(0, 1) distribution. 32
Figure 2.5 Proposed approach for detecting documentary scenes. 36
Figure 2.6 Video ToC for Movie 1 (scene level). 36
Figure 2.7 Video ToC for Moviel (group level). 37
Figure 3.1 Our proposed framework: an overview. 40
Figure 3.2 Audio markers for sports highlights extraction. 40
Figure 3.3 Examples of visual markers for different sports. 41
Figure 3.4 MDL{K, 6) (Y-a,xis) with respect to different numbers of GMM mix- 45
tures AT (X-axis) to model (a) applause, (b) cheering, (c) music, (d)
speech, and (e) speech with music sound shown in the raster-scan
order. K = 1 • • • 20. The optimal mixture numbers at the lowest
positions of the curves are 2, 2, 4, 18, and 8, respectively.
Figure 3.5 Precision-recall curves for the test golf game. Left: by the current 49
approach; right: by the previous approaches; Y-axis: precision;
X-axis: recall.
Figure 3.6 The interface of our system displaying sports highlights. The hori- 50
zontal line imposed on the curve is the threshold value the user can
choose to display those segments with a confidence level greater than
the threshold.
Figure 3.7 A snapshot of the interface of the results in our previous approach. 51
The horizontal line imposed on the curve is the threshold.
Figure 3.8 Some examples of the typical view of the squatting baseball catcher. 53
Figure 3.9 A typical video camera setup for live soccer broadcast. 54
Figure 3.10 Some examples of the typical view of the first view of the goalpost. 55
Figure 3.11 Some examples of the typical view of the second view of the goalpost. 56
Figure 3.12 Some examples of the first view of the golfer. 57
Figure 3.13 Some examples of the second view of the golfer. 58
Figure 3.14 Some examples of the third view of the golfer. 59
XI
Xll List of Figures
Figure 3.15 Example rectangle features shown relative to the enclosing detection 60
window. The sum of the pixels that lie within the white rectangles are
subtracted from the sum of pixels in the gray rectangles. Two-rectangle
features are shown in (a) and (b). Figure (c) shows a three-rectangle
feature, and (d) shows a four-rectangle feature.
Figure 3.16 The AdaBoost algorithm. 61
Figure 3.17 Schematic depiction of the detection cascade. A series of classifiers are 62
appHed to every subwindow. The initial classifier eliminates a large
number of negative examples with very little processing. Subsequent
layers eliminate additional negatives but require additional computa
tion. After several stages of processing, the number of subwindows
has been reduced radically. Further processing can take any form, such
as additional stages of the cascade or an alternative detection system.
Figure 3.18 The first few weak classifiers learned by the AdaBoost algorithm for 63
the catcher model.
Figure 3.19 The precision-recall curve of baseball catcher detection. 64
Figure 3.20 Two examples of the preprocessing step for the two soccer goal- 66
post views. The thumbnail images (bottom row) are taken from the
"whitened" images (middle row).
Figure 3.21 The first few weak classifiers learned by the AdaBoost algorithm for 67
the first view of the goalpost model.
Figure 3.22 The first few weak classifiers learned by the AdaBoost algorithm for 67
the second view of the goalpost model.
Figure 3.23 The precision-recall curve of goalpost detection for view 1. 68
Figure 3.24 The first few weak classifiers learned by the AdaBoost algorithm for 69
the first view of the golfer model.
Figure 3.25 The first few weak classifiers learned by the AdaBoost algorithm for 70
the second view of the golfer model.
Figure 3.26 An example of the change of color characteristics in a baseball hit. 73
Figure 3.27 The scaled version of each video frame's average hue value over 75
time for the 18 training "putt" sequences. The scaling factor is
1000/MAX(-). X-axis: video frames; Y-axis: scaled average hue
values.
Figure 3.28 The scaled version of each video frame's average hue value over 76
time for the 32 training "swing" sequences. The scaling factor is
1000/MAX(-). X-axis: video frames; Y-axis: scaled average hue
values.
Figure 3.29 The scaled version of each video frame's average hue value over time 77
for the 95 test sequences.
Figure 3.30 The scaled version of each video frame's average motion intensity 78
value over time for the 18 training "putt" sequences. The scaHng fac
tor is 1000/MAX(-). X-axis: video P-frames; Y-axis: scaled average
motion intensity values.
Figure 3.31 The scaled version of each video frame's average motion intensity 79
value over time for the 32 training "swing" sequences. The scal
ing factor is 1000/MAX(-). X-axis: video P-frames; Y-axis: scaled
average motion intensity values.
List of Figures xiii
Figure 3.32 The scaled version of each video frame's average motion intensity 80
value over time for the 95 test sequences. The scaUng factor is
1000/MAX(-). X-axis: video P-frames; Y-axis: scaled average motion
intensity values.
Figure 3.33 Comparison results of three different modeling approaches in terms 81
of precision-recall curves. Solid line: audio modeling alone; dashed
line: audio + dominant color modeling; dotted line: audio -h dominant
color + motion modeling. X-axis: recall; Y-axis: precision.
Figure 3.34 The graphical model structure of the DCHMM. The two rows of 83
squares are the coupled hidden state nodes. The circles are the
observation nodes.
Figure 3.35 Precision-recall curves for the test golf game. The four highlights 92
extraction methods compared here are (1) audio classification fol
lowed by long, contiguous applause selection; (2) HMM classification
using the models trained from audio labels of the highlight and non-
highlight examples; (3) HMM classification using the models trained
from video (motion) labels of the highlight and nonhighlight exam
ples; and (4) coupled HMM classification using the models trained
from both audio and video (motion) labels of the highlight and non-
highlight examples. X-axis: recall; Y-axis: precision. The first two
curves coincide with each other (see text for detail).
Figure 3.36 Precision-recall curves for the test soccer game. The four highlights 94
extraction methods compared here are (1) audio classification fol
lowed by long, contiguous cheering selection; (2) HMM classification
using the models trained from audio labels of the highlight and non-
highlight examples; (3) HMM classification using the models trained
from video (motion) labels of the highlight and nonhighlight exam
ples; and (4) coupled HMM classification using the models trained
from both audio and video (motion) labels of the highlight and
nonhighlight examples. X-axis: recall; Y-axis: precision.
Figure 3.37 A snapshot of the interface of our system. The horizontal line imposed 96
on the curve is the threshold.
Figure 4.1 A hierarchical video representation for unscripted content. 98
Figure 4.2 Proposed event discovery framework for analysis and representation 99
of unscripted content for summarization.
Figure 4.3 MFCC feature extraction. 103
Figure 4.4 MFCC feature extraction. 104
Figure 4.5 MDCT coefficients for the first block. 105
Figure 4.6 MDCT coefficients for the second block. 106
Figure 4.7 MDCT coefficients for a frame from the mean of six MDCT blocks 107
without absolute value operation.
Figure 4.8 MDCT coefficients for a frame from the mean of six MDCT blocks 108
with the absolute value operation.
Figure 4.9 Energy normalization for MDCT coefficients. 108
Figure 4.10 rjYsq for effective rank. 109
Figure 4.11 />6 Vs ^ for effective rank. 110
Figure 4.12 Audio classification framework. Ill
xiv List of Figures
Figure 4.13 Input time series, (a) Time along X-axis and symbols (1, 2, 3) along 115
Y-axis, (b) The computed affinity matrix.
Figure 4.14 Zoomed-in view of the input time series from Pi and P2, time along 116
X-axis and symbols (1, 2, 3) along Y-axis.
Figure 4.15 Proposed outlier subsequence detection framework. 120
Figure 4.16 Generative model for synthetic time series with one background 122
process and one foreground process.
Figure 4.17 Cases of interest. 123
Figure 4.18 Performance of normalized cut on synthetic time series for case 2. 124
(a) X-axis for time, Y-axis for symbol, (b) Affinity matrix, (c) X-axis
for candidate normalized cut threshold, Y-axis for value of normalized
cut objective function, (d) X-axis for time index of context model,
Y-axis for cluster indicator value, (e) X-axis for time, Y-axis for symbol.
Figure 4.19 Performance of /:-means on synthetic time series for case 2. (a) X-axis 129
for time index of context model, Y-axis for cluster indicator value.
(b) X-axis for time, Y-axis for symbol.
Figure 4.20 (a) Performance of dendrogram cut on synthetic time series for 130
case 2. (b) X-axis for time, Y-axis for symbol.
Figure 4.21 Structured "salient" foreground in unstructured background. 131
Figure 4.22 Performance of modified normalized cut on synthetic time series for 133
case 2. (a) X-axis for candidate threshold for cut, Y-axis for cut value.
(b) X-axis for candidate threshold for modified normalized cut, Y-axis
for modified normalized cut value, (c) X-axis for time index of context
model, Y-axis for cluster indicator value, (d) X-axis for time, Y-axis
for symbol.
Figure 4.23 Performance comparison of normalized cut and modified normalized 136
cut on synthetic time series for case 3. (a) X-axis for time, Y-axis
for symbol, (b) X-axis for candidate threshold for normalized cut,
Y-axis for normalized cut value, (c) X-axis for time index of context
model, Y-axis for cluster indicator value (normalized cut), (d) X-axis
for time index of context model, Y-axis for cluster indicator value
(foreground cut).
Figure 4.24 Performance of hybrid (normalized cut and foreground cut) 138
approach on synthetic time series for case 4. (a) X-axis for time, Y-axis
for symbol, (b) (top) X-axis for time index of context model, Y-axis
for cluster indicator; (bottom) corresponding temporal segmentation.
(c) (top) X-axis for time index of context model, Y-axis for cluster
indicator (second normalized cut); (bottom) corresponding temporal
segmentation, (d) Final temporal segmentation.
Figure 4.25 PDF of the defined distance metric for binomial PDF as a back- 144
ground model for context sizes of 200 and 400 symbols.
Figure 4.26 PDF of the defined distance metric for multinomial PDF as a back- 147
ground model for a context size of 200.
List of Figures xv
Figure 4.27 PDFs of distance metrics for different background models, (a) PDF 150
of an estimate of a context model parameter, (b) PDF of distances
for a binomial context model, (c) PDF of distances for a multinomial
context model, (d) PDF of distances for a GMM as a context model.
(e) PDF of distances for a HMM as a context model.
Figure 4.28 Ranking outliers based on PDF of distance between inliers. 153
Figure 4.29 (a) Affinity matrix for a 3-hour long British Open golf game and 157
(b) 30-minute news video using one-second-classification labels.
Figure 4.30 Comparison of outlier subsequence detection with low-level audio fea- 158
tures and frame-level classification labels for sport and surveillance,
(a) Outlier subsequences in frame labels time series for golf, (b) Out
lier subsequences in low-level features time series for golf, (c) Outlier
subsequences in frame labels time series for soccer, (d) Outlier
subsequences in low-level features time series for soccer.
Figure 4.31 Inlier/Outlier-based segmentation for a 20-minute clip of Japanese 162
baseball content, (a) First normalized cut. (b) Second normalized
cut. (c) Third normalized cut. (d) Final temporal segmentation after
foreground cut from each background.
Figure 4.32 Affinity matrices and their corresponding temporal segmentation for 167
the three clips of Footl002 clip of MPEG-7. (a, b) -4 minutes to
19 minutes, (c, d) —19 minutes to 36 minutes, (e, f) —36 minutes to
52 minutes.
Figure 4.33 Inlier/Outlier-based segmentation for second half of World Cup final 171
soccer game.
Figure 4.34 Scene segmentation for situation comedy content using the proposed 173
time series outlier subsequence detection framework.
Figure 4.35 Systematic acquisition of domain knowledge using the inlier/outlier- 180
based representation framework.
Figure 4.36 Comparison of highlights extraction using audio classification: 181
Applause and cheering versus excited speech with cheering.
Figure 4.37 Comparison of highlights ranking using cheering and applause against 182
using the "highlight" audio class.
Figure 4.38 Comparison of precision-recall performance using cheering and 183
applause against using the discovered "highlight" audio class.
Figure 4.39 Four-second segments of a 20-minute soccer test clip in (foreground- 184
background) likelihood space.
Figure 4.40 Example hierarchical segmentation for an event-free clip in elevator 187
surveillance content.
Figure 4.41 Example of hierarchical segmentation for a clip with suspicious 191
activity in elevator surveillance content.
Figure 4.42 Classification results on test clips with suspicious activity. 193
Figure 4.43 Combining offline time series analysis with online analysis for event 196
detection in surveillance.
Figure 5.1 MPEG-7 motion descriptions. 201
Figure 5.2 Perspective projection and camera motion parameters. 202
Figure 5.3 Motion trajectory. 208
List of Figures
Figure 5.4 Video browsing. Extracting the 10 most "active" video segments in a 208
news program.
Figure 5.5 Illustration of video browsing with and without descriptors of spatial 209
distribution of motion activity.
Figure 5.6 Video indexing using motion activity combined with color. 210
Figure 5.7 Illustration of adaptive subsampling approach to video summarization. 210
The top row shows a uniform subsampling of the surveillance video,
while the bottom row shows an adaptive subsampling of the surveil
lance video. Note that the adaptive approach captures the interesting
events, while the uniform subsampling mostly captures the highway
when it is empty. The measure of motion activity is the average motion
vector magnitude in this case, which is similar to the MPEG-7 intensity
of a motion activity descriptor in performance.
Figure 5.8 WWW-based interface for dynamic video summarization. 211
Figure 6.1 The condensed ToC. 222
Figure 6.2 The expanded ToC. 222
Figure 6.3 Proposed approach for sports highlights extraction. 223
Figure 6.4 Audio markers for sports highlights extraction. 224
Figure 6.5 Examples of visual markers for different sports. 224
Figure 6.6 The precision-recall curve of baseball catcher detection. 225
Figure 6.7 From video clip to cluster to index for scripted content. 228
Figure 6.8 From video clip to cluster to index for unscripted content. 229
Figure 6.9 A unified framework. 230
Figure 6.10 Subclusters. 231
Figure 6.11 Interface for going from the semantic index to the highlights. 233
Figure 6.12 An image used as a visual index (catcher detected). 234
Figure 6.13 Interface for going from the highlights to the visual index. 234
Figure 7.1 Architecture of a summarization-enabled DVD recorder. 239
Figure 7.2 Prototype of a summarization-enabled DVD recorder. 240
Figure 7.3 Architecture of a summarization-enabled home network. 241
Figure 7.4 Architecture for summarization-enhanced multimedia services over 242
the Internet.
Figure 7.5 An exemplar airport security system. 245
List of Tables
Table 2.1 Scene structure construction results. 34
Table 3.1 Performance of traditional GMM, every class is modeled using 10 46
Gaussian mixtures: (1) applause, (2) cheering, (3) music, (4) speech,
and (5j "speech with music." Classification accuracy on the 10%
data by models trained on the 90% data.
Table 3.2 Performance of MDL-GMM. Classification accuracy on the 10% 46
data by models trained on the 90% data. (1) to (5) are the same as
described in Table 3.1.
Table 3.3 The confusion matrix on all the audio data. The results are based 47
on MDL-GMMs with different "optimal" numbers of mixtures (see
Fig. 3.4).
Table 3.4 Recognition results of those in Table 3.3. The number of sound 47
examples that are correctly or incorrectly classified for each class is
shown.
Table 3.5 Number of contiguous applause segments and highlights found by 48
the MDL-GMMs in the golf game. These highlights are in the vicin
ity of the applause segments. These numbers are plotted in the left
on Figure 3.5.
Table 3.6 Precision-recall values for catcher detection. 65
Table 3.7 Precision-recall values for goalpost detection for view 1. 68
Table 3.8 Results before and after the finer-resolutionh ighlights classification. 74
Table 3.9 Audio labels and class names. 83
Table 3.10 Video labels and class names. 84
Table 3.11 Recognition matrix (or confusion matrix) on a 90%/10% training/ 86
testing split of a data set composed of seven classes: (1) applause,
(2) ball-hit, (3) cheering, (4) music, (5) female speech, (6) male
speech, and (7) speech with music. The results here are based on
MFCCs and GMMs.
Table 3.12 Number of observed labels in each state. 88
Table 3.13 Averaged classification accuracy on 10% of the total golf highlight 88
and nonhighlight data using the audio HMMs.
Table 3.14 Number of observed labels in each state. 90
Table 3.15 Averaged classification accuracy on 10% of the total golf highlight 90
and nonhighlight data using the video HMMs.
Table 3.16 Averaged classification accuracy on 10% of the total golf highlight 91
and nonhighhght data using the CHMMs.
XVll
XVUl List of Tables
Table 3.17 Averaged classification accuracy on 10% of the total soccer highlight 93
and nonhighlight data using CHMMs.
Table 4.1 Outlier ranks in baseball audio; Ri: Average normalized rank using 164
PDF estimate; /?2- Average normalized distance.
Table 4.2 OutUer ranks in soccer audio; Ri: Average normahzed rank using 165
PDF estimate; /?2- Average normalized distance.
Table 4.3 Outlier ranks in golf audio; /? i: Average normalized rank using PDF 166
estimate; /?2- Average normalized distance.
Table 4.4 Ground truth of highlight moments for Footl002 MPEG-7 clip. 170
Table 4.5 Systematically acquired sound classes from detected outliers and 186
inliers after analysis of surveillance content from elevators.
Table 6.1 From the semantic, visual, camera index to the ToC. 232
Table 6.2 From the ToC (shots) to the index. 235
Table 7.1 AppUcation categories. 238
Preface
The goal of this book is to develop novel techniques for constructing the
video table of contents (ToC), video highlights, and the video index as well
as to explain how to integrate them into a unified framework. To achieve
these goals, we have introduced a hierarchical representation that includes
key frames, shots, groups, and scenes as presented for scripted video and
another hierarchical representation that includes play/break, audio-visual
markers, highlight candidates, and highlight groups for unscripted video.
We also have presented a state-of-the-art content-adaptive representation
framework on unsupervised analysis for summarization and browsing.
We have shown how this framework can support supervised analysis as
well. We have reviewed different video indexing techniques that are based
on color, texture, shape, spatial layout, and motion activity. We then have
presented a unique and unified framework for video summarization, brows
ing, and retrieval to support going back and forth between the video ToC
and the video index for scripted video, and between video highlights and
video index for unscripted video. Furthermore, we have reviewed many
of the video summarization, browsing, and retrieval applications that have
already used these technologies.
The book is designed for sharing our research results with graduate
students and researchers in video analysis and computer vision. We look
forward to inquiries, collaborations, criticisms, and suggestions from them.
XIX
Acknowledgments
Part of this work (Rui and Huang) was supported by Army Research Labs
(ARL) Cooperative Agreement No. DAALO1-96-2-0003 and in part by a
Computer Science Education (CSE) Fellowship, College of Engineering,
University of Illinois at Urbana-Champaign (UIUC). The authors (Xiong,
Radhakrishnan, and Divakaran) would like to thank Dr. Mike Jones of
MERL for his help in visual marker detection. They also would like to thank
Mr. Kohtaro Asai of Mitsubishi Electric Corporation (MELCO) in Japan
for his support and encouragement and Mr. Isao Otsuka of MELCO for
his valuable application-oriented comments and suggestions. The authors
(Rui and Huang) would like to thank Sean X. Zhou, Atulya Velivelli, and
Roy R. Wang for their contributions.
We would also like to thank Mr. Charles B. Glaser of Elsevier for initiat
ing this book project and Rachel Roumeliotis and Rick Adams of Elsevier
for supporting the publication of this book.
One author (Xiong) carried out the research reported in the book when
he was a Ph.D. student at UIUC. His current employer, the United Tech
nologies Research Center (UTRC), does not warrant or assume any legal
liability or responsibility for the accuracy, completeness, or usefulness of
any information, apparatus, product, or process reported in this book. The
views and opinions of authors expressed in this book do not necessarily
state or reflect those of UTRC, and they may not be used for advertising or
product endorsement purposes.
XXI
