Robust acoustic modeling and front-end design for distant speech recognition

126 Pages·2017·2.72 MB·English

by  Mirsamadi, Seyedmahdad

ROBUST ACOUSTIC MODELING AND FRONT-END DESIGN FOR
DISTANT SPEECH RECOGNITION
by
Seyedmahdad Mirsamadi
APPROVED BY SUPERVISORY COMMITTEE:
John H. L. Hansen, Chair
P. K. Rajasekaran
Carlos Busso
Yang Liu
Chin-Tuan Tan
Copyright (cid:13)c 2017
Seyedmahdad Mirsamadi
All rights reserved
Dedicated to my parents, Mansour and Soheila
ROBUST ACOUSTIC MODELING AND FRONT-END DESIGN FOR
DISTANT SPEECH RECOGNITION
by
SEYEDMAHDAD MIRSAMADI, BS, MS
DISSERTATION
Presented to the Faculty of
The University of Texas at Dallas
in Partial Fulfillment
of the Requirements
for the Degree of
DOCTOR OF PHILOSOPHY IN
ELECTRICAL ENGINEERING
THE UNIVERSITY OF TEXAS AT DALLAS
December 2017
ACKNOWLEDGMENTS
First and foremost, I would like to express my special appreciation and thanks to my PhD
advisor, Dr. John H. L. Hansen, for his continuous support and remarkable patience, his
valuable guidance, and his unique professional and moral standards in directing academic
research. The opportunity of working with him in the past few years has been an extremely
valuable experience which not only made me a far better researcher in speech technology,
but also helped me learn many other aspects of being an effective researcher and contributor
by observing his unique style of leadership and management.
Iwouldliketoexpressmysinceregratitudetomycommitteemembers,Dr.P.K.Rajasekaran,
Dr.CarlosBusso, Dr.YangLiuandDr.Chin-TuanTanfortheirprecioustimeandinvaluable
comments on my research.
It was a pleasure working with my fellow researchers and colleagues at the Center for Robust
Speech Systems (CRSS). Working together with them was an important aspect of my PhD
experience. In particular, I would like to thank Navid Shokouhi, Abhinav Misra, Chengzhu
Yu, Qian Zhang, and Shabnam Ghaffarzadegan for all the fruitful scentific discussions, sup-
port, and friendship throughout my years at CRSS. I am also thankful to CRSS alumni
Dr. Omid Sadjadi, Dr. Oldooz Hazrati, and Dr. Ali Ziaei for all their valuable help during
my first years as a PhD student at CRSS.
This acknowledgement would be incomplete without thanks to my mentors at Microsoft
Research (MSR) during my summer internships, Dr. Ivan Tashev and Dr. Cha Zhang. The
opportunity of working with them at MSR greatly enriched my PhD experience and helped
me become a better researcher.
I am also appreciative of the efforts of the administrative staff at the Electrical Engineering
Department at UT-Dallas. My special thanks goes to Ms. Tammy Emery for her continuous
help at the different stages of my studies at CRSS.
v
Last but not least, I would like to thank my parents, Mansour and Soheila, for their con-
tinuous support throughout my life, particularly during the years of studying abroad. I am
very grateful for their significant role in enabling me to initiate my studies at UT-Dallas,
and also for their invaluable support during my PhD years.
July 2017
vi
ROBUST ACOUSTIC MODELING AND FRONT-END DESIGN FOR
DISTANT SPEECH RECOGNITION
Seyedmahdad Mirsamadi, PhD
The University of Texas at Dallas, 2017
Supervising Professor: John H. L. Hansen, Chair
In recent years, there has been a significant increase in the popularity of voice-enabled tech-
nologies which use human speech as the primary interface with machines. Recent advance-
ments in acoustic modeling and feature design have increased the accuracy of Automatic
Speech Recognition (ASR) to levels that enable voice interfaces to be used in many appli-
cations. However, much of the current performance is dependent on the use of close-talking
microphones, (i.e., scenarios in which the user speaks directly into a hand-held or body-worn
microphone). There is still a rather large performance gap experienced in distant-talking
scenarios in which speech is recorded by far-field microphones that are placed at a dis-
tance from the speaker. In such scenarios, the distorting effects of distance (such as room
reverberation and environment noise) make the recognition task significantly more challeng-
ing. In this dissertation, we propose novel approaches for designing a distant-talking ASR
front-end as well as training robust acoustic models to reduce the existing gap between far-
field and close-talking ASR performance. Specifically, we i) propose a novel multi-channel
front-end enhancement algorithm for improved ASR in reverberant rooms using distributed
non-uniform microphone arrays with random unknown locations; ii) propose a novel neural
network model training approach using adversarial training to improve the robustness of
multi-condition acoustic models that are trained directly on far-field data; iii) study alter-
vii
nate neural network adaptation strategies for far-field adaptation to the acoustic properties
of specific target environments. Experimental results are provided based on far-field bench-
mark tasks and datasets which demonstrate the effectiveness of the proposed approaches for
increasing far-field robustness in ASR. Based on experiments using reverberated TIMIT sen-
tences, the proposed multi-channel front-end provides WER improvements of +21.5% and
+37.7% in two-channel and four-channel scenarios over a single-channel scenario in which
the channel with best signal quality is selected. On the acoustic modeling side and based
on results of experiments on AMI corpus, the proposed multi-domain training approach
provides a relative charachter error rate reduction of +3.3% with respect to a conventional
multi-condition trained baseline, and +25.4% with respect to a clean-trained baseline.
viii
TABLE OF CONTENTS
ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
LIST OF ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
CHAPTER 1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Dissertation Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Dissertation Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
CHAPTER 2 BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1 Speech recognition fundamentals . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 The ASR front-end . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Acoustic modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.1 GMM-HMM models . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.2 Hybrid DNN-HMM models . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.3 End-to-end RNN models . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Distant speech model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.5 Fundamental problem in distant speech recognition . . . . . . . . . . . . . . 24
2.6 Existing solutions for distant speech recognition . . . . . . . . . . . . . . . . 26
2.6.1 Microphone array processing . . . . . . . . . . . . . . . . . . . . . . . 26
2.6.2 Robust features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.6.3 Feature enhancement . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.6.4 Model adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.6.5 Multi-condition training . . . . . . . . . . . . . . . . . . . . . . . . . 28
CHAPTER3 AROBUSTMULTICHANNELFRONT-ENDFORDISTRIBUTEDMI-
CROPHONE ARRAYS: CNTF ALGORITHM . . . . . . . . . . . . . . . . . . . 29
3.1 Limitations of conventional microphone arrays . . . . . . . . . . . . . . . . . 29
3.2 Distributed microphone arrays . . . . . . . . . . . . . . . . . . . . . . . . . . 30
ix
3.3 CNTF algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3.1 Nonnegative tensor model for reverant spectrograms . . . . . . . . . . 32
3.3.2 Alpha-beta divergence . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3.3 Multiplicative update rules . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4 Parameter selection by score-matching . . . . . . . . . . . . . . . . . . . . . 38
3.4.1 The score-matching principle . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.2 Score-matching estimator for α and β in CNTF algorithm . . . . . . 40
3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.5.1 Speech Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.5.2 ASR system setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.5.3 ASR results and algorithm analysis . . . . . . . . . . . . . . . . . . . 43
3.5.4 Experiments with multi-condition and enhanced training data . . . . 46
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
CHAPTER 4 MULTI-DOMAIN ADVERSARIAL TRAINING OF NEURAL NET-
WORK ACOUSTIC MODELS FOR IMPROVED FAR-FIELD ROBUSTNESS . 51
4.1 Robust representation learning with multi-condition DNNs . . . . . . . . . . 51
4.2 Multi-domain training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3 Environment invariance in hidden representations . . . . . . . . . . . . . . . 55
4.3.1 Environment-specific features in hidden layers . . . . . . . . . . . . . 55
4.3.2 domain classification accuracy in hidden layers . . . . . . . . . . . . . 56
4.3.3 Links to factor-aware training . . . . . . . . . . . . . . . . . . . . . . 59
4.4 Multi-domain adversarial acoustic model training . . . . . . . . . . . . . . . 60
4.4.1 Generative Adversarial Networks (GAN) . . . . . . . . . . . . . . . . 60
4.4.2 Multi-domain Adversarial training of RNN-CTC models . . . . . . . 61
4.4.3 Comparison with multi-task learning . . . . . . . . . . . . . . . . . . 65
4.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.5.1 System setup and data . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.5.2 Results on multi-domain adversarial training . . . . . . . . . . . . . . 68
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
x