Robust Speech Recognition in a Car Using a Microphone Array.

131 Pages·2012·1.55 MB·English

(cid:176)c 2006 by Bowon Lee, All rights reserved.
ROBUST SPEECH RECOGNITION IN A CAR USING A MICROPHONE ARRAY
BY
BOWON LEE
B.S., Seoul National University, 2000
M.S., University of Illinois at Urbana-Champaign, 2003
DISSERTATION
Submitted in partial fulflllment of the requirements
for the degree of Doctor of Philosophy in Electrical and Computer Engineering
in the Graduate College of the
University of Illinois at Urbana-Champaign, 2006
Urbana, Illinois
ABSTRACT
Performanceofautomaticspeechrecognitionreliesonavastamountoftrainingspeechdata
mostly recorded with little or no background noise. The performance degrades signiflcantly
with existence of background noise, which increases type mismatch between train and test
environments. Speech enhancement techniques can reduce the amount of type mismatch by
extracting reliable speech features from the background noise.
AtverylowSNRwithnonstationarynoise, theenhancedspeechmaystillcontainsigniflcant
noise either in noise-only segments or speech segments. The former masquerade as nonexistent
speech features and the latter as distorted speech features. Both signiflcantly degrade the
performance of the automatic speech recognizer. This encourages the use of voice activity
detection (VAD) algorithms to determine parts with speech present. To use only the reliable
speech features, we need to further determine whether the features from the speech region are
mainly from speech or from nonstationary noises masking the speech. For more robust speech
recognition, this thesis proposes a three-hypothesis VAD consisting of H : noise-only region;
0
H : speech-dominant speech region; and H : noise-dominant speech region.
S N
Spectrum-based VAD uses knowledge of the noise spectrum to detect voice activity using
the nonstationary nature of speech. This thesis proposes a method of estimating the instan-
taneous noise spectrum for VAD at low SNR when the noise estimation becomes inaccurate
due to its high variance. The spectrum-based VAD, however, cannot distinguish speech from
nonstationary noise because both appear nonstationary to the VAD, and thus look like speech.
A microphone array can determine the noise-corrupted speech region when the nonstationary
iii
noise is from a location other than that of the speech source. This thesis proposes a method
of distinguishing H from H based on the steered response power (SRP) method, which
S N
estimates power from any location.
Phonemic restoration is a phenomenon in which humans claim to hear missing phonemes
that have been replaced by noise with the same power. Given strong nonstationary noises
occasionallymaskingthespeechregion,aswellasknowledgeofH andH ,thisthesisproposes
S N
a phoneme restoration approach for automatic speech recognition in the hidden Markov model
framework.
We have built the AVICAR (Audio-Visual Speech in a CAR) database, a multimodal in-car
speech database consisting of recordings of 100 talkers using eight microphones and four video
cameras.
Thisdissertation,usingtheAVICARdatabase,describesmethodsofnoise-robustautomatic
speech recognition in a car with a microphone array. The approach has two steps: speech
enhancement as a preprocessor of noisy speech signals, followed by the phoneme restoration for
robust speech recognition against nonstationary noises.
iv
ACKNOWLEDGMENTS
This dissertation would not have been possible without the support of many people and I
would like to express my sincere gratitude to the following people.
First of all, my adviser, Professor Mark A. Hasegawa-Johnson, for his constant encour-
agement and guidance. His support enabled me to expand on my idea and added theoretical
depth to the work. His assistance and patience were instrumental to the completion of this
dissertation.
The committee members, Professor Stephen Levinson, Professor Thomas Huang, Professor
Douglas Jones, and Professor Charissa Lansing showed great interest in my work and have
made valuable advice, which improved the quality of this dissertation.
I also thank people of the Integrated Systems Lab of the Beckman Institute. The director,
Hank Kaczmarski, provided expertise and prompt support of various equipment and was in-
valuable in completing data collection for this dissertation. Camille Goudeseune, who has been
my mentor as well as a friend since the beginning of my graduate study, provided insightful
ideas and valuable advice and suggestions in critical occasions.
I also acknowledge Suketu Kamdar and Ming Liu, with whom I spent enormous time to-
gether during the data collection on the road in rural area of Urbana-Champaign. Also, more
than100anonymoushumansubjectsfromtheUniversityofIllinoisprovidedvaluablerecordings
to the database.
Special thanks go to my family, who showed unceasing love and trust on everything I have
decided to do. Very special thanks to my wife Leesa, whose support enabled me to concentrate
on my work and make it through.
v
TABLE OF CONTENTS
LIST OF TABLES : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : ix
LIST OF FIGURES : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : x
CHAPTER 1 INTRODUCTION : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1
CHAPTER 2 THE AVICAR DATABASE : : : : : : : : : : : : : : : : : : : : : : : : : : 6
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Array of Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.1 Microphone array and beamforming . . . . . . . . . . . . . . . . . . . . . 8
2.2.2 Visual feature extraction and 3D face modeling . . . . . . . . . . . . . . . 9
2.3 Equipment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.1 Audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.2 Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.3 Control sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.4 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 Scripts and Speakers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 Recording Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.6 Postprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.6.1 Audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.6.2 Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.7 Distribution of Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
CHAPTER 3 BACKGROUND : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 20
3.1 HMM-Based Automatic Speech Recognition . . . . . . . . . . . . . . . . . . . . . 20
3.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.1.2 Elements of a hidden Markov model . . . . . . . . . . . . . . . . . . . . . 21
3.1.3 Isolated word recognition using the HMM . . . . . . . . . . . . . . . . . . 23
3.1.4 Testing of HMMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2 Phonemic Restoration Efiect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2.2 Top-down and bottom-up processes . . . . . . . . . . . . . . . . . . . . . 28
3.3 Missing Feature Approach for ASR . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.3.1 Computational auditory scene analysis . . . . . . . . . . . . . . . . . . . . 29
3.3.2 Missing feature approach for ASR . . . . . . . . . . . . . . . . . . . . . . 31
3.3.3 Phonemic restoration approaches with ASR . . . . . . . . . . . . . . . . . 32
3.4 Microphone Array Signal Processing: Beamforming . . . . . . . . . . . . . . . . . 33
3.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4.2 Delay-and-sum beamforming . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4.3 Adaptive beamforming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4.4 Time-delay estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
vi
3.4.5 Source localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5 Voice Activity Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.5.2 Spectrum-based voice activity detection . . . . . . . . . . . . . . . . . . . 52
3.5.3 Noise adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
CHAPTER 4 THREE HYPHOTHESIS VOICE ACTIVITY DETECTION : : : : : : : 57
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2 Voice Activity Detection With Noise Estimation . . . . . . . . . . . . . . . . . . 58
4.2.1 MMSE a posteriori noise estimate with speech presence uncertainty . . . 60
4.2.2 Proposed noise estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.3 Voice Activity Detection With a Microphone Array . . . . . . . . . . . . . . . . . 62
4.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3.2 Location-based VAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.3.3 SRP-based VAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.3.4 PDF of SRP with one source . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3.5 SRP pdf computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.3.6 Simplifled SRP-based VAD . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
CHAPTER 5 PHONEME RESTORATION HIDDEN MARKOV MODEL : : : : : : : : 74
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.2 Top-Down Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2.1 A method to compute the probabilities of PRHMM . . . . . . . . . . . . 76
5.3 Bottom-Up Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.3.1 Similarity measure to determine reconstruction of missing phonemes . . . 79
5.3.2 Reconstruction of missing phonemes . . . . . . . . . . . . . . . . . . . . . 79
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
CHAPTER 6 EXPERIMENTS : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 82
6.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.1.1 Data for VAD experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.1.2 Data for ASR experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.2 Voice Activity Detection Experiments . . . . . . . . . . . . . . . . . . . . . . . . 83
6.2.1 Spectrum-based VAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.2.2 Array-based VAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.3 Automatic Speech Recognition Tests . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.3.1 Trained models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.3.2 Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.3.3 Speech recognition with the enhanced speech . . . . . . . . . . . . . . . . 89
6.3.4 Speech recognition test with the VAD . . . . . . . . . . . . . . . . . . . . 90
6.3.5 Speech recognition test with the VAD and PRHMM . . . . . . . . . . . . 90
6.3.6 Evaluating performance and determining the statistical signiflcance . . . . 91
vii
CHAPTER 7 RESULTS : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 93
7.1 VAD Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.1.1 Spectrum-based VAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.1.2 Array-based VAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.2 Speech Recognition Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.2.1 Baseline (BL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.2.2 Speech Recognition with the enhanced speech (BF) . . . . . . . . . . . . . 100
7.2.3 Speech recognition with VAD (BF VAD) . . . . . . . . . . . . . . . . . . . . 102
7.2.4 Phoneme restoration approach for speech recognition (BF PR) . . . . . . . 103
7.2.5 Summary of recognition results . . . . . . . . . . . . . . . . . . . . . . . . 105
CHAPTER 8 DISCUSSION : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 107
8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
8.2 Implications and Contributions of this Dissertation . . . . . . . . . . . . . . . . . 108
8.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
APPENDIX A LIST OF SPEAKER ID WITH THEIR DIALECTS OF THE AVICAR
DATABASE: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 111
APPENDIX B AVICAR LICENSE TERMS : : : : : : : : : : : : : : : : : : : : : : : : : 113
REFERENCES : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 115
AUTHOR’S BIOGRAPHY : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 121
viii
LIST OF TABLES
Table Page
2.1 Categories for each script set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Example fllenames for segmented audio data and their locations. . . . . . . . . . . . 17
2.3 Examples of frame information for the video data and their locations. . . . . . . . . 17
2.4 Examples of transcription for the video data. . . . . . . . . . . . . . . . . . . . . . . 18
6.1 Five noise conditions in the AVICAR database. . . . . . . . . . . . . . . . . . . . . . 83
6.2 Four categories of the ASR experiments. . . . . . . . . . . . . . . . . . . . . . . . . 87
6.3 List of groups in each set of the flve-fold cross-validation paradigm. . . . . . . . . . 89
6.4 Total number of utterances for baseline ASR experiment. . . . . . . . . . . . . . . . 89
6.5 Total number of utterances for ASR experiment. . . . . . . . . . . . . . . . . . . . . 90
7.1 P ’s of the VADs with P = 5% under four difierent noise conditions from the
d f
AVICAR database (NE: Proposed noise estimation, NA: noise adaptation, FN:
Fixed noise spectrum). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.2 Summary of P ’s of the VADs (P = 5%) under various SNRs. . . . . . . . . . . . . 96
d f
7.3 List of experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.4 Recognition results (% correct recognition) of the baseline experiment (BL) with
trained model set IDL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.5 Recognition results (% correct recognition) of the baseline experiment (BL) with
trained model set ALL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.6 Recognitionresults(%correctrecognition)ontheenhancedspeech(BF)withtrained
model set IDL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.7 Recognitionresults(%correctrecognition)oftheenhancedspeech(BF)withtrained
model set ALL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
7.8 Recognition results (% correct recognition) of BL, BF, BF VAD FN, BF VAD NA, and
BF VAD NE with trained model set IDL. . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.9 Recognitionresults(%correctrecognition)of BL, BF, BF VAD FN, BF VAD NA,and
BF VAD NE with trained model set ALL. . . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.10 Recognitionresults(%correctrecognition)of BF VAD, BF PR LVAD, and BF PR SVAD
with trained model set IDL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.11 Recognition results (% correct recognition) of BF VAD, BF PR LVAD, and BF PR SVAD
with trained model set ALL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.12 Summary of WER with trained model set IDL. . . . . . . . . . . . . . . . . . . . . . 106
A.1 Dialects of female subjects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
A.2 Dialects of male subjects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
ix
LIST OF FIGURES
Figure Page
2.1 Circuit diagram of the microphone preamplifler. . . . . . . . . . . . . . . . . . . . . 10
2.2 Eight microphones on the sun visor, four cameras on the dashboard. . . . . . . . . . 11
2.3 Script with the telephone handset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Equipment diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.5 The speech waveform ‘seven’ under flve noise conditions. . . . . . . . . . . . . . . . 16
2.6 A snapshot of the video stream. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1 An example of a flve state left-to-right HMM. . . . . . . . . . . . . . . . . . . . . . 23
3.2 An illustration of the picket fence efiect. . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 An illustration of the delay-and-sum beamforming. . . . . . . . . . . . . . . . . . . . 36
3.4 Beam patterns of delay-and-sum beamformer. . . . . . . . . . . . . . . . . . . . . . 37
3.5 A diagram of the generalized sidelobe canceler (GSC). . . . . . . . . . . . . . . . . . 38
4.1 Histogram of power at the 100th frequency bin of 400 point DFT of white Gaussian
noise and pdf of an exponential distribution with mean 0:4. . . . . . . . . . . . . . . 59
5.1 Procedure of the phoneme restoration approach. . . . . . . . . . . . . . . . . . . . . 81
6.1 Diagram of microphone positioning and source location of the AVICAR database. . 85
7.1 ROC curve for 5 dB SNR with 55U noise conditions. FN: Fixed noise, NA: Noise
adaptation, NE: Proposed noise estimation. . . . . . . . . . . . . . . . . . . . . . . . 94
7.2 Spectrogramoftheenhancedspeechandsourcelocationestimationoftheutterance
EM4 55D P6 C1. The four speech regions detected by the VAD correspond to the
breath groups of the utterance: \two one," \flve nine zero eight," \zero flve one,"
and \seven." Notice that the VAD detects the /f/ at the beginning of \flve" and
the /s/ at the beginning of \seven." . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.3 Summary of results in terms of the word error rate with trained model set IDL. . . . 106
x