Sub-Lexical and Contextual Modeling of Out-of-Vocabulary Words in Speech Recognition

19 Pages·2013·1.42 MB·English

THE JOHNS HOPKINS UNIVERSITY
Sub-Lexical and Contextual Modeling of
Out-of-Vocabulary Words in Speech Recognition
Carolina Parada, Mark Dredze, Abhinav Sethy, Ariya Rastrow
TECHNICAL REPORT 10
APRIL 27, 2013
(cid:13)c HLTCOE, 2013
Acknowledgements This work is supported, in part, by the Human Language
Technology Center of Excellence. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the sponsor.
HLTCOE
810 Wyman Park Drive
Baltimore, Maryland 21211
http://hltcoe.jhu.edu
Sub-Lexical and Contextual Modeling of Out-of-Vocabulary Words in
Speech Recognition
CarolinaParada1,MarkDredze1,AbhinavSethy2,andAriyaRastrow1
1HumanLanguageTechnologyCenterofExcellence,JohnsHopkinsUniversity
3400NCharlesStreet,Baltimore,MD,USA
[email protected], [email protected], [email protected]
2IBMT.J.WatsonResearchCenter,YorktownHeights,NY,USA
[email protected]
Abstract mation rich nouns, such as named entities and for-
eign words, and mis-recognizing them can have a
Large vocabulary speech recognition systems disproportionateimpactontranscriptcoherence.
fail to recognize words beyond their vocab- Hybridword/sub-wordrecognizerscanproducea
ulary, many of which are information rich
sequenceofsub-wordunitsinplaceofOOVwords.
terms, like named entities or foreign words.
Ideally, the recognizer outputs a complete word for
Hybrid word/sub-word systems solve this
in-vocabulary (IV) utterances, and sub-word units
problembyaddingsub-wordunitstolargevo-
cabularywordbasedsystems; newwordscan forOOVs. Considertheword“Slobodan”,thegiven
then be represented by combinations of sub- name of the former president of Serbia. As an un-
word units. We present a novel probabilis- commonEnglishword,itisunlikelytobeinthevo-
tic model to learn the sub-word lexicon op- cabulary of an English recognizer. While a LVCSR
timized for a given task. We consider the
system would output the closest known words (e.x.
task of Out Of vocabulary (OOV) word de-
“slow it dawn”), a hybrid system could output a
tection, which relies on output from a hybrid
sequence of multi-phoneme units: s l ow, b ax,
system. Wecombinetheproposedhybridsys-
d ae n. The latter ismore useful for automatically
temwithconfidencebasedmetricstoimprove
OOV detection performance. Previous work recovering the word’s orthographic form, identify-
addressOOVdetectionasabinaryclassifica- ing that an OOV was spoken, or improving perfor-
tiontask, whereeach regionisindependently manceofaspokentermdetectionsystemwithOOV
classifiedusinglocalinformation.Wepropose queries. Infact,hybridsystemshaveimprovedOOV
to treat OOV detection as a sequence label-
spoken term detection (Mamou et al., 2007; Parada
ingproblem,andweshowthat1)jointlypre-
etal.,2009),achievedbetterphoneerrorrates,espe-
dicting out-of-vocabulary regions, 2) includ-
cially in OOV regions (Rastrow et al., 2009b), and
ing contextual information from each region,
and3)learningsub-lexicalunitsoptimizedfor obtained state-of-the-art performance for OOV de-
this task, leads to substantial improvements tection(Rastrowetal.,2009a).
with respect to state-of-the-art on an English In this work, we consider how to optimally cre-
BroadcastNewsandMITLecturestask. ate sub-word units for a hybrid system. These
units are variable-length phoneme sequences, al-
though in principle our work can be used for other
1 Introduction
unit types. Previous methods have relied on simple
Most automatic speech recognition systems operate statisticscomputedfromthephoneticrepresentation
withalargebutlimitedvocabulary,findingthemost oftexttoobtainthemostfrequentunitsinthedictio-
likelywordsinthevocabularyforthegivenacoustic nary(Rastrow et al., 2009a; Bazzi and Glass, 2001;
signal. While large vocabulary continuous speech Bisani and Ney, 2005). However, it isn’t clear why
recognition (LVCSR) systems produce high quality theseunitswouldproducethebesthybridoutput. In-
transcripts, they fail to recognize out of vocabulary stead, we introduce a probabilistic model for learn-
(OOV)words. Unfortunately,OOVsareofteninfor- ing the optimal units for a given task. Our model
learns a segmentation of a text corpus given some detail several implementation aspects which have
side information: a mapping between the vocabu- notbeenaddressedinpreviouspublications. InSec-
lary and a label set; learned units are predictive of tion3wedetailhowthelearnedunitsareintegrated
classlabels. into a hybrid speech recognition system. Section 4
We learn sub-word units optimized for OOV de- describes the experimental setup and the OOV de-
tection. OOV detection aims to identify regions in tectionframework (Paradaetal.,2010). Wepresent
the LVCSR output where OOVs were uttered. To- results in Section 5 combining sub-lexical and con-
wards this goal, we are interested in selecting units textualmodels,andanalyzetheeffectofseveralpa-
suchthattherecognizeroutputsthemonlyforOOV rametersandvariationsofthemodelpreviouslyun-
regions while prefering to output a complete word examinedinParadaetal.(2011). Weconcludewith
for in-vocabulary regions. We combine the learned areviewofrelatedworkinSection6.
sub-words inthe output ofa hybridrecognizer with
confidence based metrics to improve OOV detec- 2 LearningSub-WordUnits
tionperformanceonanEnglishBroadcastNewsand
Givenrawtext,ourobjectiveistoproducealexicon
MITLecturestask.
of sub-word units that can be used by a hybrid sys-
Themaincontributionsofthepresentworkare:
temforopenvocabularyspeechrecognition. Rather
• Anovelstatisticalapproachtolearnsub-lexical than relying on the text alone, we also utilize side
units optimized for a given labeling task. Our information: a mapping of words to classes so we
model is a log-linear model which combines canoptimizelearningforaspecifictask.
overlapping features from the phonetic seg- TheprovidedmappingassignslabelsY tothecor-
mentation of a corpus. We apply this model to pus. We maximize the probability of the observed
theOOVdetectiontask. labeling sequence Y given the text W: P(Y|W).
We assume there is a latent segmentation S of this
• Several parameters and variations of the pro-
corpus which impacts Y. The complete data likeli-
posed sub-lexical model are studied that have (cid:80)
hoodbecomes: P(Y|W) = P(Y,S|W)during
S
not been addressed comprehensively in previ-
training. Since we are maximizing the observed Y,
ous publications. In particular, we examine
segmentationS mustdiscriminatebetweendifferent
theeffectofthemodelhyper-parametersonthe
possiblelabels.
size of the sub-word lexicon and average sub-
We learn variable-length multi-phone units by
word length, as well its impact in OOV detec-
segmentingthephoneticrepresentationofeachword
tionperformance.
in the corpus. Resulting segments form the sub-
word lexicon.1 Learning input includes a list of
• Wealsodiscussseveralimplementationaspects
words to segment taken from raw text, a mapping
of the proposed model for learning sub-word
between words and classes (side information indi-
units, which will be available online at time of
cating whether token is IV or OOV), a pronuncia-
publication.
tiondictionaryD,andalettertosoundmodel(L2S),
Our final system combines sub-lexical modeling suchastheonedescribedin StanleyF.Chen(2003).
with a state of the art OOV detector for substan- The corpus W is the list of types (unique words) in
tial additive gains on a Broadcast News and MIT- the raw text input. This forces each word to have a
Lectures data-set. We present results in terms of uniquesegmentation,sharedbyallcommontokens.
OOV detection, and previously unpublished results Words are converted into phonetic representations
in terms of hits and false alarms and phone-error- according to their most likely dictionary pronunci-
rate. ation;non-dictionarywordsusetheL2Smodel.2
We begin by presenting our log-linear model for
1Since sub-word units can expand full-words, we refer to
learning sub-word units with a simple but effective
bothwordsandsub-wordssimplyasunits.
inference procedure in Section 2. This model was 2The model can also take multiple pronunciations (Sec-
first described in Parada et al. (2011). Thereby, we tion2.2.1).
2.1 Model s l ow b ax d ae n
Inspired by the morphological segmentation model
of Poon et al. (2009), P(Y,S|W) is a log-linear
modelparameterizedbyΛ:
s l ow b ax d ae n
1
PΛ(Y,S|W) = uΛ(Y,S,W) (1) (#,#, ,b,ax) (l,ow, ,d,ae) (b,ax, ,#,#)
Z(W)
Figure1:Unitsandbigramphonecontext(inparenthesis)
where u (Y,S,W) defines the score of the pro-
Λ foranexamplesegmentationoftheword“slobodan”.
posed segmentation S for words W and labels Y
according to model parameters Λ. Sub-word units
parametersareΛ = {λ ,λ : ∀σ,c,y}. Theneg-
σcomposeS,whereeachσisaphonesequence,in- σ,y c,y
ative weights for the lexicon (α) and corpus priors
cludingthefullpronunciationforvocabularywords;
(β) are tuned on development data. The normalizer
the collection of σs form the lexicon. Each unit
Z sumsoverallpossiblesegmentationsandlabels:
σ is present in a segmentation with some context
c = (φl,φr) of the form φlσφr. Features based on Z(W) = (cid:88)(cid:88)u (Y(cid:48),S(cid:48),W) (3)
Λ
thecontextandtheunititselfparameterizeu .
Λ S(cid:48) Y(cid:48)
In addition to scoring a segmentation based on
Consider the example segmentation for the word
features,weincludetwopriorsinspiredbytheMin-
“slobodan”withpronunciations,l,ow,b,ax,d,ae,n
imum Description Length (MDL) principle sug-
(Figure1). Thebigramphonecontextasafour-tuple
gested by Poon et al. (2009). The lexicon prior
appears below each unit; the first two entries corre-
favors smaller lexicons by placing an exponential
spondtotheleftcontext,andlasttwotherightcon-
prior with negative weight on the length of the lex-
(cid:80) text. The example corpus (Figure 2) demonstrates
icon |σ|, where |σ| is the length of the unit σ
σ how unit features f and context features f are
in number of phones. Minimizing the lexicon prior σ,y c,y
computed.
favors a trivial lexicon of only the phones. The
corpus prior counters this effect, an exponential 2.2 ModelTraining
prior with negative weight on the number of units
Learning maximizes the log likelihood of the ob-
in each word’s segmentation, where |s | is the seg-
i servedlabelsY∗ giventhewordsW:
mentation length and |w | is the length of the word
i
in phones. Learning strikes a balance between the (cid:96)(Y∗|W) = log(cid:88) 1 u (Y∗,S,W) (4)
Λ
two priors. Using these definitions, the segmenta- Z(W)
S
tionscoreu (Y,S,W)isgivenas:
Λ
We use the Expectation-Maximization algorithm,
(cid:32) where the expectation step predicts segmentations
(cid:88)
uΛ(Y,S,W) = exp λσ,yfσ,y(S,Y) S given the model’s current parameters Λ (Sec-
σ,y tion2.2.1),andthemaximizationstepupdatesthese
(cid:88)
+ λ f (S,Y) parametersusinggradientascent. Thepartialderiva-
c,y c,y
tivesoftheobjective(4)withrespecttoeachparam-
c,y
(cid:88) eterλ are:
+ α· |σ| i
σ∈S ∂(cid:96)(Y∗|W)
= E [f ]−E [f ] (5)
(cid:33) ∂λ S|Y∗,W i S,Y|W i
(cid:88) i
+ β· |s |/|w | (2)
i i
The gradient takes the usual form, where we en-
i∈W
couragetheexpectedsegmentationfromthecurrent
f (S,Y) are the co-occurrence counts of the pair modelgiventhecorrectlabelstoequaltheexpected
σ,y
(σ,y)whereσ isaunitundersegmentationS andy segmentation and expected labels. We now discuss
isthelabel. f (S,Y)aretheco-occurrencecounts howtocomputetheseexpectations.
c,y
for the context c and label y under S. The model
Labeledcorpus: president/y =0milosevic/y =1
E [f ]asfollows:
S|Y∗,W i
Segmentedcorpus:
p r eh z ih d ih n t/0 1 (cid:88)
m ih/1 l aa/1 s ax/1 v ih ch/1 ES|Y∗,W[fi] ≈ M fi[Sj]
Unit-feature:Value j
p r eh z ih d ih n t/0:1
Similarly, to compute E we sample a seg-
m ih/1:1 l aa/1:1 s ax/1:1 v ih ch/1:1 S,Y|W
Context-feature:Value mentation and a label for each word. We com-
(#/0,#/0, ,l/1,aa/1):1, pute the joint probability of P(Y,S|W) for each
(m/1,ih/1, ,s/1,ax/1):1, segmentation-label pair using Eq. (1). A sampled
(l/1,aa/1, ,v/1,ih/1):1, segmentation can introduce new units, which may
(s/1,ax/1, ,#/0,#/0):1,
havehigherprobabilitythanexistingones.
(#/0,#/0, ,#/0,#/0):1
UsingtheseapproximationsinEq.(5),weupdate
theparametersusinggradientascent:
Figure 2: A small example corpus with segmentations
and corresponding features. The notation m ih/1:1
λ¯ = λ¯ +γ∇(cid:96) (Y∗|W)
represents unit/label:feature-value. Overlapping context new old λ¯
featurescapturerichsegmentationregularitiesassociated
whereγ > 0isthelearningrate.
witheachclass.
To obtain the best segmentation, we use deter-
ministic annealing. Sampling operates as usual,
2.2.1 Inference
except that the parameters are divided by a value,
Inference is challenging since the lexicon prior
which starts large and gradually drops to zero. To
renders all word segmentations interdependent.
make burn in faster for sampling, the sampler is
Consider a simple two word corpus: cesar
initialized with the most likely segmentation from
(s,iy,z,er), and cesium (s,iy,z,iy,ax,m). Nu-
the previous iteration. To initialize the sampler the
merous segmentations are possible; each word has
firsttime,wesetalltheparameterstozero(onlythe
2N−1 possible segmentations, where N is the num-
priors have non-zero values) and run deterministic
ber of phones in its pronunciation (i.e., 23 ×
annealing to obtain the first segmentation of the
25 = 256). However, if we decide to segment
corpus.
the first word as: {s iy,z er}, then the seg-
mentation for “cesium”:{s iy,z iy ax m} will
incur a lexicon prior penalty for including the 2.2.2 EfficientSampling
new segment z iy ax m. If instead we seg- Sampling a segmentation for the corpus requires
ment “cesar” as {s iy z,er}, the segmentation computing the normalization constant (3), which
{s iy,z iy ax m} incurs double penalty for the contains a summation over all possible corpus seg-
lexicon prior (since we are including two new units mentations. Instead, we approximate this constant
in the lexicon: s iy and z iy ax m). This depen- by sampling words independently, keeping fixed all
dency requires joint segmentation of the entire cor- other segmentations. Still, even sampling a single
pus, which is intractable. Hence, we resort to ap- word’ssegmentationrequiresenumeratingprobabil-
proximationsoftheexpectationsinEq.(5). itiesforallpossiblesegmentations.
One approach is to use Gibbs Sampling: it- We sample a segmentation efficiently using dy-
erating through each word, sampling a new seg- namic programming. We can represent all possible
mentation conditioned on the segmentation of all segmentations for a word as a finite state machine
other words. The sampling distribution requires (FSM) (Figure 3), where arcs weights arise from
enumerating all possible segmentations for each scoring the segmentation’s features. This weight is
word (2N−1) and computing the conditional prob- the negative log probability of the resulting model
abilities for each segmentation: P(S|Y∗,W) = afteraddingthecorrespondingfeaturesandpriors.
P(Y∗,S|W)/P(Y∗|W) (the features are extracted However, the lexicon prior poses a problem for
from the remaining words in the corpus). Using M thisconstructionsincethepenaltyincurredbyanew
sampled segmentations S ,S ,...S we compute unit in the segmentation depends on whether that
1 2 m
AA_N_JH_AA_N
N_JH_AA_N_IY
AA_N_JH_AA
N_JH_AA_N
AA
0 1 N_JH_AA
N_JH JH_AA 4 N
AA_N N JH_AA_N 5 IY
2 AA AA_N
AA_N_JH JH
AA_N_IY 6
3
JH_AA_N_IY
Figure3: FSMrepresentingallsegmentationsforthewordANJANIwithpronunciation: AA,N,JH,AA,N,IY
unit is present elsewhere in that segmentation. For Algorithm1Training
example, consider the segmentation for the word Input: Lexicon L from training text W, Dictionary D,
ANJANI:AA N,JH,AA N,IY.Ifnoneoftheseunits MappingM,L2Spronunciations,AnnealingtempT.
Initialization:
are in the lexicon, this segmentation yields the low-
Assignlabely∗ =M[w ]. λ¯ =¯0
estpriorpenaltysinceitrepeatstheunitAA N.3This m m 0
S =randomsegmentationforeachwordinL.
globaldependencymeanspathsmustencodethefull 0
unit history, making computing forward-backward fori=1toKdo
/* E-Step */
probabilitiesinefficient.
S =bestSegmentation(T,λ ,S ).
OursolutionistousetheMetropolis-Hastingsal- i i−1 i−1
gorithm, which samples from the true distribution fork =1toNumSamplesdo
P(Y,S|W) by first sampling a new label and seg- (Sk(cid:48),Yk(cid:48))=sampleSL(P(Y,Si|W),Q(Y,Si|W))
mentation (y(cid:48),s(cid:48)) from a simpler proposal distribu- S˜k =sampleS(P(Si|Y∗,W),Q(Si|Y∗,W))
tionQ(Y,S|W). Thenewassignment(y(cid:48),s(cid:48))isac- endfor
ceptedwithprobability:
/* M-Step */
α(Y(cid:48),S(cid:48)|Y,S,W)=min(cid:18)1,PP(Y(Y(cid:48),,SS|(cid:48)W|W))QQ((YY(cid:48),,SS|(cid:48)Y|Y(cid:48),,SS,(cid:48),WW))(cid:19) EESS,|YY|∗W,W[f[fi]σ=,l]=NuNmuS1mamS1apmlespl(cid:80)esk(cid:80)fkσ,fl[σS,lk(cid:48)[,S˜Ykk,(cid:48)]Y∗]
λ¯i =λ¯i−1+γ∇Lλ¯(Y∗|W)
endfor
We choose the proposal distribution Q(Y,S|W)
as Eq. (1) omitting the lexicon prior, removing the S=bestSegmentation(T,λK,S0)
Output: LexiconL fromS
challengeforefficientcomputation. Theprobability o
ofacceptingasamplebecomes:
2.3 ImplementationAspects
(cid:18) (cid:80) |σ|(cid:19) This section highlights some important aspects of
α(Y(cid:48),S(cid:48)|Y,S,W)=min 1, σ∈S(cid:48) (6)
(cid:80)
|σ| ourimplementationofthetrainingalgorithm1). The
σ∈S
key operation for training is sampling a segmen-
Alg. 1 shows our full sub-word learning procedure, tation for a word given the current model param-
where sampleSL (Alg. 2) samples a segmenta- eters (needed for bestSegmentation, sampleSL,
tion and label sequence for the entire corpus from andsampleS methods). Wesampleasegmentation
P(Y,S|W),andsampleSsamplesasegmentation for a word by sampling a path in a finite state ma-
fromP(S|Y∗,W). chinerepresentingallpossiblesegmentationsofthat
word. We use the OpenFst toolkit 4 and available
3Splittingatphoneboundariesyieldsthesamelexiconprior
butahighercorpusprior. 4http://www.openfst.org
Algorithm2sampleSL(P(S,Y|W),Q(S,Y|W)) probabilities after normalizing for the total weight
form=1toM(NumWords)do leaving each state. Once we sample a path, we ac-
(s(cid:48) ,y(cid:48) ) = Sample segmentation/label pair for ceptitaccordingtoEq.(6)orkeepthepreviousseg-
m m
wordwmaccordingtoQ(S,Y|W) mentationifrejected.
Y(cid:48) ={y ...y y(cid:48) y ...y }
1 m−1 m m+1 M To sample a segmentation for a word given its
S(cid:48) ={s ...s s(cid:48) s ...s }
α=min(cid:16)11,(cid:80)mσ∈−S1(cid:48)|σm|(cid:17)m+1 M true label (sampleS), we sample a path through an
(cid:80) |σ| acceptor including all segmentations for that word
σ∈S
withprobα:y =y(cid:48) ,s =s(cid:48)
m,k m m,k m withitstrue-label. Tosampleasegmentationandla-
withprob(1−α):y =y ,s =s
m,k m m,k m bel(sampleSL)wesampleapaththroughtheunion
endfor
oftwoacceptorsforthesamewordonewithlabel0
return (S(cid:48),Y(cid:48))=[(s ,y )...(s ,y )]
k k 1,k 1,k M,k M,k
and a second one with label 1. Finally, to obtain a
segmentation of the entire corpus, we sample one
word at a time assuming the segmentations of all
operationsasfollows.
otherwordsfixed. Whenusingannealingallweights
Given a word and a label (0-IV or 1-OOV) we
aredividedbyagiventemperature.
build a “segmentation” finite state machine, where
arc labels indicate the features for that segmenta-
3 OOVdetectionusinghybridmodels
tion and label. For example, in Figure 3, the la-
bel for the arc between nodes 2 and 3 would be:
To evaluate our model for learning sub-word units,
{JH/1,AA/1 N/1 AA/1 N/1}, where the por-
we consider the task of out-of-vocabulary (OOV)
tionofthelabelbeforethecommaindicatesthesub-
word detection. Since detectors process hybrid rec-
word JH, and its label 1, and the rest indicates the
ognizer output, we can evaluate different sub-word
bi-gram phone context of that sub-word. Note that
unit lexicons for the hybrid recognizer and measure
eachphone/sub-wordhasitsassociatedlabel.
thechangeinOOVdetectionaccuracy.
Each time we need to sample a segmentation for
Our model (Section 2.1) can be applied to this
a word, we load the transducer associated with that
task by using a dictionary D to label words as IV
word and label and assign a score to each arc using
(y = 0 if w ∈ D) and OOV (y = 1 if w ∈/ D).
i i i i
a Mapper.5 In our implementation, the mapper as-
This results in a labeled corpus, where the label-
signs weights to each arc according to the features
ing sequence Y indicates the presence of out-of-
encoded in the label and the current model parame-
vocabulary words (OOVs). Given a sub-word lex-
ters,plusthecorpusprior: β/|w |.
i icon,thewordandsub-wordsarecombinedtoform
To sample a path we run the forward-backward
a hybrid language model (LM) to be used by the
algorithm, where the backward computations
LVCSRsystem. ThishybridLMcapturesdependen-
are carried out explicitly by computing the
ciesbetweenwordandsub-words. IntheLMtrain-
ShortestDistance to the final state in the
ing data, all OOVs are represented by the smallest
LogSemiring. The forward pass is done through
numberofsub-wordsinalefttorightgreedyfashion
sampling, i.e. we traverse the machine only
whichcorrespondstotheirpronunciation. Pronunci-
computing forward probabilities for arcs leav-
ationsforallOOVsareobtainedusinggraphemeto
ing the sampled state. Specifically, we use the
phonemodels(StanleyF.Chen,2003).
RandGen operation with a customed arc selector
The output can be the one-best transcripts, lat-
BetaArcSelector. Thisoperationrandomlyselects
tices or confusion networks. While lattices contain
an outgoing transition at a given state with respect
moreinformation,theyarehardertoprocess;confu-
tothe weightsofeach arcTimestheprobability of
sionnetworksofferatrade-offbetweenrichnessand
leaving the destination state(computed in the back-
compactness (Mangu et al., 1999). Figure 4 depicts
ward pass). All weights are treated as negative log
a confusion network decoded by the hybrid system
for a section of an utterance in our test-set. Below
5“Mappers are function objects used by the
the network we present the reference transcription.
Map operation to transform arcs and/or final states”
(http://www.openfst.org/twiki/bin/view/FST/FstAdvancedUsage). In this example, two OOVs were uttered: “slobo-
mean
slow it
the most
president stalled i jh
former presidents s_l_ow but on in loss of edge in india
0 1 president’s 2 well 3 vote 4 i’m 5 a 6 l_aa_s 7 ax_v_ih 8 h. 9 10 11
know dose
spoke which
at
IV IV B-OOV I-OOV I-OOV I-OOV B-OOV I-OOV I-OOV IV IV
former president slobodan milosevic in india
Figure4: ExampleconfusionnetworkfromthehybridsystemwithOOVregionsandBIOencoding. Itrepresentsa
compactrepresentationsoftherecognizer’shypotheses,wherethetoppathcorrespondstothemostlikelyoutputfrom
therecognizer. Aconfusionnetworkiscomposedofasequenceofconfusedregions,indicatingthesetofmostlikely
word/sub-wordhypothesesutteredandtheirposteriorprobabilities.
dan”and“milosevic”anddecodedasfourandthree Maximum Entropy: Rastrow et al. (2009a)
in-vocabulary words, respectively. A confused re- present a MaxEnt classification based OOV detec-
gion(alsocalled“bin”)correspondstoasetofcom- tor, in which each region is classified as either IV
peting hypothesis between two nodes. The goal is (in-vocabulary) or OOV. Their two features are the
to correctly label each of the “bins” as OOV or IV. existence of a) sub-words and b) high entropy in a
Note the presence of both fragments (e.g. s l ow, networkregion.
l aa s)andwordsinsomeofthehypothesisbins. CRF: While Maximum Entropy assigns a label
Since sub-words represent OOVs while building to each region in the confusion network indepen-
the hybrid LM, the existence of sub-words in ASR dently, OOV words tend to co-occur since they are
output indicates an OOV region. Better sub-word often recognized as two or more IV words. In our
outputyieldsbetterrecognition. example (Figure 4), the OOV word “slobodan” was
Baseline Unit Selection: We used Rastrow et al. recognized as four IV words: “slow but i mean”.
(2009a)asourbaselineunitselectionmethod,adata Therefore, Parada et al. (2010) used a sequence
driven approach where the language model training model, specifically a Conditional Random Fields
textisconvertedintophonesusingthedictionary(or (CRF) (Lafferty et al., 2001), to capture this con-
a letter-to-sound model for OOVs), and a N-gram textual information. In addition to using this new
phoneLMisestimatedonthisdataandprunedusing model, they introduced several sets of contextual
a relative entropy based method. The hybrid lexi- features:
conincludesresultingsub-words–rangingfromun-
• Current Word: Words from the LVCSR de-
igramsto5-gramphones,andthe83Kwordlexicon.
coding of the sentence are used in the CRF
OOV detector. For each bin in the confusion
3.1 OOVDetectors
network, we select the word with the highest
Weconsidertwomodels: aMaximumEntropyOOV probability (best hypothesis). We then add the
detection system with features from filler and con- best hypothesis word as a feature of the form:
fidence estimation models proposed by Rastrow et current word=X.
al. (2009a), and a CRF sequence based model us-
ingcontextualinformationproposedbyParadaetal. • Context-BigramsUnigramsandbigramsfrom
(2010). While the sequence model does substan- the best hypothesis in a window of 5
tially better than the Maximum Entropy classifica- words around current bin. This feature ig-
tion model, we evaluate both to consider how our nores the best hypothesis in the current bin,
learned sub-word units compliments different types i.e., word[-2],word[-1] is included, but
ofinformation. Webrieflyreviewbothmodelshere. word[-1],word[0]isnot.
• Current-Trigrams Unigrams, bigrams, and approach (Section 2) or a state-of-the-art baseline
trigrams in a window of 5 words around and approach (Rastrow et al., 2009a) (Section 3). Our
includingcurrentbin. hybrid system’s lexicon has 83K words and 5K or
10K sub-words. Note that the word vocabulary is
• All-WordsAlloftheabovefeatures.
common to both systems and only the sub-words
are selected using either approach. The word vo-
Inadditiontolocalcontextfeatures, theyalsoin-
cabulary used is close to most modern LVCSR sys-
cluded global context features based probabilities
tem vocabularies for English Broadcast News; the
fromalanguagemodel: astandard4-gramlanguage
resultingOOVsaremorechallengingbutmorereal-
model with interpolated modified KN discounting
istic(i.e. mostlynamedentitiesandtechnicalterms).
and a syntactic language model (Filimonov and
The1290wordsareOOVstoboththewordandhy-
Harper, 2009). Both language models were trained
bridsystems.
on 130 million words from Hub4 CSR 1996 (Garo-
In addition we report OOV detection results on a
foloetal.,1996)andwererestrictedtoLVCSRsys-
MITlecturesdataset(Glassetal., 2010)consisting
tem’svocabulary. SeeParadaetal.(2010)forafull
of 3 Hrs from two speakers with a 1.5% OOV rate.
descriptionofthesefeatures.
These were divided into 1 Hr for training the OOV
4 ExperimentalSetup detectorand2Hrsfortesting. NotethattheLVCSR
systemistrainedonBroadcastNewsdata. Thisout-
4.1 Datasets
of-domaintest-sethelpusevaluatethecross-domain
We used the data set constructed by Can et al. performance of the proposed and baseline hybrid
(2009) (OOVCORP) for the evaluation of Spoken systems. OOVsinthisdatasetcorrespondmainlyto
Term Detection of OOVs since it focuses on the technical terms in computer science and math. e.g.
OOV problem. The corpus contains 100 hours of ALGORITHM, DEBUG, COMPILER, LISP.
transcribed Broadcast News English speech. There
4.2 Learningparameters
are 1290 unique OOVs in the corpus, which were
selectedwithaminimumof5acousticinstancesper For learning the sub-words we randomly selected
word and short OOVs inappropriate for STD (less fromtraining5,000wordswhichbelongtothe83K
than 4 phones) were explicitly excluded. Example vocabulary and 5,000 OOVs7. For development we
OOVs include: NATALIE, PUTIN, QAEDA, selected an additional 1,000 IV and 1,000 OOVs.
HOLLOWAY, COROLLARIES, HYPERLINKED, This was used to tune our model hyper parameters
etc. Thisresultedinroughly24K(2%)OOVtokens. (set to α = −1, β = −20). There is no overlap
For LVCSR, we used the IBM Speech Recogni- of OOVs in training, development and test sets. All
tion Toolkit (Soltau et al., 2010)6 to obtain a tran- feature weights were initialized to zero and had a
script of the audio. Acoustic models were trained Gaussian prior with variance σ = 100. Each of the
on 300 hours of HUB4 data (Fiscus et al., 1998) wordsintraininganddevelopmentwasconvertedto
andutterances containingOOVwordsas markedin their most-likely pronunciation using the dictionary
OOVCORP were excluded. The language model was forIVwordsortheL2SmodelforOOVs.8
trained on 400M words from various text sources Thelearningratewasγ = γ ,wherek is
k (k+1+A)τ
with a 83K word vocabulary. The LVCSR system’s theiteration,Aisthestabilityconstant(setto0.1K),
WER on thestandard RT04 BN testset was 19.4%. γ = 0.4, and τ = 0.6. We used K = 40 itera-
Excluded utterances amount to 100hrs. These were
7Thiswasusedtoobtainthe5Khybridsystem.Tolearnsub-
divided into 5 hours of training for the OOV detec-
words for the 10K hybrid system we used 10K in-vocabulary
torand95hoursoftest. NotethattheOOVdetector
wordsand10KOOVs.Allwordswererandomlyselectedfrom
trainingsetisdifferentfromtheLVCSRtrainingset. theLMtrainingtext.
We also use a hybrid LVCSR system, combining 8Inthisworkweignorepronunciationvariabilityandsim-
ply consider the most likely pronunciation for each word. It
word and sub-word units obtained from either our
isstraightforwardtoextendtomultiplepronunciationsbyfirst
6TheIBMsystemusedspeakeradaptivetrainingbasedon sampling a pronunciation for each word and then sampling a
maximumlikelihoodwithnodiscriminativetraining. segmentationforthatpronunciation.