Table Of ContentDetecting gross alignment errors in the Spoken British National Corpus
LadanBaghai-Ravary,SergioGrau,GregKochanski
PhoneticsLaboratory,UniversityofOxford,UnitedKingdom
{ladan.baghai-ravary,sergio.graupuerto,greg.kochanski}@phon.ox.ac.uk
Abstract tended periods of time can introduces numerical errors
whichdistortthealignmentprocess[3].
The paper presents methods for evaluating the accuracy of
alignments between transcriptions and audio recordings. The
As a result, automatic alignment of large quantities of
methods have been applied to the Spoken British National
spontaneous unconstrained speech is invariably error-prone.
Corpus, which is an extensive and varied corpus of natu-
When labelling “speech in the wild” such as the Spoken
ral unscripted speech. Early results show good agreement
BritishNationalCorpus(BNC)[4], theabovefourconditions
with human ratings of alignment accuracy. The methods
frequentlyoccur,andleadtofailuresofthealignmentsystem.
1 also provide an indication of the location of likely alignment Identifyingwheresuchanalignmentsucceedsandwhereitfails
1 problems; this should allow efficient manual examination of
allowsbadregionstobeavoidedorrealigned.
0 large corpora. Automatic checking of such alignments is
Previousworkonthistopicissparse.Traditionally,aligner
2 crucial when analysing any very large corpus, since even the
accuracy is assessed by comparison with manually estimated
bestcurrentspeechalignmentsystemswilloccasionallymake
n labels in one way or another [5]. However, the task here is
serious errors. The methods described here use a hybrid
a fundamentally different in that we operate under conditions
approachbasedonstatisticsofthespeechsignalitself,statistics
J
whereanyalignermayfail,regardlessofwhetheritisgenerally
ofthelabelsbeingevaluated,andstatisticslinkingthetwo.
9 accurateornot. [6]haveaddressedourprobleminadifferent
IndexTerms:ASPA,HMM,phonetic,transcription,label,seg-
context(cleanspeechthatisintendedforuseinTTSsystems)
ment,alignment,accuracy,qualityassessment,errordetection
] with some success. They flagged 24% of the segments as
D
suspicious and detected 43% of the total errors, which would
1. Introduction
S haveledtoamodestreductionintheeffortrequiredtoverify
s. In linguistic and phonetic research, increasing emphasis is acorpus. [7]lookedattherelatedproblemoffindingerrorsin
c being placed on the analysis of very large corpora, to give thelexiconusedinthealignmentprocess.Sometext-to-speech
[ more general, stronger conclusions. However, the analysis of systembuildersmayalsohaveusedsimilarideas(e.g.[8])but
such databases often requires aligned textual and audio data, details are lacking. Some preliminary work was done by [9],
1
and performing such alignments manually is extremely time- usingideasrelatedtoourimprobableandunexpectedfeatures.
v
consumingandhenceexpensive. Automaticalignmentcanbe The work described here is part of a project [10] to label
2
performedbytranscribingthewordsspokeninthecorpusand speechfromtheSpokenBNC,originallyrecordedonanalogue
8
thenaligningthemtothespeechviastandardHMMtechniques cassette tapes between 1991 and 1994. The data consists
6
1 (e.g. [1]). These Automatic Speech-to-Phoneme Alignment of recently-digitised recordings with an associated word-level
. (ASPA) systems can produce accurate estimates of word and transcription of the audio data. The recordings are mostly of
1
phonemelocations,butoftenfailincasessuchas: unscripted, spontaneous speech, and include a diverse range
0
of recording conditions, accents, and microphone positions
1. The speech has been recorded in an environment
1
relative to the speakers. Each track of the original cassette
1 with non-stationary background noise, competing
recordings has been digitised to a single file, so the data to
: un-transcribedspeech,distortion,and/orreverberation.
v bealignedisgenerallyjustovereither45or60minuteslong.
2. Thephonemictranscriptionofthespeechisnotaccurate.
i Background noise varies widely, both in terms of amplitude
X The phonemic transcription is usually obtained by a
andcharacter(competingspeech,mechanicalnoise,musicand
dictionary look-up process. Citation forms are often
r speech from television, radio or other sources, microphone-
a used,andareoftenunrealistic,especiallywhendealing
handlingnoise,etc.).
withspontaneousspeech. Innatural,unscriptedspeech,
Ourearlierwork[11,12]investigatedalignmenterrorsby
peoplewillsometimestalksimultaneouslysotheremay
comparingthealignmentsproducedbyalargenumberofASPA
benosequenceofwordsorphonemesthatcancorrectly
systems. However, inthatworkwewereattemptingtoassess
representtheaudio. Evenifthewordscanbeidentified
the general ability of aligners to identify particular phone-
and transcribed individually, they cannot be organised
transitions,andthegeneralqualityofalignmentsproducedbya
into a simple sequence, and so require more complex
specificsystem,respectively.
techniques (e.g. [2]), which are impractical for large
corpora.
2. Methods
3. Theword-leveltranscriptionisinaccurateand/orincon-
sistent in its handling of nonspeech sounds, backchan-
We used human evaluations of overall alignment quality for
nels,mumbles,andspeech-likenoises(e.g.dogbarks).
eachofasetofrecordings,toconstructalgorithmsthatshould
4. Thespeechisonlyavailableinlongcontinuousrecord- beabletoidentifysuspiciousregions,thentocomparetheseto
ings: the accumulation of HMM probabilities over ex- thehumanevaluations.
2.1. AlignmentProcedures in duration between the phone class under consideration and
the previous phone class: log(D(p )/D(p )). The third
To evaluate our methods, we used the Penn Phonetics Lab i i−1
capturesthetypicaldifferenceinlog(P)betweenthephoneme
ForcedAligner,P2FA[13]toanalyse46recordingstakenfrom
class under consideration and the preceding phoneme class:
the BNC, each consisting of a full audio session recorded on
λ(p )−λ(p ).Thefourthandfiftharethesame,exceptthey
onesideofanaudiotapewithoutabreak. i i−1
refertothesucceedingphoneme.Thepredictor(alongwiththe
P2FAisanautomaticphoneticalignerbasedonHTK,and
medians) is trained on phonemes with 0.04s ≤ δ ≤ 0.18s.
developed at the Phonetics Laboratory of the University of i
Theoutputofthisalgorithmbecomesthe“unexpected”feature.
Pennsylvania.ItemploysmonophoneGaussianMixtureModel
Word Log(P): we consider words with four or more
basedHMMswhichweretrainedusing39perceptuallinearpre-
phonemestoavoidvariationsduetothevagariesofindividual
diction(PLP)coefficients. Weusedthe39phonesetfromthe
phonemes. The individual L values for each phoneme are
Carnegie-MellonUniversityPronouncingDictionary,CMUdict i
summedovertherespectiveword,thennormalisedbydividing
[14], with the addition of OH for British English /6/ distinct
by the word duration. This log probability per unit time
from/!/, includinglexicalstressmarkingforthevowels. The
provides a stable indication of the goodness of fit of the
currentCMUPronouncingDictionarywasextendedtoinclude
observed data to the HMM. The final stage of of the Word
all the out-of-vocabulary words and to include a range of
Log(P)methodcomparesthenormalisedlogprobabilitywitha
commonBritishEnglishwordpronunciations. Thisextension
fixedthreshold,yieldingthe“improbable”feature.Thismethod
was performed using semi-automatic methods by experienced
iscomplemementarytotheUnexpectedLog(P)method,above,
phoneticians.
in that it combines data from a whole word, whereas the
Unexpected Log(P) method utilises features at the individual
2.2. HumanEvaluations
phoneme level, before smoothing them, i.e. the Unexpected
We observed that some of the alignments were much more Log(P)algorithmnormalisesL bydurationoveralargerunit.
i
successful than others, and we quantified that with a rating IntheUnexpectedLog(P)algorithm,thethresholdwasset
procedure.Oneoftheauthorsexamined5-secondlongregions, byexperiment,toidentifyabout100suspiciousregionsperhour
approximatelyonceevery60secondsthroughoutthefiles,and on files that had substantial alignment problems. The Word
checked whether that region was correctly aligned at a word Log(P)thresholdwassettoidentifyasimilarnumberofevents
level. The overall score of the file was subjective on a scale on poorly aligned files, and typically fewer on well-aligned
between0(verypoor)and10(verygood),butwasintendedto ones.
reflectthenumberandmagnitudeofalignmentproblems. Extremes of Amplitude: Many alignment systems will
produceerroneousalignmentswhenseveralconsecutivespeech
2.3. Algorithms labelsbunch-upintoashortregion,withtheremainingspeech
labelledasanextendedsilence.Thismisidentificationofspeech
We developed five algorithms to indicate potential problems
and silence can be detected by examining the amplitude of
with an alignment, listed below. Each one takes the aligner
signalsineachlabelledphoneme.Acontiguousregionofquiet,
output(segmentlabeltimes,andlog-probabilities)andoption-
ofalengthcomparabletoashortwordwithinasegmentlabelled
allytheaudiofile,andidentifiesalistofsuspiciouslocations.
asspeech,indicatesthatanerrormayhaveoccurred.Similarly,
Inthesedescriptions,L isthealigner’sHMMlog-probability
i an error is likely if there is a word-length region of high
value for phoneme instance i, p is the phoneme (i.e. /a/, /t/,
i amplitudewithinasegmentlabelledassilence.Highamplitude
/S/,...),andδ istheduration. Thealgorithmsweredeveloped
i “silence”regionsshouldbemarkedforhumaninspectioneven
without reference to the human evaluations, except for the
if they do not contain speech, because they represent high-
settingofeachalgorithm’sthreshold.
amplitude background noise, which is itself a potential cause
Unexpected Log(P): This algorithm builds a prediction
ofproblemsinreal-worlddata.
of the aligner’s log-probability score per unit length from the
The thresholds for “quiet” and “high amplitude” were set
corpusasawhole(exceptthedatafileunderanalysis). Itthen
as the 3rd and the 97th percentile of the amplitudes observed
computesthedifferencebetweenL /δ andtheprediction1. It
i i overthewholeoftherecording.Thenominallengthofa“short
operatesontheassumptionthatmostoftheaudiointhecorpus
is correctly aligned so that its predictions correspond to good word”wassetto1/4second,assumingfourphonemeswithan
alignment. Thus, whenthealignerisdoingworsethanusual, average duration of 1/16 second each. These parameters were
estimatedbyexperiment,andchosentogivearelativelysmall
and L is low, the difference will be substantially negative.
i numberoffalsepositives. Thisalgorithmproducestwofactors
Wehaveobservedthatwhenthealignerfails,ittypicallyfails
(“loud”,and“quiet”)asitreportstheextremesseparately.
for a relatively large region: a word or more. To make use
WordDuration:thisalgorithmsimplyexaminesthedura-
of this knowledge, we smooth the difference over a 1 second
tionsofthesegments,andifthereareanywhichareunexpect-
long region. When this smoothed difference of log(P) is
edlylongorshort,indicatesanerror.Itisdifficult(simplyfrom
more negative than a threshold, the algorithm has identified a
theirduration)todetectperiodsofsilencewhichhavebecome
suspiciousregion.
extendedormergedduetoanalignmenterror.Butthedurations
The predictor for L starts with the median value for that
i ofsegmentslabelledasspeechcanbeofgreathelp.
phoneme, λ(p ) = median(L /δ if p = p ). Itthenadds
i j j j i This algorithm takes a word-based approach to detecting
ina5-termlinearprediction. Theindependentvariableinthe
unusualsegmentdurations. Individualphonemedurationsare
first term is log(δ /D(p )), where D is the median duration
i i notreliableindicatorsbecauseofvariabilitiesinpronunciation
of a phoneme class, and D(p ) = median(δ if p = p ).
i j j i dueeithertodialect,styleofspeaking,ortheeffectsoftransient
The remaining four terms capture some information on the
background noise. Thus we take all words with four or more
phonemesequence. Thesecondcapturesthetypicaldifference
phonemes, calculate the duration of the region labelled as the
1Wedropphoneswithδi =0orwithδi ≥1sonthegroundsthat word, normalise it by dividing by the number of phonemes,
thescalingofLiwithδimaynotbeaccurateonthesephones. and compare it with two thresholds representing the largest
andthesmallestaveragephonemeduration. Anyresultoutside
the range 1/32 s < meanduration < 1/8 s is flagged. The
lower threshold is just above the minimum possible duration
of a phoneme label for our HMMs (which use 3 left-to-right
statesperphone). Thisalgorithmyieldstwofeatures(“short”
and“long”),asitreportsthetwoextremesseparately.
Duration Mismatch: This algorithm builds a duration
modelforphonemesandthenmeasureshowfareachphoneme2
deviatesfromthemodel.Regionsareidentifiedassuspiciousif
thesmoothedabsolutevalueofthedeviationislargeenoughto
exceedathreshold.
Thedurationmodelpredictsthelogofthephonemedura-
Figure 1: Quiet segment error: the “quiet” detector has
tion as d . It starts with a value typical of that phone: ∆ =
i i identified a region (shaded), labelled as the word “it’s”. The
median(log(δ ) if p = p ). Itthenaddsona25-termlinear
j i j vowel is omitted from the labelled region, and the end of the
predictor: the constant term captures a constant offset from
region extended into silence. The whole region is very quiet.
∆ .Then,twelvetermscapturethelengthofnearbyphonemes
i Thetoptieristhespectrogram,thenphonemeandwordlabels,
relative to their median durations (six neighbours on each
respectively.
side),viafactorsthatareq(D δ ,D δ ),wherekspecifies
i i+k i+k i
(cid:26)2(a/b)0.5−2, if a≤b(cid:27)
whichneighbour3. Theq(a,b) =
2−q(b,a), else
elected to regress both with and without the transforms. This
function is a sigmoid whose domain is [0,∞], and it is well-
ledtofourregressionsforeachchoiceofs,or12regressionsin
behaved at the endpoints, an important property since some
all.
phoneme durations are zero. The final 12 terms similarly
Of those 12 regressions, short was statistically significant
capturethedifferencesbetweenthetypicaldurationsofneigh-
on10(P < 0.01),badlengthwassignificanton6,loudon4.4
bouringphones. Thearerepresentedbyfeaturesq(D ,D ).
i i+k At least one of those factors was significant at the P < 0.01
This duration model is trained to match log(δ ) as in the
i level in each regression. Pearson’s R2 averaged 0.66 with a
UnexpectedLog(P)algorithm.
standarddeviationof0.13overtheregressions,indicatingthat
Finally,eachphoneisscoredbyS =|δ −d |/m ,where
i i i i a combination of the algorithms was reasonably effective at
m =median(|log(δ )−∆ |if p =p )isthemeanabsolute
i j i j i matchingthehumanjudgementsofoverallalignmentaccuracy.
deviation of the log duration. The scores are then smoothed
Ofthebestthreefits(R2 = 0.81, 0.81, 0.87), short was
and thresholded as inthe Unexpected Log(P) algorithm. This
significantoneachatP <0.01,alongwithbadlengthandloud
producesthe“badlength”feature.
once each. Of these, one used sdd and did not transform the
data at all; the other two used snw and did not transform the
3. Results
independentvariables.
Figures1and2showtwoexamplesofregionscorrectlyiden- Inthesecondanalysis,thecorrelationsbetweenthevarious
tifiedasmisaligned. Manysuchidentificationsarecorrect. We algorithmswerecalculatedforeachrecording. Figure3shows
combinedtheresultsforeachaudiofiletogiveanoverallscore theresultsforonerecording. Eachpairofalgorithmsscoreda
basedonthetotalnumberordurationofsuspectedofregions. pointwhentheyidentifiedapairofregionswithin5sofeach
The outputs of the above seven features were then correlated other. We compared the number of such pairs to the number
with the evaluations using a linear regression, via the “glm” ofaccidentalpairsthatwouldoccurifthealgorithm’soutputs
methodoftheRsoftwarepackage[15]. wereuncorrelatedwitheachother.
The lengths of the audio files varied, as did the amount There is a pattern of correlation between some of the
of speech, so we devised three ways to define a score, and detectorsinFigure3.Severalpairsofalgorithmswerestrongly
sincewehadnoclearcriteriontopickoneovertheother, we correlated: notably badlength and unexpected, badlength and
computedseparatelinearregressionswitheach. Thefirst,snd, long, and long and unexpected. These coincided 3.6 to 5.1
is the number of identified regions divided by the duration of times more often than chance, with statististical significances
theaudiofile;thesecond,snw,isthenumberofregionsdivided well beyond P < 0.001. Several of the pairs of algorithms
bythenumberofwordsintheaudiofile(asdeterminedfrom wereanticorrelated,notablyloudvs.short,loudvs.badlength,
theBNCtranscriptions);andthethird,sdd,isthetotalduration andloudvs.long. Thisisduetothedesignsofthealgorithms:
oftheregionsdividedbythedurationoftheaudiofile. specifically, loud triggers only on silences, while the others
Thedistributionsoftheevaluationvariableandthevarious triggeronlyonspeechsounds. Asaresult,theyneverpickthe
s variables were strongly non-Gaussian, with a maxima at same phoneme, and only occasionally pick phonemes within
one edge. Transforming the independent variables by raising 5 s of each other. The remaining pairs were either nearly
themtothepower0.3madethedistributionsubjectivelymore independent (loud & unexpected, badlength & short, long &
normal, as did squaring the evaluations. However, these short,short&unexpected)ordidnothaveenoughoccurrences
transforms were not more than partially successful, so we todrawanyreliableconclusion.
Duration-based measurements seem to perform best (i.e.
2Wedonotcomputearesultforphonemeswithδi = 0orwith short and badlength). One of the most useful indicators of a
δi≥1sonthegroundsthattheyareoutsidetherangeofvalidityofthe
durationmodel,andarealmostallsilences,anyway. However,these 4Notethatwith12regressions, weexpect3falsesignificancesat
phonesmaybeusedasneighboursinthecomputationofotherphones. theP < 0.05levelandoneattheP < 0.01level. Therefore,ofthe
3Inpractice,thesefirst12termsamounttoanormalisationofthe 9significancesreportedattheP < 0.05level(4forbadlength,3for
duration for changes in the local speech rate: the model adjusts the loud,2forlong),halfareprobablyspurious. Forsimplicity,wewill
phonedurationby36%oftheaveragechangeinnearbydurations ignorethemall.
Figure2:Longworderror.DisplayedasperFigure1,itshowswherethe“long”detectorhasidentifiedaregion(shadedintheFigure)
thatwasalignedasasingleword,“wasn’t”,butactuallyincludedseveralwords.
improbable
unexpected
hm loud
orit qluoinegt
g
Al short
badlength
0 500 1000 1500 2000 2500 3000
Time (s)
Figure3: Identifiedregions: eachrowshowswhereanalgorithmflaggedapotentialalignmenterror. Theaudiofilewasrated8for
overallalignmentsuccess.NB:theimprobabledetectorneverfiredonthisfile.
grossalignmenterrorwasalsothesimplest:theshortalgorithm UsingDurationandProsodicPhraseBreaks”,inProc.5thISCA
detectedasequenceofphonemeswhosedurationswereatthe SpeechSynthesisWorkshop,Pittsburgh,June,2004.
minimum allowed by their state topology (here, 3 states or [4] “TheBritishNationalCorpus”,http://www.natcorp.ox.
30milliseconds).Therelativelackofsuccessoftheimprobable ac.uk/
and unexpected algorithms was unexpected: good and bad [5] De Villiers, E., “Automatic alignment and error detection for
alignmentshavesimilardistributionsofLog(P)scores. phonetictranscriptionsintheAfricanspeechtechnologyproject
databases”, MScEng (Electrical and Electronic Engineering)
Thesis,UniversityofStellenbosch,2006.
4. Conclusions
[6] Barnard, E and Davel, M., “Automatic error detection in
alignmentsforspeechsynthesis”,17thAnnualSymposiumofthe
Wehaveshownthatautomatedtechniquescanusefullyidentify
PatternRecognitionAssociationofSouthAfrica(PRASA),Parys,
regionsofbadalignment.Theregionsidentifiedbysomeofour
SouthAfrica,29Nov-1Dec,pp.53-56,2006.
algorithmscorrelatewellwithhumanevaluationsoftheoverall
[7] Davel, M. and Barnard, E., “Bootstrapping Pronunciation
quality of the alignment. In general it appears that the most Dictionaries:PracticalIssues”,inProc.Interspeech2005,Lisboa,
reliable method for judging the quality is simply to consider Portugal,pp.1561–1564,2005.
thestatisticsofthesegmentdurations,eitheroverafixedtime [8] Huang, X., Acero, A., Adcock, J., Hon, H.-w., Goldsmith, J.,
window, or a linguistic unit (e.g. a word). This research Liu,J.,andPlumpe,M.,“Whistler: ATrainableText-To-Speech
should allow semi-automatic evaluation of the alignment of System”, Proc.ICSLP96, Philadelphia, PA,October1996, pp.
large speech corpora, which will be important for their future 2387-2390.
useinspeechresearch. [9] Das, R., Izak, J., Yuan, J., Liberman, M., “Forced Alignment
Under Adverse Conditions”, University of Pennsylvania, CIS
Dept.SeniorDesignProjectReport,2010.
5. Acknowledgements
[10] Coleman, J., Liberman, M., Kochanksi, G., Burnard, L., and
Yuan. J., “Mining a Year of Speech,” New Tools and Methods
We thank JISC (in the UK) and NSF (in the USA) for their
for Very-Large-Scale Phonetics Research Workshop, University
support of Mining a Year of Speech, under the Digging into
ofPennsylvania,January28-31,2011.
Data programme. This work is also partly supported by the
[11] Baghai-Ravary,L.,Kochanski,G.andColemanJ.,“Precisionof
UKESRC(awardsRES-062-23-2566,RES-062-23-1172,and
Phoneme Boundaries Derived using Hidden Markov Models”,
RES-062-23-1323). WethankJohn ColemanandRanjanSen Proc. Interspeech 2009. ISSN 1990-9772, Brighton, UK, pp.
forthedictionaries,andJohnColemanforhiscomments. 2879-2882,2009.
[12] Baghai-Ravary, L., Kochanski, G., andColeman, J., “Objective
6. References OptimisationofAutomaticSpeech-to-PhonemeAlignmentSys-
tems”, in Human Language Technologies as a Challenge for
[1] Sjo¨lander,K.,“AnHMM-basedsystemforautomaticsegmenta- ComputerScienceandLinguistics,Vetulani,Z.,(ed.),2009.
tionandalignmentofspeech”,Umea˚ University,Departmentof
[13] Yuan, J. and Liberman, M., “Speaker identification on the
PhilosophyandLinguistics,PHONUM9,pp.93-96,2003.
SCOTUScorpus”,inProc.Acoustics’08,pp.5687-5690,2008.
[2] M. Cooke, J. R. Hershey, S. J. Rennie, “Monaural speech [14] Carnegie Mellon University Pronouncing Dictionary, available
separation and recognition challenge”, Computer Speech and from http://www.speech.cs.cmu.edu/cgi-bin/
Language24(1),January,2010 cmudict
[15] “TheRReferenceIndex”,http://www.r-project.org/
[3] Toth,A.R.,“ForcedAlignmentforSpeechSynthesisDatabases
