Speech Repairs, Intonational Boundaries and Discourse Markers: Modeling Speakers’ Utterances in Spoken Dialog

284 Pages·0.829 MB·English

by  Heeman Peter Anthony.

Speech Repairs, Intonational Boundaries
and Discourse Markers:
Modeling Speakers’ Utterances
in Spoken Dialog
by
Peter Anthony Heeman
Submitted in Partial Fulfillment
of the
Requirements for the Degree
Doctor of Philosophy
Supervised by
Professor James F. Allen
Department of Computer Science
The College
Arts and Sciences
University of Rochester
Rochester, New York
1997

iii
Curriculum Vitae
Peter Heeman was born October 22, 1963, and muchto his dismayhis parents had
already moved away from Toronto. Instead he was born in London Ontario, where he
grew up on a strawberry farm. He attended the University of Waterloo where he re-
ceiveda Bachelors of Mathematics witha joint degree inPure Mathematics and Com-
puterScienceinthespringof1987.
After working two years for a software engineering company, which supposedly
used artificial intelligence techniques to automate COBOL and CICS programming,
Peter was ready for a change. What better way to wipe the slate clear than by going
tograduateschoolattheUniversityofToronto,butnotwithoutfirst spendingthesum-
mer in Europe. After spending two months in countries where he couldn’t speak the
language, Peter became fascinated by language, and so decided to give computational
linguisticsatry.
In the fall of 1989, Peter started his Masters degree in Computer Science at the
University of Toronto, supported by the National Science and Engineering Research
Council (NSERC) of Canada and by working as a consultant at ManuLife. Peter took
the introductory course in computational linguistics given by Professor Graeme Hirst,
wholaterbecamehisadvisor. InsearchingforaMastersthesistopic,GraemegotPeter
interested in Herbert Clark’s work on collaboratingin discourse. With the guidance of
Graeme and Visiting Professor Janyce Wiebe, Peter made a computational model of
Clark’s workoncollaboratinguponreferringexpressions.
iv
With the guidance of Professor Hirst, Peter decided to attend the University of
Rochester in the fall of 1991 to do his Ph.D. with Professor James Allen, with two
years of funding supplied by NSERC. The first few months were a bit difficult since
Peter was still finishing up his Masters thesis. But he did manage to graduate from
Torontointhat fallwitha MastersofScience.
Rochester was of course a major culture shock to Peter; but he survived. He even
survived the first year and a half in Rochester without an automobile, relying on a
bicycleto get him around Rochester and to and from school. LuckilyPeter livedclose
tohisfavoritebar,theAvenuePub,whichiswherehemetCharles onafateful evening
inthesummerafterhis firstyear.
As a sign of encouragement (or funding regulations), Peter received a Masters of
Science, againin Computer Science, from the Universityof Rochester in the springof
1993. ItwasaroundthistimethatJamesumlikegotPeterinterestedincomputationally
understandingdisfluencies,whichofcourseisamajorthemeinthisthesis.
Having had a taste of the fast pace of Toronto, one couldimage that five and a half
yearsinRochesterwouldtaketheirtoll. Luckilyinthefallof1996,Peterwasinvitedto
spend four months in Japan at ATR in the Interpreting Telecommunications Research
laboratory working with Dr. Lokem-Kim, an offer that he quickly accepted. Peter’s
secondchancetoescapeoccurredimmediatelyafterhisoraldefenseofthisthesis. This
time the location was in France, where he did a post-doc at CNET, France Te´le´com.
Although located far from Paris, it did give Peter a chance to become a true Canadian
by forcing him to improve his French. It was at CNET Lannion that final revisions to
thisthesiswerecompleted.
v
Acknowledgments
To begin with, I would like to thank my advisor, James Allen, for his support and
encouragementduringmystayattheUniversityofRochester. Iwouldalsoliketothank
the other members of my committee: Len Schubert and Michael Tanenhaus. Their
feedback helped shaped this thesis, especially Mike’s encouragement to use machine
learningtechniquestoavoidusingad-hocrulesthat happentofit thetrainingdata.
I also want to thank my co-advisors from my Masters degree at the University of
Toronto: Graeme Hirst and Janyce Weibe. Their involvement and encouragement did
notstoponceIhadleft Toronto.
I wish to thank the Trains group. I wish to thank the original Trains group, espe-
cially George Ferguson, Chung Hee Hwang, Marc Light, Massimo Poesio, and David
Traum. IalsowishtothankthecurrentTrainsgroup,especiallyGeorge,DonnaBryon,
MarkCore,EricRingger,AmonSeagullandTeresaSikorski. AspecialthankstoMark
andAmonforhelpingmeproofreadthisthesis.
I also wish to thank the other members of my entering class, especially Hannah
Blau, Ramesh Sarukkai and Ed Yampratoon. I also wish to thank everyone else in the
department for making it such a great place, especially Chris Brown, Polly Pook, the
administrativestaff—JillForster,PeggyFranz,PatMarshall,andPegMeeker—andthe
supportstaff—TimBecker, LiudBukys,Ray Frank,Brad MillerandJimRoche.
I would also like to thank Lin Li, Greg Mitchell, Mia Stern, Andrew Simchik, and
Eva Bero, who helped in annotating the Trains corpus over the last three years and
helpedinrefiningtheannotationschemes.
vi
I also wish to thank members of the research community for their insightful ques-
tionsandconversations,especiallyEllenBard,JohnDowding,JuliaHirschberg,Lynette
Hirschman, Robin Lickley, Marie Meteer, Mari Ostendorf, Liz Shriberg and Gregory
Ward.
I wish to thank Kyung-ho Loken-Kim for providing me with the opportunity to
work on this thesis at ATR in Japan. In addition to a welcome change in environment
(and being able to avoid several major snow storms), I had the opportunity to present
myworkthere,fromwhichIreceivedvaluablecomments,especiallyfromAlanBlack,
Nick Cambell, Laurie Fais, Andrew Hunt, Kyung-ho Loken-Kim, and Tsuyoushi Mo-
rimoto.
IalsowishtothankDavidSadekforprovidingmetheopportunitytoworkatCNET,
FranceTe´le´com,whereImadethefinalrevisionstothisthesis. Ialsowanttothankthe
many people at CNET who made my stay enjoyable and gave me valuable feedback.
I would especially like to thank Alain Cozannet, Geraldine Damnati, Alex Ferrieux,
Denis Jouvet, David Sadek, Jacque Simonin and Christel Sorin and the administrative
supportofJanineDenmat.
This material is based upon work supported by the NSF under grant IRI-9623665,
DARPA—RomeLaboratoryunderresearchcontractF30602-95-1-0025,ONR/DARPA
undergrantN00014-92-J-1512,andONRundergrantN0014-95-1-1088. Fundingwas
also received from the Natural Science and Engineering Research Council of Canada,
from the Interpreting Telecommunications Laboratory at ATR in Japan, and from the
Centre Nationald’EtudesdesTe´le´communications,France Te´le´com.
Finally,Iwishtothankthepeoplewhoaredearesttome. Iwishtothankmyparents
andsiblingswhohavealwaysbeenthereforme. IwishtothankmyfriendsinToronto
andelsewhere,especiallyGregandRandy,forlettingmeescapefromRochester. Ialso
wishtothankmydepartedfriendAndre´. Finally,IwishtothankCharlesBuckner,who
haspatientlyputupwithmewhileIhaveworkedawayonthisthesis,andaccompanied
meontheoccasionalescape awayfrom it.
vii
Abstract
Interactive spoken dialog provides many new challenges for natural language under-
standingsystems. Oneofthemostcriticalchallengesissimplydeterminingthespeaker’s
intended utterances: both segmentinga speaker’s turn into utterances and determining
theintendedwordsineachutterance. Evenassumingperfect wordrecognition,thelat-
terproblemiscomplicatedbytheoccurrence ofspeechrepairs,whichoccurwherethe
speakergoesbackandchanges(orrepeats)somethingshejustsaid. Thewordsthatare
replacedorrepeatedarenolongerpartoftheintendedutterance,andsoneedtobeiden-
tified. The two problems of segmenting the turn into utterances and resolving speech
repairs are strongly intertwined with a third problem: identifying discourse markers.
Lexical items that can function as discourse markers, such as “well” and “okay,” are
ambiguous as to whether they are introducing an utterance unit, signaling a speech
repair, or are simply part of the context of an utterance, as in “that’s okay.” Spoken
dialog systems need to address these three issues together and early on in the process-
ing stream. In fact, just as these three issues are closely intertwined with each other,
they are alsointertwinedwithidentifyingthesyntacticrole orpart-of-speech (POS) of
each word and the speech recognition problem of predicting the next word given the
previouswords.
In this thesis, we present a statistical language model for resolving these issues.
Rather than finding the best word interpretation for an acoustic signal, we redefine the
speechrecognitionproblemtosothatitalsoidentifiesthePOStags,discoursemarkers,
speech repairs and intonational phrase endings (a major cue in determining utterance
viii
units). Addingtheseextraelementstothespeechrecognitionproblemactuallyallowsit
tobetterpredictthewordsinvolved,sinceweareabletomakeuseofthepredictionsof
boundary tones, discourse markers and speech repairs to better account for what word
willoccurnext. Furthermore,wecantakeadvantageofacousticinformation,suchassi-
lenceinformation,whichtendstoco-occurwithspeechrepairsandintonationalphrase
endings, that current language models can only regard as noise in the acoustic signal.
The output of this language model is a much fuller account of the speaker’s turn, with
part-of-speech assigned to each word, intonation phrase endings and discourse mark-
ers identified, and speech repairs detected and corrected. In fact, the identification of
theintonationalphraseendings,discoursemarkers,andresolutionofthespeechrepairs
allows the speech recognizer to model the speaker’s utterances, rather than simplythe
wordsinvolved,andthusitcanreturnamoremeaningfulanalysisofthespeaker’s turn
forlaterprocessing.
ix
Table of Contents
Curriculum Vitae iii
Acknowledgments v
Abstract vii
ListofTables xiii
ListofFigures xv
1 Introduction 1
1.1 UtterancesinSpokenDialog . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.3 POSTaggingandSpeech Recognition . . . . . . . . . . . . . . . . . . 20
1.4 Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.5 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.6 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.7 OverviewoftheThesis . . . . . . . . . . . . . . . . . . . . . . . . . . 28
x
2 Related Work 31
2.1 StatisticalLanguageModeling . . . . . . . . . . . . . . . . . . . . . . 32
2.2 UtteranceUnitsandBoundaryTones . . . . . . . . . . . . . . . . . . . 47
2.3 SpeechRepairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.4 DiscourseMarkers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3 The Trains Corpus 61
3.1 DialogCollection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2 InitialTranscription . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.3 IntonationAnnotations . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.4 SpeechRepair Annotations . . . . . . . . . . . . . . . . . . . . . . . . 72
3.5 POSAnnotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
3.6 DiscourseMarkerAnnotations . . . . . . . . . . . . . . . . . . . . . . 97
4 POS-Based Language Model 99
4.1 RedefiningtheSpeech RecognitionProblem . . . . . . . . . . . . . . . 100
4.2 LearningtheProbabilities . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.3 Perplexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.4 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.5 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
5 Detecting Speech Repairs andBoundary Tones 151
5.1 SplittingtheUtterance Tag . . . . . . . . . . . . . . . . . . . . . . . . 156
5.2 RedefiningtheSpeech RecognitionProblem . . . . . . . . . . . . . . . 161
5.3 DiscontinuitiesintheContext . . . . . . . . . . . . . . . . . . . . . . 163
5.4 RepresentingtheContext . . . . . . . . . . . . . . . . . . . . . . . . . 166