Table Of ContentAdvances in Natural Multimodal Dialogue Systems
Text, Speech and Language Technology
VOLUME 30
Series Editors
Nancy Ide, Vassar College, New York
Jean Véronis, Université de Provence and CNRS, France
Editorial Board
Harald Baayen, Max Planck Institute for Psycholinguistics, The Netherlands
Kenneth W. Church, AT & T Bell Labs, New Jersey, USA
Judith Klavans, Columbia University, New York, USA
David T. Barnard, University of Regina, Canada
Dan Tufis, Romanian Academy of Sciences, Romania
Joaquim Llisterri, Universitat Autonòma de Barcelona, Spain
Stig Johansson, University of Oslo, Norway
Joseph Mariani, LIMSI-CNRS, France
The titles published in this series are listed at the end of this volume.
Advances in Natural
Multimodal Dialogue
Systems
Edited by
Jan C.J. van Kuppevelt
Waalre, The Netherlands
Laila Dybkjær
University of Southern Denmark, Odense, Denmark
and
Niels Ole Bernsen
University of Southern Denmark, Odense, Denmark
A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN-10 1-4020-3934-4(PB)
ISBN-13 978-1-4020-3934-8(PB)
ISBN-10 1-4020-3932-8(HB)
ISBN-13 978-1-4020-3032-4(HB)
ISBN-10 1-4020-3933-6 (e-book)
ISBN-13 978-1-4020-3933-1(e-book)
Published by Springer,
P.O. Box 17, 3300 AADordrecht,The Netherlands.
www.springer.com
Printed on acid-free paper
All Rights Reserved
©2005 Springer
No part of this work may be reproduced, stored in a retrieval system, or transmitted
in any form or by any means, electronic, mechanical, photocopying, microfilming, recording
or otherwise, without written permission from the Publisher, with the exception
of any material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work.
Printed in the Netherlands
Contents
Preface xi
1
NaturalandMultimodalInteractivityEngineering-DirectionsandNeeds 1
NielsOleBernsenandLailaDybkjaer
1. Introduction 1
2. ChapterPresentations 2
3. NMIEContributionsbytheIncludedChapters 7
4. MultimodalityandNaturalInteractivity 16
References 19
PartI MakingDialoguesMoreNatural: EmpiricalWorkandAppliedTheory
2
SocialDialoguewithEmbodiedConversationalAgents 23
TimothyBickmoreandJustineCassell
1. Introduction 23
2. EmbodiedConversationalAgents 25
3. SocialDialogue 29
4. RelatedWork 32
5. SocialDialogueinREA 36
6. AStudyComparingECASocialDialoguewithAudio-OnlySocial
Dialogue 40
7. Conclusion 48
References 49
3
AFirstExperimentinEngagementforHuman-RobotInteractioninHost- 55
ingActivities
CandaceL.SidnerandMyroslavaDzikovska
1. Introduction 55
2. HostingActivities 56
3. WhatisEngagement? 57
4. FirstExperimentinHosting: APointingRobot 59
5. MakingProgressonHostingBehaviours 62
6. EngagementforHuman-HumanInteraction 63
7. Computational Modelling of Human-Human Hosting and Engage-
ment 70
8. ANextGenerationMel 73
v
vi AdvancesinNaturalMultimodalDialogueSystems
9. Summary 74
References 74
PartII AnnotationandAnalysisofMultimodalData: SpeechandGesture
4
FORM 79
CraigH.Martell
1. Introduction 79
2. StructureofFORM 80
3. AnnotationGraphs 85
4. AnnotationExample 86
5. PreliminaryInter-AnnotatorAgreementResults 88
6. Conclusion: ApplicationstoHLTandHCI? 90
Appendix: OtherTools,SchemesandMethodsofGestureAnalysis 91
References 95
5
OntheRelationshipsamongSpeech,Gestures,andObjectManipulation 97
inVirtualEnvironments: InitialEvidence
AndreaCorradiniandPhilipR.Cohen
1. Introduction 97
2. Study 99
3. DataAnalysis 101
4. Results 103
5. Discussion 106
6. RelatedWork 106
7. FutureWork 108
8. Conclusions 108
Appendix: QuestionnaireMYSTIII-EXILE 110
References 111
6
AnalysingMultimodalCommunication 113
PatrickG.T.Healey,MarcusColmanandMikeThirlwell
1. Introduction 113
2. BreakdownandRepair 117
3. AnalysingCommunicativeCo-ordination 125
4. Discussion 126
References 127
7
DoOralMessagesHelpVisualSearch? 131
NoëlleCarbonellandSuzanneKieffer
1. ContextandMotivation 131
2. MethodologyandExperimentalSet-Up 134
3. Results: PresentationandDiscussion 141
4. Conclusion 153
References 154
Contents vii
8
GeometricandStatisticalApproachestoAudiovisualSegmentation 159
TrevorDarrell,JohnW.FisherIII,KevinW.Wilson,andMichaelR.Siracusa
1. Introduction 159
2. RelatedWork 160
3. MultimodalMultisensorDomain 162
4. Results 166
5. SingleMultimodalSensorDomain 167
6. Integration 175
References 178
PartIII AnimatedTalkingHeadsandEvaluation
9
ThePsychologyandTechnologyofTalkingHeads: ApplicationsinLan- 183
guageLearning
DominicW.Massaro
1. Introduction 183
2. FacialAnimationandVisibleSpeechSynthesis 184
3. SpeechScience 191
4. LanguageLearning 194
5. ResearchontheEducationalImpactofAnimatedTutors 197
6. Summary 210
References 211
10
EffectiveInteractionwithTalkingAnimatedAgentsinDialogueSystems 215
BjörnGranströmandDavidHouse
1. Introduction 215
2. TheKTHTalkingHead 217
3. EffectivenessinIntelligibilityandInformationPresentation 219
4. EffectivenessinInteraction 223
5. ExperimentalApplications 231
6. TheEffectiveAgentasaLanguageTutor 235
7. Experimentsand3DRecordingsfortheExpressiveAgent 237
References 239
11
ControllingtheGazeofConversationalAgents 245
DirkHeylen,IvovanEs,AntonNijholtandBetsyvanDijk
1. Introduction 245
2. FunctionsofGaze 248
3. TheExperiment 252
4. Discussion 258
5. Conclusion 260
References 260
viii AdvancesinNaturalMultimodalDialogueSystems
PartIV ArchitecturesandTechnologiesforAdvancedandAdaptiveMultimodal
DialogueSystems
12
MIND:AContext-BasedMultimodalInterpretationFrameworkinCon- 265
versationalSystems
JoyceY.Chai,ShimeiPanandMichelleX.Zhou
1. Introduction 265
2. RelatedWork 267
3. MINDOverview 267
4. ExampleScenario 268
5. Semantics-BasedRepresentation 270
6. Context-BasedMultimodalInterpretation 277
7. Discussion 282
References 283
13
AGeneralPurposeArchitectureforIntelligentTutoringSystems 287
BradyClark,OliverLemon,AlexanderGruenstein,ElizabethOwenBratt,
JohnFry,StanleyPeters,HeatherPon-Barry,KarlSchultz,
ZackThomsen-GrayandPucktadaTreeratpituk
1. Introduction 287
2. AnIntelligentTutoringSystemforDamageControl 288
3. AnArchitectureforMultimodalDialogueSystems 294
4. ActivityModels 295
5. DialogueManagementArchitecture 298
6. BenefitsofACIforIntelligentTutoringSystems 301
7. Conclusion 302
References 303
14
MIAMM–AMultimodalDialogueSystemusingHaptics 307
NorbertReithinger,DirkFedeler,AshwaniKumar,ChristophLauer,
ElsaPecourtandLaurentRomary
1. Introduction 308
2. HapticInteractioninaMultimodalDialogueSystem 309
3. VisualHapticInteraction–ConceptsinMIAMM 313
4. DialogueManagement 319
5. TheMultimodalInterfaceLanguage(MMIL) 326
6. Conclusion 331
References 331
15
AdaptiveHuman-ComputerDialogue 333
SorinDusanandJamesFlanagan
1. Introduction 333
2. OverviewofLanguageAcquisition 334
3. DialogueSystems 337
4. LanguageKnowledgeRepresentation 340
5. DialogueAdaptation 341
6. Experiments 347
Contents ix
7. Conclusion 351
References 353
16
MachineLearningApproachestoHumanDialogueModelling 355
YorickWilks,NickWebb,AndreaSetzer,MarkHeppleandRobertaCatizone
1. Introduction 355
2. ModalityIndependentDialogueManagement 357
3. LearningtoAnnotateUtterances 362
4. Futurework: DataDrivenDialogueDiscovery 366
5. Discussion 367
References 368
Index 371
Preface
The chapters in this book jointly contribute to what we shall call the field
of natural and multimodal interactive systems engineering. This is not yet a
well-established field of research and commercial development but, rather, an
emergingoneinallrespects. Itbringstogether,inaprocessthat,arguably,was
bound to happen, contributors from many different, and often far more estab-
lished, fields of research and industrial development. To mention but a few,
theseincludespeechtechnology,computergraphicsandcomputervision. The
field’srapidexpansionseemsdrivenbyasharedvisionofthepotentialofnew
interactivemodalitiesofinformationrepresentationandexchangeforradically
transforming the world of computer systems, networks, devices, applications,
etc. from the GUI (graphical user interface) paradigm into something which
will enable a far deeper and much more intuitive and natural integration of
computersystemsintopeople’sworkandlives.
Jointly, the chapters present a broad and detailed picture of where natural
and multimodal interactive systems engineering stands today. The book is
basedonselectedpresentationsmadeattheInternationalWorkshoponNatural,
Intelligent and Effective Interaction in Multimodal Dialogue Systems held in
Copenhagen, Denmark, in 2002 and sponsored by the European CLASS pro-
ject. CLASSwasinitiatedontherequestoftheEuropeanCommissionwiththe
purposeofsupportingandstimulatingcollaborationamongHumanLanguage
Technology(HLT)projectsaswellasbetweenHLTprojectsandrelevantpro-
jects outside Europe. The purpose of the workshop was to bring together re-
searchers from academia and industry to discuss innovative approaches and
challengesinnaturalandmultimodalinteractivesystemsengineering.
The Copenhagen 2002 CLASS workshop was not just a very worthwhile
event in an emerging field due to the general quality of the papers presented.
It was also largely representative of the state of the art in the field. Given the
increasinginterestinnaturalinteractivityandmultimodalityandtheexcellent
qualityoftheworkpresented,itwasfelttobetimelytopublishabookreflect-
ingrecentdevelopments. Sixteenhigh-qualitypapersfromtheworkshopwere
selectedforpublication. Content-wise,thechaptersinthisbookillustratemost
aspectsofnaturalandmultimodalinteractivesystemsengineering: applicable
xi
