Spoken Multimodal Human-Computer Dialogue in Mobile Environments

422 Pages·8.478 MB·English

by  Minker W., Bühler D., Dybkjrær L. (eds.)

Spoken Multimodal Human-Computer
Dialogue in Mobile Environments
Text, Speech and Language Technology
VOLUME 28
Series Editors
Nancy Ide, Vassar College, New York
Jean Véronis, Université de Provence and CNRS, France
Editorial Board
Harald Baayen, Max Planck Institute for Psycholinguistics, The Netherlands
Kenneth W. Church, AT & T Bell Labs, New Jersey, USA
Judith Klavans, Columbia University, New York, USA
David T. Barnard, University ofRegina, Canada
Dan Tufis, Romanian Academy of Sciences, Romania
Joaquim Llisterri, Universitat Autonma de Barcelona, Spain
Stig Johansson, University of Oslo, Norway
Joseph Mariani, LIMSI-CNRS, France
The titles published in this series are listed at the end of this volume.
Spoken Multimodal
Human-Computer Dialogue
in Mobile Environments
Edited by
W. Minker
University of Ulm, Germany
Dirk ühler
University of Ulm, Germany
and
LailaDybkjræ
University of Southern Denmark, Odense, Denmark
<£J Springer
A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN 1-4020-3074-6 (PB)
ISBN 1-4020-3073-8 (HB)
ISBN 1-4020-3075-4 (e-book)
Published by Springer,
P.O. Box 17, 3300 AA Dordrecht, The Netherlands.
Sold and distributed in North, Central and South America
by Springer,
101 Philip Drive, Norwell, MA 02061, U.S.A.
In all other countries, sold and distributed
by Springer,
P.O. Box 322, 3300 AH Dordrecht, The Netherlands.
Printed on acid-free paper
All Rights Reserved
© 2005 Springer
No part of this work may be reproduced, stored in a retrieval system, or transmitted
in any form or by any means, electronic, mechanical, photocopying, microfilming, recording
or otherwise, without written permission from the Publisher, with the exception
of any material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work.
Printed in the Netherlands
Contents
Preface xi
Contributing Authors xiii
Introduction xxi
Part I Issues in Multimodal Spoken Dialogue Systems and Components
1
Multimodal Dialogue Systems
Alexander I. Rudnicky
1. Introduction 3
2. Varieties of Multimodal Dialogue 4
3. Detecting Intentional User Inputs 6
4. Modes and Modalities 7
5. History and Context 7
6. Domain Reasoning 8
7. Output Planning 9
8. Dialogue Management 9
9. Conclusion 10
References 11
2
Speech Recognition Technology in Multimodal/Ubiquitous Com- 13
puting Environments
Sadaoki Furui
1. Ubiquitous/Wearable Computing Environment 13
2. State-of-the-Art Speech Recognition Technology 14
3. Ubiquitous Speech Recognition 16
4. Robust Speech Recognition 18
5. Conversational Systems for Information Access 21
6. Systems for Transcribing, Understanding and Summarising
Ubiquitous Speech Documents 24
7. Conclusion 32
References 33
vi SPOKEN MULTIMODAL HUMAN-COMPUTER DIALOGUE
3
A Robust Multimodal Speech Recognition Method using Optical 37
Flow Analysis
Satoshi Tamura, Koji Iwano, Sadaoki Furui
1. Introduction 38
2. Optical Flow Analysis 39
3. A Multimodal Speech Recognition System 40
4. Experiments for Noise-Added Data 43
5. Experiments for Real-World Data 48
6. Conclusion and Future Work 49
References 52
4
Feature Functions for Tree-Based Dialogue Course Management 55
Klaus Macherey, Hermann Ney
1. Introduction 55
2. Basic Dialogue Framework 56
3. Feature Functions 59
4. Computing Dialogue Costs 63
5. Selection of Dialogue State/Action Pairs 64
6. XML-based Data Structures 65
7. Usability in Mobile Environments 68
8. Results 69
9. Summary and Outlook 74
References 74
5
A Reasoning Component for Information-Seeking and Planning 77
Dialogues
Dirk Biihler, Wolfgang Minker
1. Introduction 77
2. State-of-the-Art in Problem Solving Dialogues 80
3. Reasoning Architecture 81
4. Application to Calendar Planning 85
5. Conclusion 88
References 90
6
A Model for Multimodal Dialogue System Output Applied to an 93
Animated Talking Head
Jonas Beskow, Jens Edlund, Magnus Nordstrand
1. Introduction 93
2. Specification 97
3. Interpretation 103
4. Realisation in an Animated Talking Head 105
5. Discussion and Future Work 109
References 111
Contents vii
Part II System Architecture and Example Implementations
7
Overview of System Architecture 117
Andreas Kellner
1. Introduction 117
2. Towards Personal Multimodal Conversational User Interface 118
3. System Architectures for Multimodal Dialogue Systems 122
4. Standardisation of Application Representation 126
5. Conclusion 129
References 130
XISL: A Modality-Independent MMI Description Language 133
Kouichi Katsurada, Hirobumi Yamada, Yusaku Nakamura, Satoshi
Kobayashi, Tsuneo Nitta
1. Introduction 133
2. XISL Execution System 134
3. Extensible Interaction Scenario Language 136
4. Three Types of Front-Ends and XISL Descriptions 140
5. XISL and Other Languages 146
6. Discussion 147
References 148
9
A Path to Multimodal Data Services for Telecommunications 149
Georg Niklfeld, Michael Pucher, Robert Finan, Wolfgang Eckhart
1. Introduction 149
2. Application Considerations, Technologies and Mobile Termi-
nals 150
3. Projects and Commercial Developments 154
4. Three Multimodal Demonstrators 156
5. Roadmap for Successful Versatile Interfaces in Telecommuni-
cations 161
6. Conclusion 163
References 164
10
Multimodal Spoken Dialogue with Wireless Devices 169
Roberto Pieraccini, Bob Carpenter, Eric Woudenberg, Sasha Caskey,
Stephen Springer, Jonathan Bloom, Michael Phillips
1. Introduction 169
2. Why Multimodal Wireless? 171
3. Walking Direction Application 172
4. Speech Technology for Multimodal Wireless 173
5. User Interface Issues 174
6. Multimodal Architecture Issues 179
7. Conclusion 182
References 184
viii SPOKEN MULTIMODAL HUMAN-COMPUTER DIALOGUE
11
The SmartKom Mobile Car Prototype System for Flexible Human- 185
Machine Communication
Dirk Biihler, Wolfgang Minker
1. Introduction 185
2. Related Work 186
3. SmartKom - Intuitive Human-Machine Interaction 189
4. Scenarios for Mobile Use 191
5. Demonstrator Architecture 193
6. Dialogue Design 194
7. Outlook - Towards Flexible Modality Control 197
8. Conclusion 199
References 200
12
LARRI: A Language-Based Maintenance and Repair Assistant 203
Dan Bohus, Alexander I. Rudnicky
1. Introduction 203
2. LARRI - System Description 204
3. LARRI - Hardware and Software Architecture 208
4. Experiments and Results 213
5. Conclusion 215
References 217
Part III Evaluation and Usability
13
Overview of Evaluation and Usability 221
Laila Dybkjeer, Niels Ole Bernsen, Wolfgang Minker
1. Introduction 221
2. State-of-the-Art 223
3. Empirical Generalisations 227
4. Frameworks 234
5. Multimodal SDSs Usability, Generalisations and Theory 236
6. Discussion and Outlook 238
References 241
14
Evaluating Dialogue Strategies in Multimodal Dialogue Systems 247
Steve Whittaker, Marilyn Walker
1. Introduction 247
2. Wizard-of-Oz Experiment 251
3. Overhearer Experiment 262
4. Discussion 266
References 267
Contents ix
15
Enhancing the Usability of Multimodal Virtual Co-drivers 269
Niels Ole Bernsen, Laila Dybkjtsr
1. Introduction 269
2. The VICO System 271
3. VICO Haptics - How and When to Make VICO Listen? 272
4. VICO Graphics - When might the Driver Look? 274
5. Who is Driving this Time? 278
6. Modelling the Driver 280
7. Conclusion and Future Work 284
References 285
16
Design, Implementation and Evaluation of the SENECA Spoken 287
Language Dialogue System
Wolfgang Minker, Udo Haiber, Paul Heisterkamp, Sven Scheible
1. Introduction 288
2. The SENECA SLDS 290
3. Evaluation of the SENECA SLDS Demonstrator 301
4. Conclusion 308
References 309
17
Segmenting Route Descriptions for Mobile Devices 311
Sabine Geldof, Robert Dale
1. Introduction 311
2. Structured Information Delivery 315
3. Techniques 315
4. Evaluation 322
5. Conclusion 326
References 327
18
Effects of Prolonged Use on the Usability of a Multimodal Form- 329
Filling Interface
Janienke Sturm, Bert Cranen, Jacques Terken, Use Bakx
1. Introduction 329
2. The Matis System 332
3. Methods 335
4. Results and Discussion 337
5. Conclusion 345
References 346
19
User Multitasking with Mobile Multimodal Systems 349
Anthony Jameson, Kerstin Klockner
1. The Challenge of Multitasking 350
2. Example System 354
3. Analyses of Single Tasks 354
4. Analyses of Task Combinations 359
5. Studies with Users 364
x SPOKEN MULTIMODAL HUMAN-COMPUTER DIALOGUE
6. The Central Issues Revisited 371
References 375
20
Speech Convergence with Animated Personas 379
Sharon Oviatt, Courtney Darves, Rachel Coulston, Matt Wesson
1. Introduction to Conversational Interfaces 379
2. Research Goals 382
3. Method 383
4. Results 387
5. Discussion 391
6. Conclusion 393
References 394
Index 399