Table Of ContentIntelligent Systems Reference Library 164
Laszlo Hunyadi
István Szekrényes Editors
The Temporal
Structure of
Multimodal
Communication
Theory, Methods and Applications
Intelligent Systems Reference Library
Volume 164
Series Editors
Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland
Lakhmi C. Jain, Faculty of Engineering and Information Technology, Centre for
Artificial Intelligence, University of Technology, Sydney, NSW, Australia;
Faculty of Science, Technology and Mathematics, University of Canberra,
Canberra, ACT, Australia;
KES International, Shoreham-by-Sea, UK;
Liverpool Hope University, Liverpool, UK
The aim of this series is to publish a Reference Library, including novel advances
and developments in all aspects of Intelligent Systems in an easily accessible and
well structured form. The series includes reference works, handbooks, compendia,
textbooks,well-structuredmonographs,dictionaries,andencyclopedias.Itcontains
well integrated knowledge and current information in the field of Intelligent
Systems. The series covers the theory, applications, and design methods of
IntelligentSystems.Virtuallyalldisciplinessuchasengineering,computerscience,
avionics, business, e-commerce, environment, healthcare, physics and life science
are included. The list of topics spans all the areas of modern intelligent systems
such as: Ambient intelligence, Computational intelligence, Social intelligence,
Computational neuroscience, Artificial life, Virtual society, Cognitive systems,
DNA and immunity-based systems, e-Learning and teaching, Human-centred
computing and Machine ethics, Intelligent control, Intelligent data analysis,
Knowledge-based paradigms, Knowledge management, Intelligent agents,
Intelligent decision making,Intelligent network security, Interactiveentertainment,
Learningparadigms,Recommendersystems,RoboticsandMechatronicsincluding
human-machine teaming, Self-organizing and adaptive systems, Soft computing
including Neural systems, Fuzzy systems, Evolutionary computing and the Fusion
of these paradigms, Perception and Vision, Web intelligence and Multimedia.
** Indexing: The books of this series are submitted to ISI Web of Science,
SCOPUS, DBLP and Springerlink.
More information about this series at http://www.springer.com/series/8578
á é
Laszlo Hunyadi Istv n Szekr nyes
(cid:129)
Editors
The Temporal Structure
of Multimodal
Communication
Theory, Methods and Applications
123
Editors
LaszloHunyadi István Szekrényes
Department ofGeneral andApplied Institute of Philosophy
Linguistics University of Debrecen
University of Debrecen Debrecen, Hungary
Debrecen, Hungary
ISSN 1868-4394 ISSN 1868-4408 (electronic)
Intelligent Systems Reference Library
ISBN978-3-030-22894-1 ISBN978-3-030-22895-8 (eBook)
https://doi.org/10.1007/978-3-030-22895-8
©SpringerNatureSwitzerlandAG2020
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
hereinorforanyerrorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregard
tojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
This book is dedicated to researchers and
developers wishing to deepen their
understanding of the complexities of
human-human communication as found in
multimodal interactions and to seek
inspiration to apply specific patterns of
behavior in building future smart systems at
the crossroads of social sciences and
engineering.
Preface
Themostwidelyacceptedclaimincontemporarymultimodalityresearchisthatthe
different channels of communication complement each other. Sharing a similar
view and further extending it, this book sets as its main goal to show that the
perceivedasymmetrybetweentheverbalandnonverbalmarkersofcommunication
can be reconciled by demonstrating that all these markers are part of a multimodal
temporal structure in which their local alignment is just one constituent in a tem-
porally distributed and hierarchically organized structure. By applying a variety of
methodologies, including but not restricted to T-pattern analysis, the latter specif-
icallydesignedforthestudyoftemporalpatternsinhumaninteractions,thepresent
volumeintendstooffernewinsightsinthetheoryofmultimodalcommunicationas
well as pragmatics in general and thus contribute to dialogue modeling both in
human and robotic applications.
The most researched challenge in applied multimodal communication is the
development of multimodal user interfaces which are intended to make
human-computerinteractionmorehuman-likeanduser-friendly.Forachievingthis
goal,multimodaluserinterfaces(robotsorvirtualagents)havetobecapableofthe
appropriate interpretation and generation of both verbal and nonverbal behavior
(e.g., prosody and gestures) controlled by their communication engine.
Research is highly interdisciplinary involving a number of fields, such as
computerandcognitivescience,psychology,linguistics,biology,androbotics.The
main research areas focus on the following questions:
Motion Intelligence: How can we integrate perception with action to allow
robots to move smartly and assist humans in unexpected, everyday environments?
Attentive Systems: What mechanisms enable artificial systems to understand
and actively focus on what is important, to ignore irrelevant detail and to share
attention with humans?
Situated Communication: How can we coherently coordinate language, per-
ception,andactionsothatcooperationbetweenhumansandartificialsystemstakes
place in natural efficiency?
MemoryandLearning:Whatmemoryarchitecturesenableasystemtoacquire,
store, and retrieve knowledge, and to improve its capabilities by learning?
vii
viii Preface
There is a wide range of applications from industry to home, from health to
security,eachhavingthemaingoalofuncovering,embedding,andimplementinga
rich spectrum of knowledge about human behavior and cognition for both under-
standing ourselves as humans in general, and the building of more effective
human-machine interfaces. This volume offers a broad view on the theory,
methodology,andpracticeofthestudyofmultimodalbehaviorthatcanbeapplied
to a number of future applications. The authors are representatives of fields of
research that can significantly contribute to the achievement of the above goals,
coming from specific areas of psychology, computer science, speech science,
pragmaticsandcomputationallinguistics.Thetopicspresentedinthebookinclude
an overview of the theory of pragmatics related to multimodal communication, a
methodology for the detection of structures in indirect observation, an experiment
on how the involvement of a direct observer in a dialogue can be measured, a
description of a methodology of discovering and applying paraverbal, kinetic
behavioral patterns in an educational environment, the description of multimodal
patterns of uncertainty in a dialogue, and how machine learning techniques can be
applied tothe automatic recognition of topic change ina conversation. Time plays
an important role throughout the book, demonstrating that behavior needs to be
captured as a function of time across all modalities.
Special features of the book:
1. It offers a comprehensive analysis of multimodal communication based on a
variety of corpora, including one of the most extensively annotated corpora of
dialogues (HuComTech).
2. It captures the temporal patterns of a wide range of behaviors discovered in
human-human communication.
3. It presents work from several areas of communication research with all the
benefits of interdisciplinarity.
Debrecen, Hungary Laszlo Hunyadi
January 2019 István Szekrényes
Contents
Part I Theoretical Overview of Multimodal Communication
1 Linguistic and Contextual Clues of Intentions and Perspectives
in Human Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Enikő Németh T.
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Models of Human Verbal Communication . . . . . . . . . . . . . . . . . . 5
1.2.1 The Basic Features of Human Verbal Communication . . . . 5
1.2.2 Code Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.3 Inferential Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.4 Ostensive-Inferential Model . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.5 The Generative Theoretical-Technological Model . . . . . . . 12
1.3 Perspectives, Intentions and Their Clues in Verbal
Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Part II Methods of Observation
2 The Teacher’s Body Communicates. Detection of Paraverbal
Behaviour Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Marta Castañer and Oleguer Camerino
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.1.1 Introjection, Extension and Projection of the Teaching
Discourse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.1.2 The Non-linearity of Human Movement . . . . . . . . . . . . . . 28
2.1.3 Paraverbal Communication and Body Language . . . . . . . . 28
2.1.4 From Kine to Gesture . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
ix
x Contents
2.2 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.2.1 Pattern Analysis and the Systemic Approach. . . . . . . . . . . 32
2.2.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.2.3 Instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.2.4 Materials and Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.4 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3 Is It Possible to Perform “Liquefying” Actions in Conversational
Analysis? The Detection of Structures in Indirect Observations. . . . 45
M. Teresa Anguera
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2 Towards an Optimal Methodology for Conversational
Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2.1 Systematic Observation . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2.2 Basic Decisions in the Analysis of Turn-Taking . . . . . . . . 49
3.3 Methodological Process: Transcription, Use of Observation
Instrument, Coding and Data-Quality Control. . . . . . . . . . . . . . . . 53
3.3.1 Transcription. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.3.2 Use of the Observation Instrument . . . . . . . . . . . . . . . . . . 54
3.3.3 Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.3.4 Data Quality Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4 T-Pattern Detection in Conversational Analysis . . . . . . . . . . . . . . 58
3.5 What Do “LIQUEFYING” Actions in Conversational Analysis
Involve?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4 Research Methods for Studying Daily Life: Experience
Sampling and a Multilevel Approach to Study Time and Mood
at Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Mariona Portell, Robin M. Hogarth and Anna Cuxart
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.2 The Challenge to Study Time and Mood at Work . . . . . . . . . . . . 71
4.3 The Role of Temporal Factors. . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.3.1 Diurnal Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.3.2 Day-of-the-Week Effects . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.3.3 Weather Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.3.4 Other Factors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.4 The Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.4.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.4.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.4.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
