Dance Data Cognition and Multimodal Communication

409 Pages·2022·27.297 MB·English

by  Carla Fernandes, Vito Evola, Cláudia Ribeiro

Routledge Advances in Theatre & Performance Studies
DANCE DATA, COGNITION,
AND MULTIMODAL
COMMUNICATION
Edited by
Carla Fernandes, Vito Evola, and Cláudia Ribeiro
Dance Data, Cognition, and
Multimodal Communication
Dance Data, Cognition, and Multimodal Communication is the result of a collaborative and
transdisciplinary effort towards a first definition of “dance data”, with its complexities and
contradictions, in a time where cognitive science is growing in parallel to the need of a
renewed awareness of the body’s agency in our manyfold interactions with the world.
It is a reflection on the observation of bodily movements in artistic settings, and one that views
human social interactions, multimodal communication, and cognitive processes through a
different lens—that of the close collaboration between performing artists, designers, and scholars.
This collection focuses simultaneously on methods and technologies for creating,
documenting, or representing dance data. The editors highlight works focusing on the
dancers’ embodied minds, including research using neural, cognitive, behavioural, and linguistic
data in the context of dance composition processes. Each chapter deals with dance data from an
interdisciplinary perspective, presenting theoretical and methodological discussions emerging
from empirical studies, as well as more experimental ones.
The book, which includes digital Support Material on the volume’s Routledge website,
will be of great interest to students and scholars in contemporary dance, neuro-cognitive
science, intangible cultural heritage, performing arts, cognitive linguistics, embodiment,
design, new media, and creativity studies.
Carla Fernandes is Principal Investigator, Head of ICNOVA’s Research Group on
Performance & Cognition, and Professor at FCSH – Universidade Nova de Lisboa, where she
directs the ERC-funded “BlackBox LAB Arts&Cognition.” Her current research focus is in the
intersection of Performing Arts and Cognitive Science, Multimodal Communication, Intangible
Cultural Heritage, and New Media. She is fascinated by the complexity of the human mind and
non-verbal behavior in creativity settings. She holds a PhD in Linguistics, supervises numerous
PhD and MA theses, and is an author in international peer-reviewed journals and books.
Vito Evola is currently a Researcher in Cognitive Linguistics and Multimodal
Communication at the Universidade Nova de Lisboa, having previously lectured and
conducted research at universities in Palermo, Berkeley, Cleveland, Aachen, and Geneva.
His research lies at the intersection of language, culture, and cognition, and analyses data
from both common and more specialised contexts, such as patient–doctor interactions,
psychotherapy and forensic interviews, religious discourse, and the performing arts.
Cláudia Ribeiro is a Postdoctoral Researcher at FCUL, in the research group LASIGE.
She obtained her PhD in Information Systems and Computer Engineering at the University
of Lisbon. Her research interests include machine learning, deep learning (focusing on
emotion recognition in multimedia content), and interactive systems.
Routledge Advances in Theatre & Performance
Studies
This series is our home for cutting-edge, upper-level scholarly studies and edited
collections. Considering theatre and performance alongside topics such as religion,
politics, gender, race, ecology, and the avant-garde, titles are characterized by dynamic
interventions into established subjects and innovative studies on emerging topics.
ASHÉ
Ritual Poetics in African Diasporic
Paul Cater Harrison, Michael D. Harris, and Pellom McDaniels
Dancehall In/Securities
Perspectives on Caribbean Expressive Life
Patricia Noxolo, ‘H’ Patten, and Sonjah Stanley Niaah
Circus and the Avant-Gardes
History, Imaginary, Innovation
Anna-Sophie Jürgens and Mirjam Hildbrand
Aesthetic Collectives
On the Nature of Collectivity in Cultural Performance
Andrew Wiskowski
Dance Data, Cognition, and Multimodal Communication
Carla Fernandes, Vito Evola, and Cláudia Ribeiro
Theatre and the Virtual
Genesis, Touch, Gesture
Zornitsa Dimitrova
Old Norse Poetry in Performance
Brian McMahon & Annemari Ferreira
For more information about this series, please visit: https://www.routledge.com/
Routledge-Advances-in-Theatre--Performance-Studies/book-series/RATPS
Dance Data, Cognition,
and Multimodal
Communication
Edited by Carla Fernandes,
Vito Evola, and Cláudia Ribeiro
First published 2023
by Routledge
4 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
and by Routledge
605 Third Avenue, New York, NY 10158
Routledge is an imprint of the Taylor & Francis Group, an Informa business
© 2023 selection and editorial matter, Carla Fernandes, Vito Evola, and
Cláudia Ribeiro: individual chapters, the contributors
The right of Carla Fernandes, Vito Evola, and Cláudia Ribeiro to be
identified as the authors of the editorial material, and of the authors for their
individual chapters, has been asserted in accordance with sections 77 and 78
of the Copyright, Designs and Patents Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced or
utilised in any form or by any electronic, mechanical, or other means, now
known or hereafter invented, including photocopying and recording, or in
any information storage or retrieval system, without permission in writing
from the publishers.
Trademark notice: Product or corporate names may be trademarks or
registered trademarks, and are used only for identification and explanation
without intent to infringe.
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Cataloguing-in-Publication Data
A catalog record has been requested for this book
Access the digital Support Material online: https://www.routledge.com/
9780367617455
ISBN: 978-0-367-61745-5 (hbk)
ISBN: 978-0-367-62116-2 (pbk)
ISBN: 978-1-003-10640-1 (ebk)
DOI: 10.4324/9781003106401
Typeset in Bembo
by MPS Limited, Dehradun
Contents
List of Illustrations viii
List of Contributors xv
Introduction 1
CARLA FERNANDES
PART I
Performance-as-Research: Dance data from the artists’
perspectives 13
1 Negotiating deliberate choice-making: Insights from
an interdisciplinary and multimodal encounter
during the making of a New Contemporary Dance 15
SYLVIA RIJMER
2 Dance | Data | Storytelling 38
STEPHAN JÜRGENS
3 Enabling multimodal interaction in mixed-abled
dance: Insights into creating highly accessible
teaching tools for inclusive cultural work 55
SUSANNE QUINTEN AND MIA SOPHIA BILITZA
PART II
Dance documentation and dance scores 69
4 Recording “Effect”: A case study in technical,
practical, and critical perspectives on dance data creation 71
DAVID RITTERSHAUS, ANTON KOCH, SCOTT DELAHUNTA, AND
FLORIAN JENETT
vi Contents
5 Digital-born artworks and interactive experience:
Documentation and archiving 89
PAULA VARANDA
6 Dance scoring and en-action as a creative tool for
dance documentation 99
BERTHA BERMÚDEZ-PASCUAL
7 Terpsicore – dance and performing arts archive 109
DANIEL TÉRCIO, CATARINA CANELAS, AND ANA LUÍSA VALDEIRA
PART III
Computational dance data: Between the real and the
virtual 117
8 Augmented seeing and sensing 119
ANGUS G. FORBES
9 Motion capture and the digital dance aesthetic:
Using inertial sensor motion tracking for devising
and producing contemporary dance performance 131
DANIEL STRUTT
10 Capturing and visualizing 3D dance data:
Challenges and lessons learnt 148
CLÁUDIA RIBEIRO, RAFAEL KUFFNER, AND CARLA FERNANDES
PART IV
The brain’s experience of dance 161
11 The embodied neuroaesthetics of watching dance 163
EMILY S. CROSS AND REBECCA SMITH
12 Dancing neurons: Common brain activity fMRI
analysis of the cerebral phenomena behind dance
perception 182
SOFIA AMARAL MARTINS AND FRANK POLLICK
13 “I see something, and I like it”: Unveiling a
choreographer’s decision-making process using
quantitative and qualitative methods 202
ANA RITA FONSECA, RODRIGO ABRIL-DE-ABREU, AND
CARLA FERNANDES
Contents vii
PART V
Dance expertise and cognition 221
14 Dance expertise, embodied cognition, and the body
in the brain 223
BETTINA BLÄSING
15 What makes dancers extraordinary? Insights from a
cognitive science perspective 244
CARLA FERNANDES, VITO EVOLA, AND JOANNA SKUBISZ
16 The role of dance experience, visual processing
strategies, and quantitative movement features in
recognition of emotion from whole-body
movements 274
REBECCA SMITH AND FRANK POLLICK
PART VI
Cognitive metaphor and gesture in dance and theatre 295
17 Unpeeling meaning: An analogy and metaphor
identification and analysis tool for modern and
post-modern dance, and beyond 297
VICKY J. FISHER
18 Understanding non-verbal metaphor: A cognitive
approach to metaphor in dance 320
LACEY OKONSKI, JULIE MADDEN, AND KAITLIN TOTHPAL
19 Study on hand movements accompanied during the
description of dance appreciation 333
ZI HYUN KIM AND HEDDA LAUSBERG
20 Reduction of gesticulation and information
patterning strategies in acted speech 346
GIORGINA CANTALINI AND MASSIMO MONEGLIA
21 Lines of experience: Towards a research method 363
MICHAEL O’CONNOR
Note about Funding 373
Author Index 374
Subject Index 382
Illustrations
Figures
1.1 A crude illustration of a Body Logic filtering process
(author’s artwork) 18
1.2 Diagram of the tree metaphor illustrating the dance-making
creative process outlined in this Arts and Science
collaboration (author’s artwork) 22
1.3 The dancers exploring Dodge Mode using Re-Dodge
Perspectives in a studio rehearsal; (a) contactless; (b)
contactful. Lisbon, October 12, 2018. Screenshots
provided by BlackBox 24
1.4 The dancers exploring “Simple Basic Scan” using their
fingertips as Prime Movers of a Con-sequential Movement
Pattern (which appears as their hands and arms are lifting).
Lisbon, October 12, 2018. Screenshot taken from video
provided by BlackBox 25
1.5 Dancers Teresa’s (left), Elson’s (centre), and Allan’s (right)
selections of graphic elements (above; details adapted from
Cardew, 1967: 181, 21, 22) and annotated Re-Scored
Dance Maps (below). Reprinted with permission 28
1.6 Dancer Allan fully immersed in his Treatise environment
during a VR studio rehearsal exploration. Lisbon, October
2, 2018. Author’s photograph 29
1.7 Left: Dancer Allan’s selected element (detail adapted from
Cardew, 1967: 20; reprinted with permission), later
imported into various VR environments for the dancer’s
full-body exploration. Right: Dancer’s perspective of the
graphic element in the “Daytime” (above) and “Dust
Particle” (below) VR environments. Screenshot taken
from VR video produced by BlackBox 30
Illustrations ix
2.1 Video still from the documentary film O Que Fazer Daqui
Para Tràs (What To Do With What Remains, 2017). The
animated infographic shows the entrances, exits, and
durations of the five dancers on the stage in a score layout 42
2.2 Video still from the 360° video recording of Fifteen Dancers
and Changeable Tempo (2016), section 16, on stage view.
The animated infographics provide information about the
dancer’s name, the choreographic material that is danced,
and the relations that are formed between the dancers 44
2.3 Video still from the VR installation on the creative process
of Sylvia Rijmer and her dancers. Upon entry of the VR
installation, the viewer sees three large buildings in the
shape of elements from the Cornelius Cardew score 45
2.4 Video still from the 360° video recording of Fifteen Dancers
and Changeable Tempo (2016), section 12, audience view.
The interactive score can be used to jump to any entrance
that one of the dancers on the list makes 48
2.5 Video still from the VR installation displaying an image
sculpture and four panels with stereoscopic photographs 50
3.1 Research process 58
3.2 Documentation template first version 60
3.3 Overview of participants of the artistic laboratory in Greece 61
3.4 Introduction to contact improvisation: Starting position
CREABILITY, © Un-Label 2019 62
3.5 Introduction to contact improvisation: Bringing the partner
to the floor CREABILITY, © Un-Label 2019 63
4.1 Annotated video in Piecemaker (Motion Bank Web Systems)
from the rehearsal documentation of Effect. Annotations were
created live in the studio in parallel to the video recording 74
4.2 The custom data player by Motion Bank allows one to load
and view a data set and to stream the data to any other
application via OSC protocol, making it easier for creative
coders to work with the data 79
4.3 3D visualisation of the motion capture data with a subtle way
of representing the dancer’s bodies as not to steer the
understanding in a certain direction through the design of
the avatars. Additional graphical elements illustrate movement
concepts, in this case the so-called “Blackholes” 82
4.4 Visualisations of pathways for the piece’s section “Figure 8” 84
9.1 The Rokoko Smartsuit Pro in action (rokoko.com) 132
9.2 Studio capture with Mavin Khoo in Valletta, Malta (image
by Prickimage) 139