Table Of ContentDesign Strategies for Whole Body Interactive
Performance Systems
EvangelosLympouridis
DoctorofPhilosophy
CollegeofHumanitiesandSocialScience
EdinburghCollegeofArt
TheUniversityofEdinburgh
2012
Abstract
This practice-led research investigates a design framework within an artistic context for the
implementationofWholeBodyInteractive(WBI)performancesystemsthatemployreal-timemo-
tion capture technology. Following an Interaction Design perspective I engage in exploring the
requirements for composers, dancers, musicians and performers and their expectations, within
aseriesoftransdisciplinarycollaborativeartisticprojects. Integraltothisinvestigationisacom-
prehensive review and analysis of the progression of interactivity in fine art, music, dance and
performancepractices,presentedinthisthesis. AsIamparticularlyconcernedwiththeseamless
transferofthetacitskillsandtheimplicitknowledgeofnon-digitalartistsandpractitionerstoa
WBIperformancesetting,mypracticalexplorationsemergedinthecontextsofmusicimprovisa-
tion-Untitled#1,contemporarydance-Untitled#2,contemporarymusiccomposition-Hiroshima,
andtraditionaldance-Duende. AdoptingaHolisticDesignapproach,theexperienceandknow-
ledgegainedfrommyfirstpracticalexplorationsledtothedesignandimplementationofaWBI
prototyping software environment called EnActor, used in tandem with the Orient wireless in-
ertial motion capture system, developed by the Research Consortium in Speckled Computing,
at the University of Edinburgh. EnActor provides a simple and effective solution to the prob-
lem of linking physical actions to rich digital media responses and can serve as a blueprint for
thedevelopmentofotherWBIdesignsoftware,sinceithasoperatedsuccessfullyasaprototype,
addressing a wide range of WBI design briefs in various contexts. In this thesis I introduce the
roleoftheWBIdesignerasaspecialistinteractiondesignerabletoconceptualiseWBIscenarios
and implement complete systems that operate within various levels of body sensing and con-
trol. I also propose the development of WBI systems that are autonomous and unsupervised,
andIexplorevariouscompositionalconceptsandmappingsthatareimplementedasautomatic,
semi-automatic or manual modules and ultimately arranged into layers and to series of blocks
thatrepresentcompletecompositions. Followingtheunderstandingofinteractivityasaproperty
betweensystems,IidentifythedesignofthreebasictypesofWBIperformancesystemsthatdif-
fer in how a user engages with them: methodical, empirical and dialectic. Overall this research
aimstofacilitatedesignersandartistsinterestedintheuseofreal-timemotioncapturesystemsin
dance,music,theatreandperformanceartapplications.
Declaration
Ideclarethatthisthesiswascomposedbymyself,thattheworkcontainedhereinismyown
exceptwhereexplicitlystatedotherwiseinthetext,andthatthisworkhasnotbeensubmittedfor
anyotherdegreeorprofessionalqualificationexceptasspecified.
(EvangelosLympouridis)
Acknowledgements
Iwouldliketothankmysupervisors,Dr. MartinParkerandD.K.Arvind,fortheirguidance
and support; the Research Consortium in Speckled Computing at the University of Edinburgh,
especiallyAlexYoung,PaulMcEvanandMartinLing;theArtsandHumanitiesResearchCouncil
(AHRC)fortheirfulldoctoralawardtowardstheaccomplishmentofthisthesis(www.ahrc.ac.uk);
andtheInspacegalleryandMarkDanielsforhostingmyprojectsandpresentations.
Iparticularlywanttothankalltheartistsandresearchersthatparticipatedinthecollaborative
performances and workshops: Beatriz Cantinho, Mariza Dima, Shiori Usui, Gesa Piper, Aileen
Orr,NunoTorres,RicardoJacinto,AnaGouveia,MadalenaXavier.
I am also grateful to the following people for their useful insight and advice: Mariza Dima,
Owen Green, Mike Cullen, Dr. Soterios Papachilleos, Prof. Nigel Osborne, Prof. Michael Ed-
wards,AdrianFreed,AndrewW.Schmeder,TheCIRCLEresearchgroup,NikosChatzinikolaou,
Marientina Gotsis, Kurosh ValaNejad, Nearchos Panos, Dr. Natasha Karantinou, Dr. Kostas
Rekleitis,PabloHerrera,ChristosMichalakos,NikosTsormanandHarrisPapachristopoulos.
I am entirely thankful to my wife Natalie Lesly for proofreading and making suggestions
throughoutthewritingofthisthesiswithpatienceandextensivecare. Thisworkisdedicatedto
her,myparentsandallthosewhosupportedmyeffortsovertheyears.
VangelisLympouridis
Thisworkhasbeendevelopedthroughafulldoctoralawardbythe
ArtsandHumanitiesResearchCouncil
www.ahrc.ac.uk
Contents
1 Introduction 7
2 ArtisticExplorationsofInteractivity&RelatedTechnologies 10
2.1 InteractivityandTechnologyintheArtisticcontext . . . . . . . . . . . . . . . . . . 11
2.1.1 ViewsofInteractivityinFineArt . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.2 ViewsofInteractivityinDanceandPerformanceArts . . . . . . . . . . . . . 14
2.1.3 ViewsofInteractivityinMusicandComposition . . . . . . . . . . . . . . . 21
2.2 Interactiondesigntoolsandtechnologies . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2.1 TechnologiesinMotionCapture . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2.2 MappingStrategiesandInteractionDesignSoftwarePlatforms . . . . . . . 38
2.3 FinalThoughtsandConclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3 AHolisticDesignapproachinWholeBodyInteraction 44
3.1 The(I)inHCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2 Typesofinteractivesystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.3 WholeBodyInteraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.4 AHolisticApproachinInteractionDesign. . . . . . . . . . . . . . . . . . . . . . . . 51
3.4.1 HolisticandExperientialDesign . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.4.2 HolisticCollaborativeDesignModels . . . . . . . . . . . . . . . . . . . . . . 53
3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4 InvestigationsinMotionandSound 57
4.1 Untitled#1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 Untitled#2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3 DigitalSurfacesworkshop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.4 WBIDesignoutcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5 EnActor: ABlueprintforaPrototypeWBIDesignSystem 93
5.1 EnActor’sDesignLayout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.1.1 MotiontrackingandMotionAnalysisLayers . . . . . . . . . . . . . . . . . . 95
5.1.2 TheCoreModules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.1.3 KineticInteractions: TheFundamentalMechanism . . . . . . . . . . . . . . 96
5.1.4 SpatialInteractions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.1.5 WholeBodyMapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.1.6 ThePresetManager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.2 TheExtensionModules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.2.1 CompositionModules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.2.2 CompositionalBlocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.2.3 Digitalmusicalinstruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.2.4 SpeakersSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6 FinalProjects 113
6.1 Hiroshima-ExploringCompositioninWBIwithEnActor . . . . . . . . . . . . . . 114
6.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.1.2 CompositionalElements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.1.3 AutomaticCompositionalModules . . . . . . . . . . . . . . . . . . . . . . . 119
6.1.4 Semi-AutomaticCompositionalModules . . . . . . . . . . . . . . . . . . . . 121
6.1.5 ManualCompositionalModules . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.1.6 AdaptationoftheCompositionandtheNotionofPlay . . . . . . . . . . . . 127
6.2 Duende-ExploringthetransferoftraditionaldancerstoaWBIsetting . . . . . . . 129
6.2.1 Researchphase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.2.2 Designandimplementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.2.3 AgencyandResponsiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7 Summary&Futuredevelopments 136
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7.2 StatementoftheProblem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7.3 HolisticDesign . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.4 MotionCaptureSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.5 WholeBodyInteraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.6 WBISystems,SketchesandScenarios . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7.7 CompositionalModules,LayersandBlocks . . . . . . . . . . . . . . . . . . . . . . . 140
7.8 EnActor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.9 GeneralWBISystemDesignNotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.10 FinalThoughtsandFutureDevelopments . . . . . . . . . . . . . . . . . . . . . . . . 144
A Appendix 147
References 171
1 Introduction
Duringthepastdecades,choreographers,musicians,composersandperformers,incollaboration
withtechnologists,engagedinexploringtheutilisationofnewdigitaltechnologiesforextending
traditional and contemporary artistic practices and examining new forms of expression on the
performancestage. Theseearlyexplorationswereevolvinginparallelwiththeadvancesofmo-
tionsensingtechnologies,andalthoughtheyprimarilyshapedourunderstandingofinteractivity
intheperformanceartcontext,theyimposednumerousunnecessaryrestrictionsandnewalien-
atedprocessesonthedevelopmentofsuchwork.
In most cases this was a consequence of the interfaces that are commonly employed in in-
teractiveperformances, ascamerasandsensorsonlycaptureandpresentabstractdatafromthe
body,notthehumanbodyitself. Thisdataislinkedmoretothenatureofthesensingmechanism
thantheuser’sphysicalbodyandofferslimitedcontrolandoccasionallychaoticresponsestothe
intentions of the performer and the collaborating artists. As more advanced technological solu-
tions are developed in the field of motion capture (mocap), the ability to digitally represent the
whole body in time and space has improved, resulting in advancements in the design of Whole
BodyInteractions(WBI).
Insteadofdevelopinganewinterfaceoranintelligentinteractivesoftwaresystemforliveper-
formance,Iproposetoworkwiththeactualdigitalrepresentationofthebodyandaskthefollow-
ing questions: What is the future of interactive performances when we can precisely transcribe
the physical body in its digital replica in real-time? What are the requirements of performers,
musicians,choreographersandcomposersinthiscontextandhowcanwedesignsimpletools,to
directlyandcreativelyassistthem?
The main observation that set my initial perspective for the investigation of the above ques-
tionsconcernsanabsence,sofar,ofadesignapproachtotheexistingtechnology-basedperform-
ance explorations. This has left the artists to directly try to relate their artistic concepts with
varioustechnologieswithoutaninterpreter,aspecialistinteractiondesigner,whoseexpertisecan
offer important insight and a design perspective to the development process of interactive art
performancesandsystems. Astheartisticprocessesaremoreandmoreoccupiedbyalgorithmic
and technological interventions, non-technological practitioners are often excluded from parti-
cipatingintheseexcitingexplorations. Additionally,whentraditionalartistsattempttotransfer
their skills to a digital interactive platform, their knowledge and tacit skills become isolated as
theyarerequiredtooperatewithinanewtechnologicalframeworkthathasnotbeendesignedto
meettheirparticularrequirements.
Therefore, I began this practice-led research from an interaction design perspective with the
intention to involve non-digital performance art practitioners throughout my explorations and
practicalwork. Myearlydesignstudieswereinthecontextofimprovisationandcontemporary
danceandthenImovedontoexplorethedesignofWBIsystemsinthecontextofcontemporary
musicandtraditionaldance. TowardstheendofmypracticalworkIhaveappliedthedesignof
thesesystemsintheexplorationofnewhybridformsofperformanceartthatincludenotionsof
musiccomposition,theatreandimprovisation. WiththeoutcomesofmyresearchprojectsIaim
tofacilitatedesignersandartistsinterestedintheuseofinteractivesystemsthatemployreal-time
motioncapturetechniquesindance,music,theatreandperformanceartcontexts.
Soundwastheunifyingforceinallofmyprojects,servingastheprimaryfeedbackmedium.
When I first started designing WBI experiences, I developed systems by following two central
7
metaphors: designing a virtual musician for a dancer or a virtual instrument for a musician.
These two approaches differ in the amount of control the performer has over the system’s re-
sponses. Musiciansexpectaninteractivesystemtoprovideimmediateandrelativelypredictable
sound on demand, in order to compose particular sonic experiences. In contrast, dancers are
primarilyfocusedonthebody,itspresenceandmovementdynamicsintimeandspace,asthey
aretraditionallydancingtothesound.
Iwasfortunatetoemployinmypracticalexplorationsoneofthemostadvancedwirelessmo-
tioncapturesystemstodate, developedattheResearchConsortiuminSpeckledComputing, at
theUniversityofEdinburgh.1 Thisinnovativetechnologicalmotioncapturesolution, discussed
furtherinChaptertwo, isapromisingtechnicalinnovationfortrackingbodymovementonthe
performancestagewithreliability, transparencyandmobility. Becausethenoveltyfactorofany
newtechnologyistemporary, especiallyinourrapidlydevelopingdigitalera, myaimthrough-
outthediscussionandpresentationofthisthesisistogobeyondthetechnicalmeansthatassisted
theseexplorationsandextractmeaningful,fundamentaldesignconceptsthatcanbeappliedirre-
spectiveofthemotioncapturetechnologyutilised. Myobjectiveistodemystifytheunderlying
complexityandsharetheextractedfindingsfrommyendeavourswithrespecttotherelatedthe-
oriesandinreferencetowiderexistingpractise.
ThisthesisadoptsaholisticapproachfortheinvestigationofWBIdesignandaimstolayout
atheoreticalandtechnicalframeworkforthedevelopmentofadiverserangeofartisticworksin
thecontextofbody-driveninteractiveperformances. Theholisticdesignframeworkasproposed
byWoo(2007)describesthreemainphasesinthedesignprocess: research,designandinnovation,
that are further described in the third chapter. The introduction of this model in my practical
workenabledafruitfulcollaborativeenvironmentforexploringthecomplexitiesofthistransdis-
ciplinary research area. It encouraged the investigation of multiple perspectives and a mutual
exchangeofexpertisethroughestablishedcollaborationswithfellowresearchersinperformance
arts,choreography,musiccompositionandinformatics.
Duringthefirststage, theresearchphase, Iworkedinthecontextofimprovisation, contem-
porarydanceandcomposition. Threepracticalstudiesintheformofresidenciesandworkshops,
assisted my explorations to define the key notions for designing interactive performances and
the practical requirements of choreographers, dancers, musicians and composers. These collab-
orativeprojectstookplacebetweenSeptember2007andSeptember2009andarepresentedand
discussedinthefourthchapter.
These first practical studies raised valuable questions and elicited knowledge that informed
the design phase of my research, which took place between September 2009 and January 2010.
MyeffortsconcernedtherealisationofaprototypeWBIinteractiondesignsystem,calledEnActor,
thatIwillextensivelydiscussinthefifthchapter.
Finally, the innovation phase of this research proved EnActor’s ability to assist the explora-
tion of diverse interactive performance concepts and raised questions about the future of WBI
performances. Fromrapidprototypingtoextensiveexplorationsincomposition,theselastprac-
tical studies indicated the system’s potential and usability. These key projects were developed
between January 2010 to April 2010, and were linked to completely diverse performance dis-
ciplines. The findings and the development process of these projects are discussed in the fifth
1ThreeprototypeversionsoftheOrientwirelessinertialmotioncapturesystemhavebeendevelopedattheUniversity
ofEdinburghbetween2007-2010bytheResearchConsortiuminSpeckledComputing.Formoreinformationpleasecheck
theattachedpaperintheAppendix:Lympouridisetal.(2009)
8
chapter, which presents the prototype interactive dance system Duende, that emerged from the
traditionaldancecontextofFlamenco;andHiroshima,anexperimental,hybridperformancepiece
that incorporates notions of play, music composition and theatre. An additional project in the
contextofYogacanbefoundintheAppendix.
Developedformusicalperformances,dance,theatreoranyotherbody-driveninteractiveap-
plication,theaimofEnActoristofacilitatetheuseofthewholebodyandthedynamicsofmove-
ment to realise complete artistic interactive media scenarios and fast prototypes seen as WBI
sketches.2 In order to achieve these goals, I developed technical skills and expertise in the use
ofinertialsensormotioncapturesystems,interactiondesign,programming,mathematics,wire-
lesssensornetworksandubiquitouscomputing. However, itismyintentiontometaboliseand
encapsulate this knowledge in abstractions, metaphors and formalities that aim to provide and
communicatethepowerofthesenewtoolsinamoreaccessibleway. Duringthediscussionofmy
thesis, Iwilltrytokeepabalancebetweenthebasictechnicalknowledgeneededfortheappre-
ciationoftheproposedtechnologiesandtheinspiringnotionsofbody,spaceandtimethatform
theessenceofperformancearts.
2IdefineaWBIsketchasaprototypeofabody-basedinteractiondesignconcept
9
2 Artistic Explorations of Interactivity & Related Technologies
The context of this research stands at the crossroad of fine art, dance, performance art, and
music composition with systems design and technology. Theoretical discussions and practical
explorationsintheseareashaveenabledthedevelopmentofadiverseandhighlyvaluablebody
ofartisticwork,whereinteractivityisseenasyetanotherpropertyofexperiencingart.
Inthisfirstchapter,Iintendtopresentabroadoverviewofthenotionofinteractivityasseen
acrosstheseinterlacedartisticpractices. Moreover,Iintendtocultivateaholisticappreciationfor
thetermanddiscusstheworkofpioneerswhoenvisioneddigitaltechnologyasanexcitingnew
waytoexpandtheboundariesoftheirartisticpractice.
Contemporary explorations of interactivity in these artistic contexts have evolved with the
developmentofspecifictechnologiesthatarecapableoftranscribingthehumanbodyintodigital
data. Thishasallowedartistsandtechnologiststocaptureandanalysemeaningfuldataconcern-
ing the time, space and dynamics of the body’s movement, in order to synthesise audio-visual
interactiveresponses.
After reviewing notable explorations in interactivity, I will provide an overview and assess
variousinteractiondesigntoolsandtechnologiesthathavebeenutilised. Theseincludecamera-
based(Optical)andsensor-based(Inertial)motioncapturesystems,aswellastoolsformapping
movementtodigitalinteractionsthroughvarioussoftwaredesignplatforms.
In addition to providing an overview of the ideas, performances, and technologies that in-
formed my own work, this chapter can offer guidance to those who are entering this emerging
andexcitingareaofresearchandartisticpractice.
10
