Autonomic Nervous System Dynamics for Mood and Emotional-State Recognition: Significant Advances in Data Acquisition, Signal Processing and Classification

168 Pages·2014·3.16 MB·English

by  Gaetano Valenza

Series in BioEngineering
Gaetano Valenza
Enzo Pasquale Scilingo
Autonomic Nervous System
Dynamics for Mood
and Emotional-State
Recognition
Signifi cant Advances in Data
Acquisition, Signal Processing
and Classifi cation
Series in BioEngineering
Forfurthervolumes:
www.springer.com/series/10358
Gaetano Valenza (cid:2) Enzo Pasquale Scilingo
Autonomic Nervous System
Dynamics for Mood and
Emotional-State Recognition
Significant Advances in Data Acquisition,
Signal Processing and Classification
GaetanoValenza EnzoPasqualeScilingo
BioengineeringandRoboticsResearch BioengineeringandRoboticsResearch
Center“E.Piaggio” Center“E.Piaggio”
UniversityofPisa UniversityofPisa
Pisa,Italy Pisa,Italy
ISSN2196-8861 ISSN2196-887X(electronic)
SeriesinBioEngineering
ISBN978-3-319-02638-1 ISBN978-3-319-02639-8(eBook)
DOI10.1007/978-3-319-02639-8
SpringerChamHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2013951995
©SpringerInternationalPublishingSwitzerland2014
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub-
lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany
errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect
tothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Dedicatedtomybelovedparents, Francesco
and Carmen
G.V.
Dedicatedtomyloves, Laura andEmma
E.P.S.
Foreword
This book fervently presents a wide collection of methodological and applicative
studies related to the assessment of autonomic nervous system (ANS) dynamics
in healthy subjects and patients with mood disorders. Indeed, the high technical
contentmakesthebookquiteattractivetoanyoneinterestedinmathematics,applied
physics,electronics,statisticsand,mostespecially,insignalprocessing.
Firstly,theauthorsamplyreviewanddiscusseveryaspectthatconcernsemotions
andmoodrecognition,emphasizingexperimentalset-up,procedures,andinterpre-
tationoftheresults,andallowingphysiciansandclinicalpsychologiststotakefull
advantage of this monograph. Secondly, I am glad that the book emphasizes the
probabilisticapproachbasedonpoint-processmodels,whichIhavebeenstudying
in-depth during the last decade, in order to instantaneouslyassess ANS linear and
nonlinear dynamics. Here the authors present a novel extension of the point pro-
cessmathematicalframeworkascharacterizedthroughanalysisofheartbeatseries,
thus providing a wide range of novel ANS markers that can be used for model-
ing and classification. Remarkably, the book is also a unique contribution to the
study of ANS nonlinear dynamics. Thirdly, although the emphasis of this work is
on advanced signal processing techniques, it contains critical points regarding the
developmentofwearablesystemsforANSmonitoring.Iliketheauthors’philoso-
phy to “sensorize” everyday clothes such as gloves, t-shirts, and hats to make the
recordingprocessasnon-invasive,effectiveandcomfortableaspossible.
IknowtheauthorspersonallyandIdidnotexpectanythinglessfromthisbook.
Ithinkthattheycanbeconfidentthattherewillbemanygratefulreaderswhowill
havegainedabroaderperspectivehowthestudyofANSdynamicscanbeintegrated
within the fields of biomedical engineering for affective computing and psycho-
physiology. I do recommend the book for the active research scientists and PhD
studentsinterestedinsuchinterdisciplinaryapproaches.
HarvardMedicalSchool RiccardoBarbieri
Boston,MA,USA
August2013
vii
Preface
Thebookreportsonrecentadvancesinstudyingautonomicnervoussystem(ANS)
dynamicsfortheassessmentofmoodandemotionalstates.Wewillillustrateseveral
concepts,someofwhicharecurrentlysparseoverdifferentmanuscripts,inorderto
bring out a clear breakthrough in the field of affective computing, mood assess-
ment, biomedical engineering, biomedical signal processing, and data acquisition.
The aim is to describe some personalized methodologies able to characterize the
affectivestateofasubjectbymeansoftheanalysisofawidespectrumofperiph-
eralbiosignalssuchasHeartRateVariability,ElectrodermalResponse,Respiration
Activity,Eyegazeinformation.
These methodologies will be presented with applications on actual data gath-
eredfromhealthysubjectsaswellaspatientsaffectedbymooddisorders,although
the reported advances can also be applied to several other (clinical) fields. Start-
ingformapsychologicaldescriptionofhow-to-elicitanemotion(includingmodels
ofemotions,affectivestimulietc.),conceptswillmovetotheneuro-physiologyof
emotions, explaining the physiological bases of emotion recognition through non-
invasive monitoring of the ANS. Afterwards, advanced methodologies of biomed-
icalsignalprocessingwillbethoroughlydescribedpointingoutthecrucialroleof
ANSnonlineardynamics.Then,thebookwillemphasizeaprobabilisticframework
basedonpoint-processesabletoinstantaneouslyassesstheANScontrolonthelin-
ear and the nonlinear cardiovascular dynamics. Concerning the signal acquisition,
novelwearablemonitoringsystemswillbedescribedinfurthersectionsalongwith
experimental procedures on healthy subjects and patients with bipolar disorders.
The high technical content and all the proposed pioneering approaches make this
monographoflargeinterest.Severalprofessionalssuchasbiomedicalengineersas
well as physiologists, neuroscientists, etc. could benefit from the content of this
book.
Pisa,Italy GaetanoValenza
August2013 EnzoPasqualeScilingo
ix
Acknowledgements
Wewouldliketoacknowledgeallthepeoplewhocontributedtodataacquisitionand
analysis. In particular, we would like to express our deepest and sincere gratitude
toDr.AntonioLanatá,Dr.RiccardoBarbieri,Dr.LucaCiti,Dr.RitaParadiso,Dr.
Paolo Allegrini, Prof. Danilo De Rossi, Dr. Claudio Gentili, and Prof. Dr. Gilles
Bertschy.
xi
Contents
PartI IntroductoryRemarksandStateoftheArt
1 IntroductiontoAdvancesinAutonomicNervousSystemDynamics
forMoodandEmotional-StateRecognition . . . . . . . . . . . . . 3
2 EmotionsandMoodStates:Modeling,Elicitation,andClassification 9
2.1 ModelingEmotions . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 AutonomicNervousSystemCorrelatesofEmotions . . . . . . . 11
2.2.1 HeartRateVariability . . . . . . . . . . . . . . . . . . . 12
2.2.2 TheElectrodermalResponse . . . . . . . . . . . . . . . 12
2.2.3 InformationComingfromtheEyes:PupilSizeVariation
andEyeTracking . . . . . . . . . . . . . . . . . . . . . 13
2.2.4 Cardio-RespiratoryCoupling . . . . . . . . . . . . . . . 15
2.3 EmotionElicitation . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 AffectiveComputing:FromTheorytoEmotionRecognition . . . 17
2.5 EmotionsandMoodDisorders:TheBipolarDisorders . . . . . . 17
2.6 AutonomicNervousSystemasaNonlinearPhysiologicalSystem 21
PartII Methodology
3 Data Acquisition: Experimental Procedures and Wearable
MonitoringSystems . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1 ProceduresonHealthySubjects . . . . . . . . . . . . . . . . . . 25
3.1.1 RecruitmentofEligibleSubjects . . . . . . . . . . . . . 26
3.1.2 StimulusElicitation . . . . . . . . . . . . . . . . . . . . 26
3.2 ProceduresonBipolarPatients . . . . . . . . . . . . . . . . . . 28
3.2.1 RecruitmentofEligibleSubjectsandExperimental
Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.2 TheMoodModel . . . . . . . . . . . . . . . . . . . . . 32
3.3 PortableandNovelWearableSystemsforAutonomicNervous
SystemMonitoring . . . . . . . . . . . . . . . . . . . . . . . . 33
3.3.1 TheGloveSystem . . . . . . . . . . . . . . . . . . . . . 33
xiii
xiv Contents
3.3.2 ThePSYCHESystem . . . . . . . . . . . . . . . . . . . 38
3.3.3 HATCAM—WearableEyeGazeTrackingSystem . . . . 39
3.3.4 BIOPAC:SetofPhysiologicalSignalsandInstrumentation 43
4 AdvancedSignalProcessingandModelingforANSData . . . . . . 45
4.1 OverallMethodology . . . . . . . . . . . . . . . . . . . . . . . 46
4.2 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.2.1 MovementArtifactRemoval . . . . . . . . . . . . . . . 48
4.2.2 ElectrocardiogramandHeartRateVariability. . . . . . . 48
4.2.3 Respiration . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.2.4 ElectrodermalResponse . . . . . . . . . . . . . . . . . . 50
4.3 FeatureSets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.3.1 StandardFeatureSet . . . . . . . . . . . . . . . . . . . . 52
4.3.2 FeaturesfromHigherOrderSpectra. . . . . . . . . . . . 54
4.3.3 PupillometryandGazePoint . . . . . . . . . . . . . . . 57
4.3.4 NonlinearMethodsforFeatureExtraction . . . . . . . . 63
4.3.5 Cardio-RespiratorySynchronizationAnalysis . . . . . . 68
4.4 FeatureReductionStrategy . . . . . . . . . . . . . . . . . . . . 70
4.4.1 PrincipalComponentAnalysis . . . . . . . . . . . . . . 70
4.5 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.5.1 QuadraticDiscriminantClassifier . . . . . . . . . . . . . 71
4.5.2 k-NearestNeighborhood. . . . . . . . . . . . . . . . . . 73
4.5.3 Multi-layerPerceptron . . . . . . . . . . . . . . . . . . . 73
4.5.4 SupportVectorMachine . . . . . . . . . . . . . . . . . . 73
4.6 Point-ProcessTheoryandtheInstantaneousNonlinearDynamics 74
4.6.1 Point-ProcessNonlinearModeloftheHeartbeat . . . . . 77
4.6.2 EstimationoftheInput–OutputVolterraKernels . . . . . 79
4.6.3 QuantitativeTools:HighOrderSpectralAnalysis . . . . 80
PartIII Results
5 ExperimentalEvidencesonHealthySubjectsandBipolarPatients 85
5.1 ResultsfromtheHealthySubjectsStudy . . . . . . . . . . . . . 86
5.1.1 EffectiveArousalandValenceLevelsRecognition
ThroughAutonomicNervousSystemDynamics . . . . . 86
5.1.2 ApproximateEntropyandDominantLyapunovExponent
AnalysisonHeartRateVariability . . . . . . . . . . . . 88
5.1.3 Cardio-RespiratorySynchronizationAnalysis . . . . . . 90
5.1.4 UsingCardio-RespiratorySynchronizationInformation
forEmotionRecognition . . . . . . . . . . . . . . . . . 92
5.1.5 InstantaneousBispectralCharacterization
oftheAutonomicNervousSystemThrough
Point-ProcessNonlinearModels . . . . . . . . . . . . . 94
5.1.6 InstantaneousEmotionalAssessmentThroughNonlinear
Point-ProcessModels . . . . . . . . . . . . . . . . . . . 95
5.1.7 ElectrodermalResponseAnalysisandSensorizedGlove
Assessment . . . . . . . . . . . . . . . . . . . . . . . . 99