Recognition of Intensive Valence and Arousal Affective States via Facial Electromyographic Activity

14 Pages·2016·0.62 MB·English

RESEARCHARTICLE
Recognition of Intensive Valence and Arousal
Affective States via Facial Electromyographic
Activity in Young and Senior Adults
Jun-WenTan1☯*,AdrianoO.Andrade2☯,HangLi1,SteffenWalter3,DavidHrabal3,
StefanieRukavina3,KerstinLimbrecht-Ecklundt3,HolgerHoffman3,HaraldC.Traue3
1 CollegeofTeacherEducation,LishuiUniversity,Lishui,P.R.China,2 BiomedicalEngineeringLaboratory,
FacultyofElectricalEngineering,FederalUniversityofUberlândia,Uberlândia,Brazil,3 EmotionLab,
SectionMedicalPsychology,UniversityofUlm,Ulm,Germany
☯Theseauthorscontributedequallytothiswork.
*[email protected]
Abstract
OPENACCESS
Citation:TanJ-W,AndradeAO,LiH,WalterS, Background
HrabalD,RukavinaS,etal.(2016)Recognitionof
Researchsuggeststhatinteractionbetweenhumansanddigitalenvironmentscharacter-
IntensiveValenceandArousalAffectiveStatesvia
FacialElectromyographicActivityinYoungand izesaformofcompanionshipinadditiontotechnicalconvenience.Tothiseffect,humans
SeniorAdults.PLoSONE11(1):e0146691. haveattemptedtodesigncomputersystemsabletodemonstrablyempathizewiththe
doi:10.1371/journal.pone.0146691
humanaffectiveexperience.Facialelectromyography(EMG)isonesuchtechnique
Editor:LeontiosHadjileontiadis,AristotleUniversity enablingmachinestoaccesstohumanaffectivestates.Numerousstudieshaveinvesti-
ofThessaloniki,GREECE
gatedtheeffectsofvalenceemotionsonfacialEMGactivitycapturedoverthecorrugator
Received:April19,2015 supercilii(frowningmuscle)andzygomaticusmajor(smilingmuscle).Thearousalemotion,
Accepted:December21,2015 specifically,hasnotreceivedmuchresearchattention,however.Inthepresentstudy,we
soughttoidentifyintensivevalenceandarousalaffectivestatesviafacialEMGactivity.
Published:January13,2016
Copyright:©2016Tanetal.Thisisanopenaccess
Methods
articledistributedunderthetermsoftheCreative
CommonsAttributionLicense,whichpermits
Tenblocksofaffectivepictureswereseparatedintofivecategories:neutralvalence/low
unrestricteduse,distribution,andreproductioninany
arousal(0VLA),positivevalence/higharousal(PVHA),negativevalence/higharousal
medium,providedtheoriginalauthorandsourceare
credited. (NVHA),positivevalence/lowarousal(PVLA),andnegativevalence/lowarousal(NVLA),
andtheabilityofeachtoelicitcorrespondingvalenceandarousalaffectivestateswas
DataAvailabilityStatement:Allrelevantdataare
withinthepaperanditsSupportingInformationfiles. investigatedatlength.Onehundredandthirteenparticipantsweresubjectedtothesesti-
muliandprovidedfacialEMG.Asetof16featuresbasedontheamplitude,frequency,
Funding:Thisresearchwassupportedbygrants
fromtheTransregionalCollaborativeResearch predictability,andvariabilityofsignalswasdefinedandclassifiedusingasupportvector
Center(DFG-ProjectSFBTRR62)“Companion- machine(SVM).
TechnologyforCognitiveTechnicalSystem”funded
bytheGermanResearchFoundation(DFG)and
Results
doctoralscholarshipsbytheChinaScholarship
Council(CSC)forJun-WenTanandHangLi.The
Weobservedhighlyaccurateclassificationratesbasedonthecombinedcorrugatorand
authorsthanktheBrazilianGovernmentforthe
zygomaticusEMG,rangingfrom75.69%to100.00%forthebaselineandfiveaffective
financialsupport.InparticulartheagenciesFAPEMIG
(ResearchFoundationofthestateofMinasGerais), states(0VLA,PVHA,PVLA,NVHA,andNVLA)inallindividuals.Thereweresignificant
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 1/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
CNPq(NationalCouncilofTechnologicaland differencesinclassificationrateaccuracybetweenseniorandyoungadults,buttherewas
ScientificDevelopment)andCAPESFoundationfor nosignificantdifferencebetweenfemaleandmaleparticipants.
theCoordinationandImprovementofHigherLevelor
EducationPersonnel).Thefundershadnorolein
Conclusion
studydesign,datacollectionandanalysis,decisionto
publish,orpreparationofthemanuscript.
Ourresearchprovidesrobustevidencesforrecognitionofintensivevalenceandarousal
CompetingInterests:Theauthorshavedeclared affectivestatesinyoungandsenioradults.Thesefindingscontributetothesuccessful
thatnocompetinginterestsexist. futureapplicationoffacialEMGforidentifyinguseraffectivestatesinhumanmachineinter-
action(HMI)orcompanionroboticsystems(CRS).
Introduction
Alongsidetherapidandextensivedevelopmentofinteractivedevicessuchastabletsandsmart-
phones,theinteractionbetweenhumansandtheirdigitalenvironmenthasbecomenotonlya
technicalactivity,butalsoaformofempathiccompanionship;theliteraturecharacterizesthis
relationshipashumancomputerinteraction(HCI)orgeneralizedhumanmachineinteraction
(HMI).Thistypeofrelationshipprovidesnotonlypassivefunctionalities,butalso,ideally,
functionsbasedonperceptionoftheuser’simplicitcurrentneeds,responses,preferences,
coherence,andintention[1–2].Thistypeofcompanionship-by-designsystemshouldbeable
tocompletelyandindividuallyadapttotheuser,andreflecttheuser’ssituationandemotional
states;inotherwords,thesystemshouldhavetheabilitytobeempathetic[3–5].
Toachieveanidealtechnicalcompanionshipsystem,theuser’saffectiveexperienceisavital
consideration;accurateandappropriaterecognitionoftheuser’semotionalstatesisparamount
tothesuccessofthistypeoftechnology.Inhuman-humaninteraction(HHI),individualsrec-
ognizeemotionalstatesusinginnatesensorsforfacialexpression,gestures,eyecontact,lan-
guage,speedortoneofspeaking,andothernaturalindicators[6].Acomputer,ofcourse,is
unabletoperceiveemotionalstatesbydefinition.Therearenumeroussignalsrelevanttoemo-
tionalresponsesthatare,though,measurablebydevicessuchascameras,microphones,and
sensors[7].Forexample,smallelectrodesattachedtotheskincanaccuratelydetectpsychobio-
logicalchangesindicativeofthehumanemotionalexperience,gainingaccesstohumanemo-
tionalstates.Onesuchpsychophysiologicalsensor,thefacialEMG,representsarobustand
effectivemethodofrecognizinghumanaffectivestatesinHMI[8–9].
Affectiveresponsesinvolvesubjectiveexperience,centralandperipheralnervoussystem
changes,andbehaviors(e.g.,facialexpressions,gestures,andvocalcharacteristics)[10].Facial
expressions,specifically,areinnateanduntrainedreactionstoaffectivestates[11]whichenable
ustorecognizeandcommunicateemotionstransientlyasweinteractwithotherpeople.Affec-
tivescienceresearchhasstruggledtoquantifiablymeasuretheaffectivestatesofhumanbeings,
however[10].Therearetwoprimaryissuesrelatedtothisproblem(and,assuch,tothepresent
study’sprimaryobjectives).First,thereisnostandardizedmodelforevaluatingorinterpreting
emotionsoraffects[12–13],however,asMaussandRobinson[10]suggested,“measuresof
affectiveresponsesseemtobestructuredalongdimensionsratherthandiscreteemotions”.Sec-
ond,duetothesizeable(andrapidlygrowing)proportionofsenioradultsinmanysocieties,
particularlyinWesterncountries,companiontechnologyishighlydesirablebecauseofashort-
ageofqualifiedhealthcarepersonnel[9].Ameta-analyticstudyreviewingemotionsandaging
suggestedthatseniorsareworsethanyoungeradultsinrecognizingemotionalstates,andthat
thereisageneraldecliningtrendinemotionrecognitionwithage[14],thoughtheresultsof
thisstudymayhavenotbeenfullyvalid.Agingdoesindeedplayanessentialroleindetecting
affectiveresponses,however.
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 2/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
Thetridimensionaltheoryofemotion,whichiscommonlyappliedinstudiesonthissubject,
evaluatesaffectivestatesaccordingtovalence,arousal,anddominance(VAD).Thisapproach
canbedatedbacktoWundt[15].“Valence”describesaffectivestatesfromhighlynegative
(unpleasant)tohighlypositive(pleasant);“arousal”measurestheintensityofaffectivestates
rangingfromhighlycalmtohighlyexcitedoralert;and“dominance”representsthefeelingof
beingcontrolledorinfluencedbyexternalstimuli[16–17].Lang,Rice,andSternbachassumed
thatemotioniscomprisedofthesethreedimensions[18];andresearchhasshownthatdomi-
nanceishighlycorrelatedwiththedimensionofvalence[16].Astheproponentsofthistheory
havesuggested,thetridimensionalmodelofemotioncanbereducedtotwoorthogonaldimen-
sions[19–21]:valenceandarousal,inwhichallemotionscanbeclassified.Weadoptedthe
two-dimensionaltheoryofaffectiveexperienceinthisstudy.
Anumberofresearchershaveinvestigatedtheeffectsofthesoledimensionofvalenceon
facialEMG.TheirfindingsdemonstratedthatfacialEMGcapturedovercorrugatorsupercilii
(frowningmuscle),whichisassociatedwithnegativeemotionalexpressions,andzygomaticus
major(smilingmuscle),whichisrelatedtopositiveemotionalexpressions[22–23],candiffer-
entiatevalenceemotionsandtheirintensities[24].Inthesestudies,corrugatorEMGamplitude
increasedinresponsetonegativeaffectivestimulianddecreasedwithpositiveaffectivestimuli
comparedtoneutralstimuli,whereaszygomaticusEMGamplitudeincreasedduringpositive
stimuli[25–29].Inaddition,zygomaticusEMGamplitudewasnotshowntodiscriminateneu-
tralandnegativeemotions,thus,corrugatorandzygomaticusEMGactivitiescanbeconsidered
indicatorsofnegativeandpositiveaffectivestates,respectively[25,28].
Todate,thedimensionofarousalhasnotreceivedmuchattentioninfacialEMGemotion
recognitionstudies.Verylimitedevidencefrompreviousstudieshasshown,though,thatzygo-
maticusEMGdifferentiatespositivevalence/lowarousalaffectivestatesfrompositivevalence/
higharousalaffectiveconditions.Intheparticipantsinthesestudies,thezygomaticusEMG
activitywaslowerwhenviewingpositivevalence/lowarousalaffectiveimagesthanwhenview-
ingpositivevalence/higharousalones[30],i.e.,thehigh-arousalaffectivepictureselicited
highercorrugatorandzygomaticusEMGactivitiesthanlow-arousalones[31].Corrugatorand
zygomaticusEMGcouldnotindependentlydifferentiatetheaffectivestatesinthearousal
dimension[28],whichmaymodulatefacialEMGresponseswhenparticipantsareconfronted
byvisualstimuli.Therefore,takingboththedimensionsofvalenceandarousalsimultaneously
intoaccountisarguablynecessaryfordetectingtheircompoundeffectsonfacialEMG
technology.
Researchhasalsoshownthatincreasingageiscorrelatedwithageneraldeclineinphysio-
logicalfunctions[32–33],andmanypreviousstudieshaveindicatedlowerphysiological
responsesinsenioradultsthaninyoungadultswhenexposedtoaffectivestimuli[34].Along
withthedeclineinphysiologicalfunctions,increasedpositivewell-being,affectcontrol,and
affectregulationhavebeenobservedinseniorcomparedtoyoungadults[35–37].Influenced
bythe“positivityeffect”,seniorstypicallyhighlyfavorpositivestimuli,whereasnegativesti-
muliaremorerecognizedbyyoungadults[38].Conversely,senioradultsincertainsituations
experiencegreateraffectiveresponsesthanyoungadultsdo,althoughevidencesuggestsless
affectivereactionsforsenioradultsthanyoungadultsinmostcontexts[39].Researchershave
claimedthattherearenosignificantagevariationsineithercorrugatororzygomaticusEMGin
relationtovalenceaffectivestates[40],butresearchershaveidentifiedloweroverallcorrugator
EMGactivityinresponsetoaffectivepicturesinsenioradults,regardlessofvalence[41].
AnotherstudyfoundthatincreasingageisassociatedwithdecreasedcorrugatorEMGampli-
tudeinresponsetoneutralstimuli[42].EvidenceforagedifferencesinfacialEMGonintensive
valenceandarousalaffectivestates,however,isscarceintheliterature.
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 3/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
Inbrief,theprimarygoalofthepresentstudywastoassessthepotentiallydistincteffectsof
intensivevalenceandarousalaffectivestatesonfacialEMGdetectedoverthecorrugatorsuper-
ciliiandzygomaticusmajorofstudyparticipants.Thesecondaryobjectivewastoinvestigate
agedifferencesintheEMGactivitybetweenseniorandyoungadultsindifferentaffective
states,andthesamebygender.Weemployedacasestudyfollowedbyclassificationmethods
andstatisticalanalysistoachievethesegoals.
MaterialsandMethods
Participants
Seventyyoungadults(from20to40yearsofage,mean=24.57,SD=4.37)and43senioradults
(from52to77yearsofage,mean=64.30,SD=7.16)participatedinthisstudy.Thedatafrom
twosubjectsintheyounggroupandthreeintheseniorgroupwereexcludedfromsubsequent
analysisduetotechnicalproblems(i.e.,movementsorartifacts.)Thefinalsamplesizewas108
(69females,39males).Allparticipantswerehealthyandhadnormalvisionorcorrectednor-
malvision.Theexperimentwasdesignedandimplementedaccordingtotheethicalguidelines
oftheUniversityofUlmandwasapprovedbytheuniversity’sEthicalCommittee(number:
245/08-UBB/se).
Stimuli
Toelicitintensiveandsustainedaffectivestates,whicharenoteasilyinducedbyastaticsingle
imageinlaboratorysituations,thestimuliconsistedof10pictureblocks.Ineachblock,10
affectiveimageswithsimilarratingscoresonvalenceandarousalwerecombined.Thus,atotal
of100pictureschosenfromtheinternationalaffectivepicturesystem(IAPS)[43]andUlmpic-
tures[44],whicharewell-designed,standardized,andgenerallyemployedforelicitingemo-
tionalstateswiththreedimensions(VAD)andareoftenutilizedinstudiesonaffectivescience
oraffectivecomputing.Thecontentofthepicturesrangedfromdailyexperiences(e.g.,house-
holdfurniture)intendedtoelicitneutralandcalmemotions,toextremeencounters(e.g.,severe
humaninjuries,)thatinducehighlynegativeandarousalemotions,ortoeroticimages
intendedtoproducehighlypositiveandarousalemotions.Tenpictureblocksweredivided
intofivecategories(i.e.,twopictureblockswithsimilarvalence-arousalrespondedtooneof
fiveaffectivestates)accordingtothestandardizedratingscores:1)0VLA,neutralvalence
(4.96±0.25)andlowarousal(2.75±0.64);2)NVLA,negativevalence(3.74±0.43)andlow
arousal(3.63±0.51);3)NVHA,negativevalence(2.20±0.64)andhigharousal(6.66±0.57);
4)PVLA,positivevalence(7.58±0.39)andlowarousal(3.26±0.47);and5)PVHA,positive
valence(7.00±0.57)andhigharousal(6.50±0.51).Fig1showsthelocationofthesefivecate-
goriesofaffectivestatesinthevalence-arousalspace.
Procedure
Participantswereseatedinacomfortablerecliningchairinasound-attenuatedroomofthe
EmotionLab,Ulm,afterbeingintroducedtotheexperimentandsigninganinformedconsent
form.EMGsensorswereattachedtotheparticipants’respectivefacialmuscles,afterwardspar-
ticipantswereaskedtorelax,keepstable,andpayattentiontotheimageblocksfortheduration
oftheexperiment.
Tenaffectiveimageblockswerepresentedrandomlyona17-inchmonitor.Eachblock,car-
ryingthesameprobabilitywithoutanyrepetition,waspresentedfor20s,duringwhicheachof
the10pictureswasshownfor2scontinuously.Therewasafixedpauseof20sbetweenblocks
(Fig2).FacialEMGactivitywasrecordedthroughouttheexperiment.
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 4/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
Fig1.Locationofaffectivestimuliinthetwo-dimensionalspace.
doi:10.1371/journal.pone.0146691.g001
EMGDataAcquisition
ANeXus-32physiologicalmeasurementsystem(NeXus-32,MindMedia,Roermond-Herten,
Netherlands)runningonadesktopcomputerwasusedfortheacquisitionoffacialEMG.The
softwarepackageBiobserveSpectator(version2.4.0.5,BIOBSERVEGmbH,Bonn,Germany)
wasusedtorecordthetriggerandpsychophysiologicaldata.FacialEMGsignalswerecaptured
withbipolarminiaturesilver/silverchloride(Ag/AgCl)skinelectrodes4mmindiameterwith
gel-filledattachspaces.Bipolarelectrodeswereplacedontheparticipants’leftcorrugator
superciliiandzygomaticusmajormuscleregions(seeFig3,EMGsignalacquisition)according
totheguidelinesforEMGplacementrecommendedbyFridlundandCacioppo[45].Facial
EMGsignalswererecordedatasamplingrateof512Hz.
EMGSignalProcessingandDataReduction
AsshowninFig3,rawfacialEMGsignalswerefilteredofflinebya20–250Hzband-passBut-
terworthfilter(order=4)toexcludemotion-relatedcomponents,andanadaptivefilterwas
appliedtoremovethe50Hzpowerlineinterference[46].Inordertoclassifythestimulated
affectivestatesusingfacialEMG,furtherprocessingmethodsincludingtheempiricalmode
decomposition(EMD)techniqueandtheHilbertSpectrum(HS)wereemployed.Datafrom
Fig2.Presentationofpictureblocks.
doi:10.1371/journal.pone.0146691.g002
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 5/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
Fig3.DiagramofEMGsignalacquisition,processing,datareduction,andclassification.
doi:10.1371/journal.pone.0146691.g003
single(corrugatororzygomaticus)andcombined(corrugatorandzygomaticus)site(s)were
submittedtoclassification.Alltheprocessing,analyses,andmachinelearningwereconducted
usingtheMATLABsoftwarepackage(versionR2009aandR2015a,MathworksInc.,Natick,
MA,USA).
DataClassification
Inordertoverifythepossibilityofdiscriminatingdistinctaffectivestates(0VLA,PVHA,
NVHA,PVLA,NVLA)andalsothebaseline(formingasix-classclassificationproblem,in
effect),thefollowingsequenceofstepsfordataprocessingwasapplied:
1. Featureextractionfromparticularregionsofinterestofthesignals.Aproblemwehad
wasdefiningthelength(inseconds)oftheregionofinterest,asthereisnospecificrule
forchoosingthislength.Weassesseddistinctempiricalcombinationsasdescribedin
Table1,wheresimilaranddistinctperiodsforbaselineandevokedemotionswere
adoptedbasedontheassumptionthatevokedemotioncouldinduceearly(<2s)and/or
late(>=2s)changesintheobservedtimeseries.
Table1. Periodoftimeadoptedforfeatureextractionfrombaselineandevokedemotionalperiods.
Label Windowlength(inseconds)for Windowlength(inseconds)for Description
thebaselineperiod evokedemotionalperiod
1–1 1s 1s Featureswereextractedfromthefirstsecondforboththebaseline
andevokedemotionalperiods
1–2 1s 2s Featureswereextractedfromthefirstsecondforthebaselineand
thefirsttwosecondsforevokedemotionalperiods
1-10-f 1s 10s Featureswereextractedfromthefirstsecondforthebaselineand
thefirsttensecondsforevokedemotionalperiods
1-10-l 1s 10s Featureswereextractedfromthefirstsecondforthebaselineand
thelasttensecondsforevokedemotionalperiods
2–2 2s 2s Featureswereextractedfromthefirsttwosecondsforboththe
baselineandevokedemotionalperiods
2-10-f 2s 10s Featureswereextractedfromthefirsttwosecondsforthebaseline
andthefirsttensecondsforevokedemotionalperiods
2-10- 2s 10s Featureswereextractedfromthefirsttwosecondsforthebaseline
1 andthelasttensecondsforevokedemotionalperiods
doi:10.1371/journal.pone.0146691.t001
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 6/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
Intotal,16featureswereestimatedfromeachregionofinterest,yieldinga16-Dfeaturevec-
torthatwasproperlylabeledaccordingtothepossibleclasses(0VLA,PVHA,NVHA,PVLA,
NVLA,andbaseline).Becausetwomuscles(corrugatorandzygomaticus)wereexaminedin
thisstudy,theresultingfeaturevectorhad16(fromcorrugatororzygomaticus)or32(from
corrugatorandzygomaticus)features.Thesefeatureswereemployedaccordingtotheliterature
[47].Theaimoffeatureextractionwastocapturechangesinthesignalsrelatedtotheirampli-
tude,frequency,predictability,andvariability.Table2presentsasummaryoftheemployed
features,andacomprehensivedescriptionofthefeaturesisprovidedinapreviouslypublished
study[48].
2. Featuresextractedfromallsubjectswereorganizedinatableinwhicheachcolumnrep-
resentedafeatureorvariable.Priortoclassification,eachfeaturewasstandardizedtohave
zeromeanandunitvariance.Classificationwasperformedwiththemachinelearning
toolboxavailableinMatlab2015Ra(Mathworks).Thesupportvectormachine(SVM)
classifierwasusedwithaGaussiankernel(i.e,templateSVM['KernelFunction','gaussian']
intheMatlabenvironment.)ThefitcecocmethodavailableinMatlab,whichfitsmulticlass
modelsforsupportvectormachinesorotherclassifiers,wasemployed,and10-foldcross-
validationwasusedforclassifierassessment.
Table2. Featuredescriptions.
Feature Feature Description
group
Amplitude Mav mav=mav(signal)
Mavfd mavfd=mavfd(signal)
Mavsd mavsd=mavsd(signal)
Peak peak=max(signal);index(max(signal))
Rms rms=rms(signal)
Frequency Zc Calculatedbycomparingeachpointofthesignalwiththenext;ifthereisacrossingbyzerothenitisaccounted.
Fmed Toobtainthemedianfrequency,findthevalueofthefrequencythatbisectstheareabelowtheXwaveform.
Fmode ThisfastFouriertransformationequationisvalidforthisandthefollowingfrequencyfeatures:.
X
XXðkÞ¼ Nj¼1xðjÞoNðj(cid:2)1Þðk(cid:2)1Þ,whereoN¼eð(cid:2)2NpiÞ.Tofindthemode,findthemaximumvalueofX.
Fmean Nk¼XF1FTXðkÞ:fðkÞ
XðkÞ
Cf Thecentralfrequencyissimplythemeanofthefrequenciesthatdelimitthebandwidth:cf¼fh(cid:2)fl.
h i 2
Predictability Fuzzyentropy Saenðm;s;dÞ¼ln ComðsÞ ,wheremisthewindowsize,sisthesimilaritystandardanddisthesignal.Itiscalculated
Comþ1ðsÞ
inaverysimilarwaytotheSampleEntropy.Theonlysimilaritybetweenthegroupsiscomputedbymeansofa
Fuzzymembershipfunction.
Approximate ForatemporalserieswithNsamples{u(i):1(cid:3)i(cid:3)N}givenm,createvectorsXmforeachXm as
j N(cid:2)mþ1
entropy Xm¼fuðiÞ;uðiþ1Þ;uðiþm(cid:2)1Þg;i¼1;...;N(cid:2)mþ1,wheremisthenumberofpointstogrouptogetherforthe
j
comparison.Foreachk(cid:3)N−m+1groups,doCmðrÞwhichisthenumberoftimesthegroupshaddistanceless
k
NX(cid:2)mþ1
lnCmðrÞ
j
thantolerancer.ThencomputethevalueφmasφmðrÞ¼ i¼1 .TheApproximatedEntropyis:ApEn(m,r)=
limN!1[φm(r)−φm+1(r)]. N(cid:2)mþ1
Variability Var XN
ðxi(cid:2)(cid:2)xÞ2
s2¼ i¼1
pffiffiffiffiffi N(cid:2)1
Std S¼ s2
Range R=MAX(U)−MIN(U)
Intrange SI¼Q3(cid:2)Q1
2
doi:10.1371/journal.pone.0146691.t002
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 7/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
3. Weestimatedthesuccessrateofeachclass,definedasthenumberofcorrectlyclassified
patternsdividedbythetotalnumberpatternsoftheclass.Theentireclassifier-training
procedureandperformanceassessmentwasrepeated10timestoestimatethemeanand
thestandarddeviationoftherateofsuccess.
Results
ClassificationResults
TheclassificationresultsarepresentedfromTables3–7,whereresultsarepresentedforeach
specificconditionlistedinTable1.
Table3describesthemean(M)andstandarddeviation(SD)ofclassificationaccuracies
(ten-foldcross-validation)offiveaffectivestatesandthebaseline,basedonthecombinedcor-
rugatorandzygomaticusEMGdata.Theclassificationaccuracyreached100%forthebaseline
inallsevenconditions,wherethefiveaffectivestatesrangedfrom87.04%to88.47%in1–1,
from87.92%to90.65%in1–2,from84.81%to88.75%in1-10-f,from84.58%to88.75%in1-
10-l,from75.69%to80.69%in2–2,from85.19%to89.07%in2-10-f,andfrom85.05%to
88.47%in2-10-l.
Table4presentstheMandSDclassificationaccuraciesofbaselineandthefiveaffective
statesbasedonthecombinedcorrugatorandzygomaticusEMGdataaccordingtoage.Forthe
baseline,theclassificationaccuracyhit100%forallconditionsexceptin2–2inyounggroups;
thefiveaffectivestatesrangedfrom77.06%to90.74%intheyounggroupandfrom86.25%to
90.25%intheseniorgroup.Theclassificationratesforsenioradultswerehigherthanyoung
adultsinallsevenconditions.
Asfarasgender,asTable5shows,classifyingthecombinationofcorrugatorandzygomati-
cusEMGshowedsimilarresultsregardlessofage;thebaselinewasnearly100%inallcondi-
tionsexceptin2-2inthefemalegroup.Thefiveaffectivestatesrangedfrom81.18%to90.59%
inthefemalegroupandfrom85.00%to90.38%inthemalegroup.
Tables6and7showwheretheMandSDofclassificationaccuracyforthebaselinereached
nearly100%witheithersinglecorrugatororzygomaticusEMG;allfiveaffectivestatesarerela-
tivelylow(from13.19%to69.86%).
Table3. MandSDofclassificationaccuracyacrossbaselineandfiveaffectivestatesviathecombinationofcorrugatorandzygomaticusEMG
dataforallparticipants.
Baseline(%) 0VLA(%) PVHA(%) NVHA(%) PVLA(%) NVLA(%)
1–1 M 100.00 88.47 88.38 87.31 88.47 87.04
SD 0.00 0.60 0.63 0.54 0.41 0.49
1-10-f M 100.00 88.75 86.02 84.95 84.81 86.90
SD 0.00 0.73 0.87 0.82 0.81 0.90
1-10-l M 100.00 88.75 86.20 84.58 85.28 87.36
SD 0.00 0.82 0.65 1.18 1.11 0.76
1–2 M 100.00 90.65 89.44 87.92 89.72 89.95
SD 0.00 0.48 0.65 0.63 0.68 0.62
2–2 M 100.00 75.69 77.13 80.69 79.72 76.57
SD 0.00 1.47 0.98 0.58 0.89 1.00
2-10-f M 100.00 89.07 85.88 85.19 85.46 87.27
SD 0.00 1.03 1.05 0.87 1.10 0.91
2-10-l M 100.00 88.47 86.81 85.05 85.37 87.45
SD 0.00 0.91 0.82 0.66 0.73 1.49
doi:10.1371/journal.pone.0146691.t003
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 8/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
Table4. MandSDofclassificationaccuracyacrossbaselineandfiveaffectivestatesviathecombinationofcorrugatorandzygomaticusEMG
dataforyoungandseniorparticipants.
Baseline(%) 0VLA(%) PVHA(%) NVHA(%) PVLA(%) NVLA(%)
1–1 Young M 100.00 88.60 87.87 88.82 88.01 87.13
SD 0.00 0.39 0.79 0.47 0.70 0.62
Senior M 100.00 90.00 90.00 90.00 90.00 90.00
SD 0.00 0.00 0.00 0.00 0.00 0.00
1-10-f Young M 100.00 88.75 85.96 83.82 84.78 85.96
SD 0.00 1.73 1.32 0.78 1.04 1.45
Senior M 100.00 89.63 89.75 88.50 90.13 89.88
SD 0.00 0.84 0.53 0.79 0.40 0.71
1-10-l Young M 100.00 88.75 86.69 84.41 85.96 85.81
SD 0.00 0.98 1.17 1.54 1.68 1.63
Senior M 100.00 90.25 89.88 89.00 90.00 90.13
SD 0.00 0.79 0.40 0.79 0.83 0.71
1–2 Young M 100.00 90.74 89.19 88.46 89.63 90.00
SD 0.00 0.71 0.36 0.36 0.54 0.38
Senior M 100.00 90.00 90.00 90.13 90.00 90.00
SD 0.00 0.00 0.00 0.40 0.00 0.00
2-10-f Young M 100.00 89.04 86.62 84.26 86.32 86.47
SD 0.00 0.64 1.38 0.86 0.86 1.05
Senior M 100.00 90.00 89.50 89.13 89.88 89.88
SD 0.00 0.59 0.65 0.60 0.71 1.38
2-10-l Young M 100.00 88.97 86.69 83.75 85.22 86.10
SD 0.00 0.60 1.36 1.12 1.36 1.32
Senior M 100.00 90.13 89.38 88.75 89.38 90.00
SD 0.00 0.92 0.88 0.00 1.35 1.02
2–2 Young M 99.99 77.65 78.31 81.76 77.06 78.46
SD 0.05 1.21 1.05 0.90 1.62 1.55
Senior M 100.00 89.00 88.63 87.88 88.88 86.25
SD 0.00 1.15 0.71 0.84 0.92 1.02
doi:10.1371/journal.pone.0146691.t004
StatisticalResults
Tocomparetheeffectsofsinglecorrugatorandzygomaticusandthecombinationofcorruga-
torandzygomaticusEMGforaffectrecognition,datafromTables3,6and7weresubmittedto
repeatedmeasuresANOVAanalysis.Theresultsdemonstratedasignificantprimaryeffect(F
(1,41)=87.34,p<0.001),andmultiplecomparisonswithBonferronirevealedsignificantdif-
ferencesbetweenthecombinationofcorrugatorandzygomaticusandthesinglecorrugator,
thecombinationofcorrugatorandzygomaticusandsinglezygomaticus,andthesinglecorru-
gatorandzygomaticus(p<0.001).
Asfarasage,datafromTable4wassubjectedtotheindependentT-test,andresultsshowed
significantdifferencesbetweenyoungandseniorparticipants(t=-2.67,p<0.01,two-tailed).
DatafromTable5wassubjectedtoindependentT-test,withnopronounceddifference
betweenfemaleandmalesubjects(t=-1.58,p>0.05,two-tailed).
DiscussionandConclusions
Theinteractionbetweenhumansandmachineswill,infuture,mostlikelybeanempathicrela-
tionshipresemblingthe“companionship”observedinHMIresearch.Insucha
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 9/14
RecognitionofIntensiveValence&ArousalAffectiveStatesviaFacialEMG
Table5. MandSDofclassificationaccuracyacrossbaselineandfiveaffectivestatesviathecombinationofcorrugatorandzygomaticusEMG
dataforfemaleandmaleparticipants.
Baseline(%) 0VLA(%) PVHA(%) NVHA(%) PVLA(%) NVLA(%)
1–1 Female M 100.00 90.07 89.56 89.49 89.41 89.49
SD 0.00 0.39 0.58 0.36 0.38 0.36
Male M 100.00 88.88 90.00 90.00 90.00 88.75
SD 0.00 0.40 0.00 0.00 0.00 0.00
1-10-f Female M 100.00 87.65 87.06 86.25 87.28 86.62
SD 0.00 1.24 1.16 1.20 0.92 1.19
Male M 100.00 89.38 88.00 89.88 89.38 90.00
SD 0.00 1.06 1.05 0.71 0.66 0.59
1-10-l Female M 100.00 86.76 87.94 86.40 87.06 86.91
SD 0.00 1.73 1.21 1.05 1.21 1.09
Male M 100.00 89.00 88.00 90.13 89.38 90.25
SD 0.00 0.79 0.65 0.40 1.06 1.29
1–2 Female M 100.00 90.59 90.59 90.00 89.71 89.85
SD 0.00 0.31 0.47 0.62 0.00 0.31
Male M 100.00 90.00 90.00 90.38 89.88 89.00
SD 0.00 0.00 0.00 0.60 0.40 0.53
2-10-f Female M 100.00 86.25 87.94 85.44 87.43 86.47
SD 0.00 1.20 0.93 1.38 1.12 1.11
Male M 100.00 88.88 88.38 90.13 89.50 90.00
SD 0.00 0.71 1.03 0.92 0.87 0.83
2-10-l Female M 100.00 87.43 87.50 85.81 87.43 86.91
SD 0.00 0.88 1.20 1.04 1.07 1.62
Male M 100.00 89.13 88.75 90.00 89.38 89.88
SD 0.00 1.03 0.83 0.59 0.66 1.09
2–2 Female M 99.97 81.18 84.04 81.32 82.57 79.41
SD 0.06 0.71 1.04 1.11 1.15 0.98
Male M 100.00 87.63 87.25 85.00 85.50 87.38
SD 0.00 0.40 0.99 1.32 1.05 0.92
doi:10.1371/journal.pone.0146691.t005
Table6. MandSDofclassificationaccuracyacrossbaselineandfiveaffectivestatesviasolelycorrugatorEMGdataforallparticipants.
Baseline(%) 0VLA(%) PVHA(%) NVHA(%) PVLA(%) NVLA(%)
1–1 M 99.87 22.22 34.07 38.94 34.68 30.83
SD 0.10 1.43 1.97 1.84 1.60 2.27
1-10-f M 100.00 55.23 61.20 56.34 60.19 62.87
SD 0.00 2.58 3.04 2.70 2.43 2.50
1-10-l M 100.00 56.53 62.18 56.81 60.14 61.85
SD 0.00 3.58 1.50 2.40 1.41 2.78
1–2 M 100.00 68.10 69.86 65.42 63.89 65.88
SD 0.00 2.74 1.55 1.73 2.82 3.18
2-10-f M 100.00 55.83 61.67 56.71 59.12 61.90
SD 0.00 3.17 2.44 2.19 3.13 3.19
2-10-; M 100.00 55.60 61.25 56.57 58.80 61.94
SD 0.00 3.64 2.91 2.33 2.56 3.06
2–2 M 99.91 14.03 23.38 34.31 25.09 25.46
SD 0.04 1.00 1.24 1.71 1.23 1.84
doi:10.1371/journal.pone.0146691.t006
PLOSONE|DOI:10.1371/journal.pone.0146691 January13,2016 10/14