Framework for Electroencephalography-based Evaluation of User Experience

4.3 MB·

by  Jérémy Frey

Framework for Electroencephalography-based Evaluation
of User Experience
Je´re´myFrey MaximeDaniel JulienCastet
Univ. Bordeaux,France ImmersionSAS,France ImmersionSAS,France
[email protected] [email protected] [email protected]
MartinHachet FabienLotte
Inria,France Inria,France
[email protected] [email protected]
6
1
0
2

n
a
J

2
1

]
C
H
Figure1: Wedemonstratehowelectroencephalographycanbeusedtoevaluatehuman-computerinteraction. Forexample,akeyboard(left)canbe
.
s comparedwithatouchinterface(middle)usingacontinuousmeasureofmentalworkload(right,hereparticipant4).
c
[

ABSTRACT INTRODUCTION
1
Measuringbrainactivitywithelectroencephalography(EEG) In practice, a tool is only as good as one’s ability to assess
v
8 is mature enough to assess mental states. Combined with it. Forinstance,evaluationsinHuman-ComputerInteraction
6 existing methods, such tool can be used to strengthen the (HCI)usuallyrelyoninquiries–e.g.,questionnairesorthink
7 understanding of user experience. We contribute a set of aloudprotocols–oronusers’behaviorduringtheinteraction–
2 methodstoestimatecontinuouslytheuser’smentalworkload, e.g.,reactiontimeorerrorrate. However,whilebothtypesof
0 attentionandrecognitionofinteractionerrorsduringdifferent methodshavebeenusedsuccessfullyfordecades,theysuffer
. interactiontasks. Wevalidatethesemeasuresonacontrolled fromsomelimitations. Inquiriesarepronetobecontaminated
1
0 virtualenvironmentandshowhowtheycanbeusedtocompare by ambiguities [23] or may be affected by social pressure
6 differentinteractiontechniquesordevices,bycomparinghere [26]. Itisalsoverydifficulttogainreal-timeinsightswithout
1 a keyboard and a touch-based interface. Thanks to such a disruptingtheinteraction. Indeed, thinkaloudprotocoldis-
: framework,EEGbecomesapromisingmethodtoimprovethe tractsusersandquestionnairescanbegivenonlyatspecific
v
overallusabilityofcomplexcomputersystems. timepoints,usuallyattheendofasession–whichleadsto
i
X abiasduetoparticipants’memorylimitations[17]. Onthe
r otherhand,metricsinferredfrombehavioralmeasurescanbe
a AuthorKeywords computedinreal-time,butaremostlyquantitative. Theydo
EEG;HCIEvaluation;Workload;Attention;Interaction
notprovidemuchinformationaboutusers’mentalstates. For
errors;Neuroergonomy
example,ahighreactiontimecanbecausedeitherbyalow
concentrationlevelorbyadifficulttask[1,14].
ACMClassificationKeywords
Recently,ishasbeensuggestedthatportablebrainimaging
H.5.2UserInterfaces: Evaluation/methodology
techniques–suchaselectroencephalography(EEG)andfunc-
tionalnearinfraredspectroscopy(fNIRS)–havethepotential
toaddresstheselimitations[8,27,11].fNIRShasbeenstudied
toassessusers’workload,forexampletoevaluateuserinter-
faces[15]ordifferentdatavisualizations[25]. However,most
PublicationrightslicensedtoACM.ACMacknowledgesthatthiscontributionwas
oftheseworksareevaluatingpassivetasksorverybasicin-
authoredorco-authoredbyanemployee,contractororaffiliateofanationalgovern-
ment.Assuch,theGovernmentretainsanonexclusive,royalty-freerighttopublishor teractionsthatarenotecological,especiallyasuserinterfaces
reproducethisarticle,ortoallowotherstodoso,forGovernmentpurposesonly andinteractionsaregettingmorecomplex.
CHI’16,May07-12,2016,SanJose,CA,USA
Copyrightisheldbytheowner/author(s).PublicationrightslicensedtoACM.
ACM978-1-4503-3362-7/16/05...$15.00
DOI:http://dx.doi.org/10.1145/2858036.2858525
In this paper, not only do we detail a framework for HCI Errorrecognitionrelatestothedetectionbyusersofanout-
designerstogainfurtherinsightsaboutuserexperienceusing come different from what is expected [22]. We focused on
brain signals – here from EEG – but we also validate this interaction errors [9], which arise when a system reacts in
frameworkonanactualandrealistictask. Weshowthatthey an unexpected way, for example if a touch gesture is badly
canbeusedtocomparedifferentHCI,inanenvironmentthat recognized. Interactionerrorsenabletoassesshowintuitive
providesmanysimulationsandwhereparticipantsareengaged a UI is, and they are hardly measurable by another physio-
inrichinteractions. logicalsignalthanEEG.Thecombinedmeasureofworkload,
attentionanderrorrecognitionconstitutesapowerfulcomple-
Inthefollowingsections,wedescribethevirtualenvironment
mentaryevaluationtoolforpeoplewhodesignnewinteraction
that we developed, specifically aimed at validating the use
techniques.
ofEEGasanevaluationmethodforHCI.Wevalidatedthe
workload induced by our environment in a first study, with Eventhoughcommercialsolutions,suchastheB-Alertsys-
NASA-TLXquestionnaires[14]. Then,wedetailhowEEG tem1, are already pointing this direction, they are validated
canbeusedtoassess3differentconstructs: workload,atten- in the literature with lab tasks only [1]. Our work, on the
tionanderrorrecognition. Finally,duringthemainstudywe otherhand,isclosertothefield. Moreimportantly,asopposed
employEEGrecordingstomeasurecontinuouslysuchwork- toproprietarysoftware,ourmethodologyistransparentand
load,altogetherwiththeattentionlevelofparticipantstoward ourmultidimensionalindexofuserexperiencecanbeeasily
externalstimuliandthenumberofinteractionerrorstheyper- replicated.
ceived,whiletheyinteractwithourvirtualenvironment. We
tooktheopportunityoftheseEEG-basedmeasurestocompare VIRTUAL3DMAZE
akeyboardinputtoatouchscreeninput. The3Dvirtualenvironmentthatwebuiltusesgamification[7]
toincreaseusers’engagementandensurebetterphysiological
Tosummarize,ourmaincontributionsare:
recordings[10]. Suchavirtualenvironmentalsoenablesusto
1. TovalidatetheuseofEEGasacontinuousHCIevaluation assessworkload,attentionanderrorrecognitionduringeco-
methodthankstocontrolledandrealisticinteractiontasks logicalandrealisticinteractiontasks. Indeed,suchconstructs
wedesigned. aretraditionallyevaluatedduringcontrolledlabexperiments
2. Todemonstratehowsuchtoolcanassesswhichofthetested basedonprotocolsfrompsychologythatarevastlydifferent
interactiontechniqueisbettersuitedforaparticularenvi- fromanactualinteractiontask,see,e.g.,[12].
ronment.
3. To propose a framework that can be easily replicated to Overalldescription
improveexistinginterfaceswithlittleornomodifications. Thevirtualenvironmenttakestheformamazewhereplayers
havetolearnandreproduceapathbytriggeringdirectionsat
regularintervals(seeFigure2). Acharacterdisplayedwitha
thirdpersonperspectivemovesbyitselfatapredefinedspeed
Relatedwork insideorthogonaltunnels. Soonafterthecharacterentersa
Since a few years, brain imaging has been used in HCI to new tunnel, symbols appear on-screen. Those symbols are
deepentheunderstandingofusers,thanksnotablytothespread basic2Dshapes,suchassquares,circles,triangles,diamonds
ofaffordableandlightweightdevices. Forinstance,EEGand or stars, and their positions (bottom, top, left or right) indi-
fNIRSareparticularlywellsuitedformobilebrainimaging catewhichdirectionsare“opened”. Playersmustselectone
[20,6]. Thesetechniquescanbeemployedtostudythespatial ofthesesymbolsbeforethecharacterreachestheendofan
focusofattention[32]ortoidentifysystemerrors[33,29]. intersection,eitherbypressingakeyortouchingthescreen.
Ifusersrespondtooearly,i.e.,beforesymbolsappeared,too
EEG and fNIRS are opportunities to assess the overall us-
late,oriftheyselectadirectionthatdoesnotexist,theyloose
ability of a system and improve the ergonomics of HCI. In
points and the character “dies” by smashing against a wall,
a recent work we showed preliminary results regarding the
respawningsoonafteratthebeginningofthecurrenttunnel.
evaluationofmentalworkloadduringa3Dmanipulationtask
[35]. Afterbeingprocessed,EEGsignalshighlightedwhich Themainelementofthegameplayconsistsinselectingthe
partsoftheinteractioninducedahighermentalworkload. In directionsinthecorrectorder. Indeed,onelevelcomprised
thepresentpaperwegomuchbeyond,exploringacontinuous twophases. Duringthe“learning”phaseaparticularsequence
indexofworkloadaswellastwoothersconstructs,namely ofsymbolsishighlighted;ateachsymbols’appearanceone
attentionanderrorrecognition. Wealsorigorouslyvalidate ofthemisbouncingtoindicatethecorrectdirection. Another
suchindexes,andstudytheminlightofbehavioralmeasures cuetakestheformofa“breadcrumbtrail”,abeamoflightthat
(performancesandreactiontimes)andinquiries(NASATLX precedesthecharacterandpointstotherightdirection(see
questionnaire). Figure2b). Selectinganavailablebutincorrectdirectiondoes
notresultinthecharacter’s“death”butleadstoalossofpoints.
Attentionreferstotheabilitytofocuscognitiveresourcesona
Avisualfeedbackisgiventouserswhentheyselectadirection:
particularstimulus[17]. InHCI,measuringtheattentionlevel
thecorrespondingsymbolturnsgreenifthechoiceiscorrect
couldhelptoestimatehowmuchinformationusersperceive.
andredotherwise. Whentheendofthemazeisreached,the
In the present work the measure of attention relates to inat-
characterloopsovertheentirepathsothatplayershaveanother
tentionalblindness;i.e.itconcernsparticipants’capacityto
processstimuliirrelevanttothetask[4]. 1http://www.advancedbrainmonitoring.com/
(a) (b) (c)
Figure2: Thevirtualenvironment,whereplayerscontrolacharacterthatmovesbyitselfinsidea3Dmaze. A:Symbolsappearineachtunnelto
indicatethepossibledirectionsforthenextturn;playershavetoselectaparticularsequenceofsymbols/directions.B:Duringthe“learning”phase,the
correctdirectionishighlightedbyabreadcrumbtrailandtheassociatedsymbolbounces(herethediscontop).C:Controlsdependonthepositionof
thecharacter.Ifthecharacterisontherightside,playershavetopressrightinordertogoup.
opportunitytolearnthesequence. Whenthetrainingphase symbolssequencetheyhavetolearn. Moresymbolstobe
ends, the“recall”phasefollows. Thesymbolsareidentical heldintheworkingmemoryincreasesworkload[12,31].
butthecuesarenomoredisplayed;playershavetoremember • Numberofdirections:ateachintersection,upto4directions
bythemselvestherightpath. Symbolspositionineachtunnel are“opened”inthemaze;thecomplexityofthesymbols
andsymbolssequencearerandomlydrawnwhenanewlevel sequencegrowsasthisnumberincreases.
starts. • Gamespeed: thepaceofthegamecanbeadjustedtoin-
creasetemporalpressure. Whenthespeedincreasessym-
Beside learning a sequence, the principal challenge comes
bols appear sooner and users must respond quicker, thus
fromhowthedirectionsareselected. Thethird-personview
increasing overall stress [14, 19]. In the easiest level the
fulfillsapurpose: theinputdevicethatusersarecontrolling
characterspends6sinatunnelandplayersmustrespond
–i.e.keyboardortouchscreen–ismappedtothecharacter
within3saftersymbolsappearance;inthehardestlevela
position. Sincethecharacterisafuturisticsurferthatdefies
tunnellasts2sandplayershave1stochooseasymbol.
thelawofgravity, itslidesbyitselffromthebottomofthe
tunneltooneofthewallsortotheceilingfromtimetotime. • Spatialorientation: inordertokeepselectingthecorrect
In this latter situation, when the character is upside down, directions, users have to perform a mental rotation if the
commandsareinvertedcomparedtowhatplayersareusedto, charactertheycontroljumpsfromthefloortothewallsor
eventhoughsymbolsremaininthesamepositions. Thisgame totheceiling. Furthermore,theyneedtoupdatetheirframe
mechanismstressesspatialcognitionabilities;usershaveto ofreferenceasoftenasthecharactershiftsfromoneside
constantlyremainawareoftwodifferentframesofreference. toanother. Dependingonthespatialabilityofusers,this
Forexample,if“up”and“left”directionsareopeninagiven mechanismcancauseanimportantcognitiveload[28].
tunnelandifthecharacter’sposition–controlledbytheappli- We used those mechanisms and dimensions to create 4 dif-
cation–isontherightwall,asillustratedinFigure2c,users ferentdifficultylevelsforthegame: “EASY”,“MEDIUM”,
havetopressrighttogoup. Thisdiscrepancybetweeninput “HARD” and “ULTRA” (see Table 1). These levels affect
and output is a reminder of the problematic often observed mostly(symbolic)memoryloadandtimepressure. Indeed,
with3Duserinterfaces,wheremostusersmanipulateadevice the3Dmazeismoreaboutrememberingasequenceofsym-
with2degreesoffreedom(DOF),suchasamouse,tointeract bolsordirectionsratherthanspatialnavigationperse.Because
witha6DOFenvironment. randomizationcouldcreateloopsinthemazetopographyand
Thecombinationofthegamedesignandgamemechanisms sincetherewerenolandmarks,itisunlikelythatparticipants
hereindescribedoffersawidevarietyofelementsthatweput wereabletoadoptanallocentricstrategy.
inusesoastoinvestigateusers’mentalstates.Inparticular,we WhiletheEASYlevelisdesignedtobecompletedwithvery
detailbelowhowwetunedthegameelementstomanipulate littleeffort,theULTRAlevel,ontheotherhand,isdesigned
theuser’sworkloadandattentionincontrolledwaysaswell tosustainaveryhighlevelofworkload,uptothepointthatit
as to trigger interaction errors. Knowing which constructs isbarelypossibletocompleteitwithnoerror. Whileduring
value(e.g.highorlowworkload)toexpect,wecanvalidate EASYlevelsthereisnoneedtoperformmentalrotationsand
whether our EEG-based estimates during interaction match playershavetomemorizeonly2symbolsthatareconstrained
theseexpectations,andthuswhethertheyarereliable. toeitherleftandrightdirections,inULTRAlevelstheframe
ofreferencechangesbetweeneachselectionandthesequence
Manipulatingworkload reaches5symbolsthatcouldappearinall4directions,and
Ourvirtualenvironmentpossessesseveralcharacteristicsthat playershavetoreactthriceasfast. Nomatterthelevel,players
couldbeusedtoinducedifferentlevelsofmentalworkload. had3“loops”tolearnthemazeandanothersetof3loopsto
Wecannotablyadjust4parameters: reproducethepath.
• Mazedepth: thenumberoftunnelsplayershavetocross
beforereachingtheendofthemaze,hencethelengthofthe
pointingisco-locatedwithsoftwareevents,sinceuserscan
Difficulty Depth Directions Resp. time Orientation
directly indicate where they want to interact. However, in
EASY 2 2 3s 0% our case, we decided to mimic exactly the behavior of the
MEDIUM 4 3 2.5s 30% keyboardinterface. Thatistosaythatwiththetouchscreen
HARD 5 4 2s 60% aswellplayershavetoorientatethemselvesdependingonthe
ULTRA 5 4 1s 100% positionofthecharacter. Hence,ifthecharacterispositioned
Table1:Fourdifficultylevelsarecreatedbyleveragingongamemech- ontheleft,playershavetotouchtherightfringeofthescreen
anisms.Depth:numberofdirections/symbolsplayershavetolearn.Di- inordertogoup.Thisismostlycounter-intuitivesinceplayers
rections: numberofpossibledirectionsateachintersection. Response havetoinhibittheurgetopointtotheactualdirectionthey
time:howmuchtimeplayershavetorespondaftersymbolsappearance.
wanttogo;thereisacognitivedissonance.
Orientation:percentagechancethatthecontrolledcharacterchangesits
orientation. Sinceinourexperimentaldesigntheuseofthedirect(touch-
based) interaction is counter-intuitive, we hypothesize that
itwillleadtoanoverallhighernumberofinteractionerrors
Assessingattention
comparedtotheindirectinterface(keyboard).
We relied on stimuli not congruent to the main task in or-
dertoprobeforinattentionalblindness,usingthe“oddball”
PILOT STUDY: VALIDATION OF THE INDUCED WORK-
paradigm. TheoddballparadigmisoftenemployedwithEEG
astheappearanceofrare(i.e. “odd”)stimuliamongastream LOADLEVEL
offrequentstimuli(i.e.distractors)triggersaparticularevent- Wedesignedourvirtualenvironmentasatest-benchaimedat
related potential (ERP) within EEG signals [5]. ERP are inducingseveralmentalstateswithinusers,notably,different
“peaks”and“valleys”inEEGrecordings,andtheamplitudeof workloadlevels. Thus,wehadtoformallyvalidatethemental
someofthemdecreasesasusersarelessattentivetostimuli. workloadthateachgamelevelseekstoinduce. Assuch,we
conducted a pilot study – separate from the main study to
Our protocol uses audio stimuli. It is based on [3], which
alleviate the protocol of the latter –, with no physiological
studiedtheimmersionofvideogameplayers. Inourvirtual
recordings but using the NASA-TLX questionnaire [14], a
environment,whileusers’characterswerenavigatinginthe
wellestablishedquestionnairethataccountsforworkload.
maze, sounds were played at regular intervals, serving as a
background “soundtrack” that was consistent with the user
Protocol
experience. 20%ofthesesoundshadahighpitch(oddevent)
15participantstookpartinthisstudy(4females),meanage
andtheremaining80%hadalowpitch(distractingevents)–
24.53(SD:3.00). Weusedawithin-subjectdesign;allpartici-
thisproportionisonparwiththeliterature[3,9].
pantsansweredforall4difficultylevels. Thegamingsession
Ourhypothesisisthattheattentionlevelofparticipantstoward used the keyboard and started with 2 “training levels”, that
sounds – as measured with the oddball paradigm – should introducedparticipantstothegamemechanisms. Inthefirst
decreaseastheworkloadincrease,sincemostoftheircognitive traininglevel,playerslearnedtheobjectiveofthegame. In
resourceswillbeallocatedtothemaintaskduringthemost thesecondtraininglevel,theydiscoveredhowthecharacter
demandinglevels. couldchangeitsorientationbyitself. Afterthistrainingphase,
participantscontinuedwiththemainphaseoftheexperiment.
Assessingerrorrecognition
Duringthemainphaseoftheexperiment,participantsplayed
EEGcouldbeusedtomeasureinteractionerrors,i.e.errors
once each of the four levels (EASY, MEDIUM, HARD or
originatingfromanincorrectresponseoftheuserinterface,
ULTRA), in a random order. Immediately after the end of
thatdiffersfromwhatuserswereexpecting[9]. Interaction
alevel,participantsweregivenaNASA-TLXquestionnaire
errorsareofparticularinterestforHCIevaluationsincethey
to inquire about their mental workload. The questionnaire
couldaccountforhowintuitiveaninterfaceis[11]. Inorderto
took the form of a 9-points Likert scale. As in the original
testthefeasibilityofsuchmeasure,wedecidedtoimplement
questionnaire[14],itcomprised6items,thatassessedmental
twodifferentinteractiontechniques. Bothofthemusediscrete
demand, physical demand, temporal demand, performance,
events–i.e.symbols’selection–sothatwecouldmoreeasily
effortandfrustration. Theexperimentlastedapproximately
synchronizeEEGrecordingswithin-gameeventslateron.
25minutesandfinishedonceparticipantsplayedall4levels
Thefirsttechniqueusesindirectinteractionsbythemeanofa andfilledthecorrespondingNASA-TLXquestionnaire.
keyboard(Figure1,left). Induetime,left,right,upordown
arrow keysare used to sendthe character inthe tunnel that Results
is situated to its left, right, top or bottom. Indeed, we have Foreachparticipantandeachlevelofdifficulty,weaveraged
seenpreviouslythatinourvirtualenvironmentplayershaveto the6itemsoftheNASA-TLXquestionnaireandnormalized
orientatethemselvesdependingofthepositionofthecharacter. thescalesfrom[1;9]to[0;1]–exceptforthe“performance”
Ifthecharacterismovingonthesides,playershavetoperform item,thatwasnormalizedfrom[1;9]to[1;0]becauseitsscale
amentalrotationof90°,ifitisontheceilingthentheangleis isinreverseordercomparedtotheotheritems(“1”for“good”
180°,i.e.commandsareinverted. and“9”for“poor”).
Thesecondtechniqueusesdirectinteractionbythemeanofa The resulting averaged scores are: EASY: 0.11 (SD: 0.09);
touchscreen(Figure1,middle). Usually,withtouchscreen, MEDIUM:0.32(SD:0.17);HARD:0.43(SD:0.13);ULTRA:
1.00 ofsuchconstructsduringdifferentandcomplexinteraction
tasks.
0.75 Concerningattention,wedidnotdevelopadedicatedtaskper
e
TLX scor0.50 sgeraftoerdittsocoaulirbvrairtitouna.lSeninvcireotnhmeaeundt,iowpersoibmesplwyeureseadlraeasdpyecinifitec-
A− levelofourgame.
S
NA l
0.25
Calibrationofworkload
WeusedtheprotocolknownastheN-backtasktoinduce2
0.00 differentworkloadlevelsandcalibrateourworkloadestimator.
EASY MEDIUMDifficultyHARD ULTRA TheN-backtaskisawell-knowntasktoinduceworkloadby
Figure3:NASA-TLXscoresobtainedduringthepilotstudy.Eachdiffi- playingonmemoryload[24]. Itshowedpromisingresultsin
cultyleveldifferssignificantlyfromtheothers(p<0.01). [35]whereitcouldbeusedtotransfercalibrationresultstoa
3Dcontext.
0.65(SD:0.13)–seeFigure3. Arepeatedmeasuresanaly-
IntheN-backtask,userswatchasequenceoflettersonscreen,
sisofvariance(ANOVA)showedasignificanteffectofthe
thelettersbeingdisplayedonebyone. Foreachlettertheuser
difficultyfactorovertheNASA-TLXscoresandapost-hoc
hadtoindicatewhetherthedisplayedletterwasidenticalor
pairwiseStudent’st-testwithfalsediscoveryrate(FDR)cor-
differenttotheletterdisplayedNlettersbefore,usingaleftor
rectionshowedthateachlevelsdifferedsignificantlyfromthe
rightmouseclickrespectively. Hence,usershavetoremember
others(p<0.01).
nitemsatalltimes.
Discussion Weimplementedaversionsimilarto[12],removingvowelsto
Inthispilotstudy, wedemonstratedthroughquestionnaires preventchunkingstrategiesbasedonphonemes. Weusedthe
thateachdifficultylevelpresentedinTable1inducesadiffer- sametimeconstraintasin[35],i.e.lettersappearedfor0.5s,
entworkloadlevel.Hence,wecanuseourvirtualenvironment withaninter-stimulusintervalof1.5s. Eachuseralternated
asabaselinetoassessthereliabilityofanalogousEEGmea- between“easy”blockswiththe0-backtask(theuserhadto
suresandputintoperspectivethisnewevaluationmethod. identify whether the current letter was a randomly chosen
targetletter,e.g. ‘X’)and“difficult”blockswiththe2-back
EEGINPRACTICE task(theuserhadtoidentifywhetherthecurrentletterwasthe
EEGmeasuresthebrainactivityundertheformofelectrical sameletterastheonedisplayed2lettersbefore),seeFigure4.
currents[21]. ToidentifymentalstatesfromEEG,3typesof
informationcanbeused:
• Frequency domain: oscillations that occur when large
groupsofneuronsfirealtogetheratasimilarfrequency
• Temporal information: ERPs possess temporal features;
positiveandnegative“peaks”withvaryingamplitudesand
delays.
• Spatial domain: position of the electrodes that record a
specificbrainactivity.
However,thereisanimportantvariabilitybetweenpeople’s Figure4: Workloadcalibrationtask. Top: difficulttask(2-backtask),
EEGsignals,andmanyexternalfactorsthatcouldinfluence thetargetletteristheonethatappearedtwostepsearlier,usershaveto
selecttrials4and5. Bottom: easytask(0-backtask),thetargetletter
EEGrecordings(amplifier’sspecifications,electrodesexact
“S”israndomlychosen,usershavetoselecttrials2and5.
location,andsoon). Assuch,itisdifficulttoidentifyauniver-
salsetoffeaturestoestimateagivenmentalstate,fordifferent
Eachblockcontained60letterspresentations. 4letterswere
sessions and participants. This is why machine learning is
drawnatthebeginningofablocksothatthenumberoftar-
typicallyusedinEEGstudies[2]. Withthisapproach,acal-
getlettersaccountedfor25%ofthetrials. Eachparticipant
ibration phase occurs so that the system could learn which
completed6blocks,3blocksforeachworkloadlevel(0-back
featuresareassociatedtoaspecificindividual,duringatask
vs2-back). Therefore,360calibrationtrials(i.e.onetrialbe-
thatisknowntoinducethestudiedconstruct.Oncethecalibra-
ingoneletterpresentation)werecollectedforeachuser,with
tioniscompleted,themachinecouldthenusethisknowledge
180 trials for each workload level (“low” vs “high”). This
togaininsightsaboutanunknowncontext,forexampleanew
calibrationphasetakesapproximately12minutes.
interactiontechniquethatonewouldwanttoevaluate.
Tocalibrateworkloadandattention,wechosetousestandard Calibrationoferrorrecognition
calibrationtasks,validatedbytheliterature,sothatourfind- Wereplicatedthestandardprotocoldescribedin[9]tocalibrate
ings could be easily reproduced. Moreover, as shown later the system regarding error recognition. The task simulates
in this paper, using a single of these tasks to calibrate each ascenarioinwhichuserscontrolthemovementsofarobot.
constructestimatorwasenoughtoobtainreliableestimations The robot appears on screen and has to reach a target. At
eachturnuserscommandtherobottogorightorleftinorder wasalowpitchedbeatof70ms–wedidnotusepuretonesto
to reach the target as fast as possible (with the least steps). improveusersexperience. Thepaceofthegamewasadjusted
However,therobotmayunderstandbadlythegivencommand. sothatasound(targetordistractor)wasplayedeverysecond.
Thisissimulatedbysometrialsduringwhichthecommand Sincetheprobesforattentionreliestotheoddballparadigm,
is(onpurpose)erroneouslyinterpreted;henceaninteraction wechosea20%likelihoodofappearanceforthetargetevent.
errorhappens. TheERPthatcanbeseeninEEGfollowing Thecalibrationlastedabout7minutes,after350soundswere
aninteractionerrorisknownasan“errorrelatedpotential”, played. Note that participants were instructed to count the
ErrP[9]. Thecalibrationtaskisasimplifiedversionofthis “odd”eventsonlyduringthecalibrationphase,andnotduring
scenario: therobotispicturedbyabluerectangleonscreen thecompletionofthe3Dmaze.
thatuserscontrolwiththearrowkeys,thetargetisrepresented
byablueoutline. TherobotisconstrainedtotheXaxisand MAINSTUDY:EEGASANEVALUATIONMETHOD
alongthisaxisthereareonly7differentpositionsbothforthe Themainstudyconsistedintheevaluationofthegameenviron-
robotandthetarget(seeFigure5). mentwithtwodifferenttypesofinterfacesusingEEGrecord-
ings. Assuchwecreateda4(difficulty: EASY,MEDIUM,
1 Press 2 Cursor 3 Cursor HARD,ULTRA)×2(interaction: KEYBOARDvsTOUCH)
right selected moves
within-subjectexperimentalplan. Ourhypothesesare:
1. TheworkloadindexmeasuredbyEEGishigherinTOUCH
and increases with the difficulty, reflecting NASA-TLX
scoresobtainedduringthepilotstudy.
1s 1s 2. The attentional resources that participants assign to the
4 Press 5 Cursor 6 Cursor soundsdecreaseasthedifficultyincreases.
right selected moves 3. TheTOUCHconditioninducesahighernumberofinterac-
Interaction error
tionerrorscomparedtotheKEYBOARDcondition.
Thegamingphasewassplitintotwosequences,oneforeach
interactiontechnique. Toavoidatootediousexperiment,par-
ticipantsalternatedbetweengamesessionsandthe3calibra-
1s 1s
Figure5:Errorrecognitioncalibrationtask. Userscontrolabluefilled tiontasks(workload,attentionanderrorrecognition). Since
rectangle.Theyhavetomoveittoanoutlinedtargetbypressingtheleft the analysis were performed offline, there was no need to
orrightarrowkey. 20%ofthetime,therectanglegoesintheopposite clusterallthecalibrationsatthebeginningoftheexperiment.
direction,thuscausinganinteractionerror.
Theorderofthegamingsessionsandcalibrationphaseswas
We choose a ratio for the occurrence of interaction errors counter-balancedbetweenparticipantsfollowingalatinsquare
thatisconsistentwiththeliterature. 80%ofthemovements (see Figure 6). After the experiment, the signals gathered
matched the actual key pressed and for the other 20% the fromthecalibrationtaskswereprocessedinordertoevaluate
“robot”movedintheoppositedirection. Itwasnecessarynot boththevirtualenvironment(difficultylevels)andthechosen
tobalancebotheventsastoofrequenterrorsmaynotbeper- interactiontechniques.
ceivedasunexpectedanymore,andthusmaynotleadtoan
ErrP.Atimerwassettopreventtheappearanceofartifacts,
suchasmusclemovements,withinEEGrecordings(seethe
Discussionsectionforfurtherconsiderationforartifacts). The
rectanglemoves1safterakeywaspressed,andaftermove-
ment completion users have to wait another 1s before they
couldpressakeyagain. Therectangleturnedyellowtotell
userstheycouldnotcontrolitduringthatsecond.
Atrialiscompletedoncetherobotreachesthetarget. Atrial
fails if after 10 attempts the robot is not yet on target. At
thebeginningofeachtrial,thescreenisreinitializedwitha
randomnewpositionfortherobotandthetarget. Thelasttrial
occurredafter350interactionswereperformed. Onaverage
thiscalibrationphaselasted15minutes.
Figure6: Theorderofthe3calibrationtasksand2interactiontech-
niqueswascounter-balancedbetweenthe12participantstoimproveen-
Calibrationofattention
gagement.
Thecalibrationofattentionoccurredwithinasimplifiedver-
sionofthevirtualenvironment. Usersdidnothavetocontrol
thecharacterduringthisspeciallevel,itwasmovingbyitself Apparatus
through the maze. They were asked to watch the character EEGsignalswereacquiredat512Hzwith2g.tecg.USBamp
andcountintheirheadhowmanytimestheyheardthe“odd” amplifiers. We used 32 electrodes placed at the AF3, AFz,
sound,i.e.,ahighpitchedbelllasting200ms. Thedistractor AF4,F7,F3,Fz,F4,F8,FC3,FCz,FC4,C5,C3,C1,Cz,C2,
C4,C6,CP3,CPz,CP4,P7,P3,Pz,P4,P8,PO7,POz,PO8, foreachbandasetofCommonSpatialPatterns(CSP)spa-
O1,OzandO2sites. tial filters. That way, we reduced the 32 original channels
downto6 “virtual”channels thatmaximize thedifferences
12participantstookpartinthisstudy(3females),meanage
betweenthetwoworkloadlevels[30]. Sincethecalibration
26.25(SD:3.70). Allofthemreportedadailyuseoftactile
(N-backtask)andusecontexts(virtualenvironment)differs
interfaces. Theexperimentoccurredinaquietenvironment,
substantially, we used a regularized version of these filters
isolatedfromtheoutside. Thereweretwoexperimentersin
calledstationarysubspaceCSP(SSCSP)[35]. SSCSPfilters
theroomandtheprocedurecomprisedthefollowingsteps:
aremorerobusttochangesbetweencontextssincetheytake
1. Participantsenteredtheroom,readandsignedaninformed into account the distributions of the EEG signals recorded
consentformandfilledademographicquestionnaire. duringboththecalibrationandtheusecontexts(inanunsu-
2. WhileoneoftheexperimenterinstalledanEEGcaponto pervisedway,i.e.withoutconsideringtheexpectedworkload
participants’heads,theotherexperimenterintroducedpar- levels)toestimatespatialfilterswhoseresultingsignalsare
ticipantstothevirtualenvironment. Theyplayed2training stableacrosscontexts(see[35]fordetails). Finally,foreach
levelsandthe4mainlevelsinanincreasingorderofdif- frequencybandandspatialfilter, weusedtheaverageband
ficulty. They could redo some levels if they did not feel powerofthefilteredEEGsignalsasfeature. Thisresultedin
confidentenough. 30EEGfeatures(5bands×6spatialfiltersperband).
3. Oneofthe3calibrationtasksoccurred(workload,attention
orerrorrecognition). Processingattentionanderrorrecognition
4. Participantsplayedthegameusingoneofthe2interaction Sincebothattentionanderrorrecognitioncanbemeasuredin
techniques(KEYBOARDorTOUCH).Thefourlevelsof ERPs,theysharethesamesignalprocessing.Weselectedtime
difficulty (EASY, MEDIUM, HARD, ULTRA) appeared windowsof1s,startingattheeventofinterest(i.e.soundsfor
twiceduringthesession,inarandomorder. ForTOUCH, attention, rectangle’s movements for error recognition). In
a dedicated training session occurred beforehand so that ordertoutilizetemporalinformation,featureextractionrelied
participantcouldgetusedtothisinteractiontechnique. on regularized Eigen Fisher spatial filters (REFSF) method
5. Anothercalibrationtaskoccurred,differentfromstep3. [16]. Thanks to this spatial filter, specifically designed for
6. Participantstestedthesecondinteractiontechnique. Asin ERPsclassification,the32EEGchannelswerereducedtoa
step 4, TOUCH was preceded by a training session, that setof5channels. Wethendecimatedthesignalbyafactor32.
lasteduntilparticipantsfeltconfidentenoughtoproceedto The“decimate”functionofMatlab, thatappliesalow-pass
themaintask. filterbeforedecimationtopreventaliasing, wasused. Asa
7. Participantsperformedthelastremainingcalibrationtask. result,therewas80featuresbyepoch(5channels×512Hz×
1s/32).
Agamesession(steps4and6)tookapproximately20minutes
andthewholeexperimentlasted2hours.
Classification
WeusedashrinkageLDA(lineardiscriminantanalysis)asa
EEGAnalyses
classifiersinceitismoreefficientthantheregularLDAwitha
The calibration tasks were used to train a classifier specific
highnumberoffeatures[18].
toeachofthestudiedconstruct. Classifierswerecalibrated
separatelyforeachparticipantwhichensuredmaximalEEG Foreachconstructtherewastwosteps: firstweusedthedata
classificationperformances. WeusedEEGLAB13.4.4b2and collectedduringthecalibrationtaskstoestimatetheperfor-
MatlabR2014atoprocessEEGsignalsoffline. EEGfeatures manceoftheclassifiers. Second,westudiedtheoutputofthe
associatedtoworkloadrelatetothefrequencydomainwhile differentclassifierstoevaluatethevirtualenvironment.
thefeaturesassociatedtoattentionanderrorrecognitionrelate
Toassesstheclassifiers’performanceonthecalibrationdata,
totemporalinformation,asdetailedbelow.
weused4-foldcross-validation(CV).Moreprecisely,wesplit
Processingworkload the collected data into 4 parts of equal size, selecting trials
From the signals collected during the N-back tasks, we ex- randomly,used3partstocalibratetheclassifiersandtestedthe
tractedEEGfeaturesfromeach2stimewindowfollowinga resultingclassifiersontheunseendatafromtheremainingpart.
letterpresentation. Weusedeachofthesetimewindowsas Thisprocessoccurred3moretimessothatintheend,eachpart
anexampletocalibrateourclassifier,whoseobjectivewasto wasusedonceastestdata. Finally,weaveragedtheobtained
learnwhetherthesefeaturesrepresentedalowworkloadlevel classificationperformances. Theperformancewasmeasured
(inducedbythe0-backtask)orahighworkloadlevel(induced usingtheareaunderthereceiver-operatingcharacteristiccurve
by the 2-back task). Once calibrated, this classifier can be (AUROCC).TheAUROCCisametricthatisrobustagainst
usedtoestimateworkloadlevelsonnewdata,herewhileour unbalancedclasses,asitisthecasewithattentionanderror
userswereinteractingwiththevirtualenvironment. recognition(20%oftargets,80%ofdistractors). Ascoreof
“1”meansaperfectclassification,ascoreof“0.5”ischance.
Asin[35],wefilteredEEGsignalsinthedelta(1-3Hz),theta
(4-6Hz),alpha(7-13Hz),beta(14-25Hz)andgamma(26- Once the classifiers were trained thanks to the calibrations
40 Hz) bands. To reduce features dimensionality, we used tasks,wecouldusethemontheEEGsignalsacquiredwhile
participantswereinteractingwiththevirtualenvironment,to
2http://sccn.ucsd.edu/eeglab/ estimatethedifferentconstructsvalues.
Forworkload,weused2slongslidingtimewindowsthatwere
overlapping by 1s, to extract signals and feed the classifier. KEYBOARD TOUCH
From the outputs that was produced by the LDA classifier
0.2
for each participant (i.e., the distance to the separating hy-
perplane),wefirstremovedoutliersbyiterativelyremoving x
e 0.1
oneoutlieratatimeusingaGrubb’stestwithp=0.05,until d
n
nomoreoutlierwasdetected[13]. Wethennormalizedthe d i
outlier-free scores between -1 and +1. As such, for all par- oa 0.0
kl
ticipantsaworkloadindexcloseto+1representsthehighest or
mentalworkloadtheyhadtoendurewhiletheywereplaying. W−0.1
Itshouldcomeclosetothe2-backconditionofthecalibration
phase. Ontheopposite,aworkloadindexcloseto-1denotes −0.2
thelowestworkload,similartothe0-backcondition.
EASYMEDIUMHARD ULTRA EASYMEDIUMHARD ULTRA
Difficulty
Theprocesswassimilarforattention,butweonlyextracted
(a)
epochsthatcorrespondedtothetargetstimulionset,i.e.when
the high pitch sound was played. Note that contrary to [3],
l
thatstudiedtheamplitudesofERPsanddidnotusethedata
gatheredduringthecalibrationphase, herewekeptthema- 0.06 l o0.4
cspehaeirnntiecaislpeaaanrcntosinnnfigodtaeipcnpecdreooiandcdhde.xeAvoesfnstthusecwhLh,DitlhAeetcrhleaessyuslwitfiienerrgeaspbclooauryetisnwgch.aenthbeer on index0.04 l l gnition rati0.3
nti co
Asfortheclassifierdedicatedtoerrorrecognition,theprocess- Atte or re0.2
ingdiffers. Indeed, wecouldnotassumewhichinteraction 0.02 l Err
yieldedornotaninteractionerror,i.e.ifandwhenparticipants
0.1
perceived a discrepancy between what they intended to do
andwhatoccurred. Consequently,wesimplycountedoveran EASY MEDIUM HARD ULTRA KEYBOARDTOUCH
Difficulty Interaction
entiregamesessionthenumberoftimestheclassifierlabelled
(b) (c)
aninteractionasbeingerroneousintheeyeoftheparticipants.
Figure7: EEGmeasures. a: Theworkloadindexsignificantlydiffers
acrossdifficultiesandbetweeninteractiontechniques. b: Theattention
indexsignificantlydiffersacrossdifficulties. c: Thenumberofinterac-
Results
tionerrorsdiffersbyatendencybetweenKEYBOARDandTOUCH.
Unlessotherwisenoted,wetestedforsignificanceusingre-
peated measures ANOVA. For significant main effects, we
usedpost-hocpairwiseStudent’st-testwithFDRcorrection. (Figure7b). Thepost-hocanalysisshowedthattheULTRA
levelsignificantlydiffersfromtheothers(p<0.05).
Workload
Onaverage,theclassifierAUROCCscoreduringthetraining Errorrecognition
taskwas0.92(SD:0.06)–seeTable2. Overthetestsetthere Onaverage,theclassifierAUROCCscoreduringthetraining
wereonaverage2171datapointspersubjectacrossallcondi- taskwas0.82(SD:0.10)–seeTable2. Overthetestsetthere
tion(timewindows). Thestatisticalanalysisoftheclassifier wereonaverage388datapointspersubjectacrossallcondi-
outputduringthegamesessionshowedasignificanteffectof tions(interactions). Duetothenatureofthedata(numbers
thedifficultyfactor(p<0.01);theworkloadindexincreasing ofinteractionerrorsacrossentiregamesessions),weuseda
along the difficulty of the levels (Figure 7a). The post-hoc one-tailedWilcoxonSignedRankTesttostressourhypothesis.
analysisshowedthatalldifficultylevelssignificantlydiffers Thenumberofinteractionerrorsdiffersbyatendency(p=
onefromtheotherwithp<0.01;exceptfortheMEDIUM 0.08)betweentheKEYBOARDandtheTOUCHconditions.
level,whichdiffersfromEASYwithp<0.05andwithHARD 19%oftheinteractions(SD:9%)werelabelledasinteraction
onlybyamargin(p=0.11). Therewasasignificanteffectof errorsbytheclassifierforKEYBOARDvs22%(SD:9%)for
theinteractionfactoraswell(p<0.01),theworkloadbeing TOUCH(Figure7c).
higheronaverageduringtheTOUCHcondition. Therewas
nointeractionbetweendifficultyandinteractionfactors. Behavioralmeasures
BesidesEEGmetrics,wehadtheopportunitytostudypartic-
Attention ipants’reactiontimeandperformancesoastogetaclearer
Onaverage,theclassifierAUROCCscoreduringthetraining pictureoftheiruserexperience.
taskwas0.86(SD:0.05)–seeTable2. Overthetestsetthere
were on average 497 data points per subject across all con- Reactiontime
ditions(oddevents). Thestatisticalanalysisoftheclassifier Therewasasignificanteffectofboththedifficultyandinter-
output during the game session showed a significant effect actionfactors,aswellasaninteractioneffectbetweenthem
of the difficulty factor (p < 0.01) but not of the interaction (p < 0.01). Post-hoc tests showed that all difficulty levels
factor. Theattentionindexdecreasesasthedifficultyincreases differfromoneanother(p<0.01),exceptforMEDIUMand
Construct P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 Average
Workload 0.85 0.93 0.98 0.95 0.97 0.97 0.79 0.87 0.87 0.98 0.95 0.94 0.92
Attention 0.83 0.82 0.96 0.81 0.85 0.90 0.82 0.82 0.86 0.92 0.88 0.83 0.86
Errorrecognition 0.88 0.57 0.90 0.90 0.86 0.90 0.78 0.80 0.88 0.78 0.85 0.74 0.82
Table2:Classificationaccuracyduringthecalibrationtasksforthe3measuredconstructs(AUROCCscores).
Thesearepromisingresultsforthosewhoseektoassesshow
KEYBOARD TOUCH KEYBOARD TOUCH intuitiveaUIiswithexocentricmeasures[11].
me1.1 e1.0
e ti1.0 anc0.8 In this study, we chose to use the particularity of the touch
s0.9 m
espon00..78 Perfor0.6 sactroeuenchtoscmreaeknefthoertiatsskpomsosribeidliitfyficoufldt.irIencdteemda,nwiphuilleatwioenu,swede
R 0.4 keptthecharacterasaframeofreference,resultingininput
E M H U E M H U E M H U E M H U
Difficulty Difficulty commandsthatwere(patently)notco-localizedwithoutput
(a) (b)
directions. Besidesresultsdenotingthedifferencesbetween
Figure8:Behavioralmeasures:reactiontimeinseconds(left)andper- theconditions,participantsalsospontaneouslyreportedhow
formance (proportion of correctly selected directions – right) signifi-
nonintuitivethisconditionwas. Wewantedtoinvestigateour
cantlydiffersbetweendifficultylevelsandinteractions. E:EASY,M:
MEDIUM,H:HARD,U:ULTRA. evaluation methodon a salientdifference atfirst. Then our
frameworkcouldwellbeemployedtogofurther;forexample
seekingphysiologicaldifferencesbetweendirectandindirect
HARD,whichdonotdiffersignificantly(p=0.91). Themean manipulationinterfacesinmoretraditionaltasks.
reactiontimeswererespectivelyforEASY,MEDIUM,HARD
It is interesting to note how those EEG measures could be
andULTRA:0.78s(SD:0.14),0.97s(0.18),0.98s(0.15)and
combinedwithexistingmethodstobroadentheoverallcom-
0.69s(0.06). Themeanreactiontimewas0.78(0.12)forKEY-
prehension of the user experience. For instance, while we
BOARDand0.93(0.13)forTOUCH.SeeFigure8a.Notethat
did show significant differences across difficulty levels and
usershadlesstimetorespondduringhigherdifficultylevels.
betweeninteractiontechniqueswithbehavioralmeasures(re-
actiontimeandperformanceindex),EEGmeasurescouldhelp
Performance
tounderstandtheunderlyingmechanisms. Becausewehave
Theperformancewascomputedastheratiobetweenthenum-
amoredirectaccesstobrainactivity,wecanmakeassump-
berofcorrectselectionsandthetotalnumberofinteractions.
tions about the cause of observed behaviors. For example
Therewasasignificanteffectofboththedifficultyandinter-
participants’worseperformancewithTOUCHthanwithKEY-
actionfactors,aswellasaninteractioneffectbetweenthem
BOARDcouldbeduetothefactthattheyanticipatelessthe
(p < 0.01). Post-hoc tests showed that all difficulty levels
outcomesoftheiractions(moreinteractionerrors);thehigher
differfromoneanother(p<0.01). Themeanperformance
reactiontimemaynotonlybecausedbytheinteractiontech-
wasrespectivelyforEASY,MEDIUM,HARDandULTRA:
niqueperse,butbyahigherworkload. Andwhileparticipants
98%(SD:3),89%(12),83%(17)and55%(21). TheMean
managetocopewiththefastpaceoftheULTRAlevel(the
performancewas85%(13)forKEYBOARDand77%(13)
smallestreactiontimes),theincreaseinperceptualloadlower
forTOUCH.SeeFigure8b.
theirawarenesstotask-irrelevantstimuli.
Additionally,wecanobservethattheperformancesobtainedat
Discussion
theEASY,MEDIUMandHARDlevelsareverysimilarwith
Mostofthemainhypothesesareverified. Theworkloadin-
thekeyboardandthetouchscreen(seeFigure8b). However,
dex as computed with EEG showed significant differences
EEG analyses revealed that the workload was significantly
thatmatchtheintendeddesignofthedifficultylevels. Itwas
higher in the TOUCH condition, meaning that users had to
alsoshownthatinthehighestdifficultytheattentionlevelof
allocate significantly more cognitive resources to reach the
participantstowardexternalstimuliwassignificantlylower–
same performance. This further highlights that EEG-based
i.e.inattentionalblindnessincreased. Concerningtheinterac-
measuresdobringadditionalinformationthatcancomplement
tiontechniques,thenumberofinteractionerrorsasmeasured
traditionalevaluationssuchabehavioralmeasures.
by EEG was higher with the TOUCH condition, but this is
atendencyandnotasignificanteffect. Theworkloadindex, Measuringusers’cognitiveprocessessuchasworkloadand
on the other hand, was significantly higher in the TOUCH attentionmayproveparticularlyusefultoassess3Duserin-
conditioncomparedtotheKEYBOARDcondition. terfaces(3DUI),sincetheyareknowntobemorecognitively
demanding. They require users to perform 3D mental rota-
Thanks to the ground truth obtained during the pilot study
tiontaskstosuccessfullymanipulateobjectsortoorientate
withtheNASA-TLXquestionnaire,theseresultsvalidatethe
themselvesinthe3Denvironment. Moreover,theusualneed
useofaworkloadindexmeasuredbyEEGforHCIevaluation
foramappingbetweentheuserinputs(withlimiteddegrees-
andsetthepathfortwootherconstructs: attentionanderror
of-freedom–e.g.,only2foramouse)andthecorresponding
recognition.Besidetheevaluationofthecontent(i.e.difficulty
actionson3Dobjects(withtypically6degrees-of-freedom),
levels)wewereabletocomparetwointeractiontechniques.
makes3DUIusuallydifficulttoassessanddesign. Werepro- Thereliabilityofmentalstatesmeasuresisstronglycorrelated
ducedpartofthisproblematicwithourgameenvironmentand to the quality of EEG signals. Interestingly enough, partic-
obtainedcoherentresultsfromEEGmeasures. ipants’ mindset during the recordings is one of the factors
influencingEEGsignals. Theirawarenessandinvolvement
towardthetasksimprovesystemaccuracy.Theformofthecal-
KEYBOARD TOUCH
ibrationtaskscouldbeenhancedtoengagemoreusers,forex-
amplethroughgamification[10]–andourvirtualenvironment
x 0.25
e provedtobesuitabletodoso. Whereasourparticipantswere
nd Difficulty volunteersenrolledamongstudents,intheendtheoutcomeof
ad i 0.00 EMAESDYIUM anevaluationmethodbasedonEEGshouldbestrengthened
o HARD
Workl−0.25 ULTRA breyliraebclryuitthiengdidffeedriecnattecdontessttreurcst,sucsoinugldabseseelseticmtioatnedcrfitreormiathhoewir
EEGsignalsduringcalibrationtasks.
Finally,oneshouldacknowledgethatwhenitcomestorecord-
0 250 500 750 1000 0 300 600 900
Time (s) ingsassensitiveasEEG,artifactssuchastheonesinducedby
Figure9: Workloadindexovertimeforparticipant3–60ssmoothing muscularactivityareofmajorconcern. Thewayweprevented
window.Left:KEYBOARDcondition,right:TOUCHcondition.Back- theappearanceofsuchbiasinthepresentstudyisthreefold.
groundcolorrepresentsthecorrespondingdifficultylevel.
1)Thehardwareweused–activeelectrodeswithAg/AgCl
coating–isrobusttocablemovements,seee.g.,[34]. 2)The
Above all, an evaluation method based on EEG enables a classifiersweretrainedonfeaturesnotrelatedtomotionarti-
continuous monitoring of users. The intended use case of factsormotorcortexactivation. 3)Thepositionofthescreen
ourframeworkistoenrolldedicatedtestersthatwouldwear duringthe“touch”conditionminimizedparticipants’motion,
theEEGequipmentandperformwellduringthecalibration andgesturesoccurredmostlybeforethetimewindowusedfor
tasks. Asamatteroffact,thebestperformerduringworkload detectinginteractionerrors. Theseprecautionsareimportant
calibration(participant3inTable2)showspatternsthatclearly forthetechnologytobecorrectlyapprehended.
meet the expectations concerning both difficulty levels and
Tofurthercontrolforanybiasinourprotocol,weranabatch
interactions,aspicturedinFigure9.
of simulations where the labels of the calibration tasks had
beenrandomlyshuffled,similarlytotheverificationprocess
describedin[35]. Shouldartifactsbiasourclassifiers,differ-
LIMITATIONSANDFUTURECHALLENGES
ences would have appeared between the KEYBOARD and
AlthoughusingEEGmeasuresasanevaluationmethodfor
TOUCHconditionsevenwithsuchrandomtraining. Among
HCIwasprovenconclusiveregardingworkload–weobtained
the20simulationsthatranforeachofthe3constructs(work-
acontinuousindexonparwithagroundtruthbasedontra-
load, attention, error recognition), none yielded significant
ditionalquestionnaires–thetwootherconstructswestudied
differences.
couldbenefitfromfurtherimprovements.
Despitethedirectinteraction(TOUCH)beingmoredisorient- CONCLUSION
ingforusersthantheindirectone(KEYBOARD),therecog- Inthispaper,wedemonstratedhowbrainsignals–recorded
nitionofinteractionerrorsdifferedonlybyatendency. This bymeansofelectroencephalography–couldbeputintoprac-
couldbeexplainedbythefactthatthecalibrationtaskwastoo tice in order to obtain a continuous evaluation of different
dissimilartothevirtualenvironment.Notably,whiletherewas interactiontechniques,forassessingtheirergonomicprosand
fewandslowpacedeventsduringthecalibration,userswere cons. Inparticular,wevalidatedanEEG-basedworkloadesti-
confrontedtomanystimuliwhiletheywereplaying, hence matorthatdoesnotnecessitatetomodifytheexistingsoftware.
overlapping ERPs must have appeared within EEG, which Furthermore,weshowedhowusers’attentionlevelcouldbe
mayhavedisturbedtheclassifier. Acalibrationtaskcloserto evaluatedusingbackgroundstimuli,suchassounds. Finally,
real-lifescenariosthantheonedescribedin[9]shouldbeenvi- weinvestigatedhowtherecognitionofinteractionerrorscould
sioned. Suchtaskshouldremaingenericinordertofacilitate helptodeterminethebestuserinterface.
thedisseminationofEEGasanevaluationmethodforHCI.
Beingabletoestimatethesethreeconstructs–workload,atten-
Signalprocessingcouldalsofacilitatethetransferoftheclassi- tionanderrorrecognition–continuouslyduringrealisticand
ficationbetweenastandardtaskandtheevaluatedHCI.Indeed, complexinteractiontasksopenednewpossibilities. Notably,
ifourresultsdemonstratethattheEEGclassificationofwork- itenabledustoobtainadditionalandmoreobjectivemetrics
loadcouldbetransferredfromtheN-backtaskstoadissimilar ofuserexperience,basedontheusers’cognitiveprocesses. It
virtual environment and user interface, we benefited from alsoprovideduswithadditionalinsightsthattraditionalmea-
spatialfiltersthatspecificallytakeintoaccountthevariance sures(e.g.,behavioralmeasures)couldnotreveal. Tosumup,
betweencalibrationcontextsandusecontexts–thestationary thissuggeststhatcombinedwithexistingevaluationmethods,
subspace CSP [35]. Since ERPs may also slightly differ in EEG-basedevaluationtoolssuchastheonesproposedhere
amplitudesanddelaysbetweencalibrationandusecontexts, canhelptounderstandbettertheoveralluserexperience. We
inthefuture,itwouldbeworthdesigningsimilarapproaches hopethattheincreasingavailabilityofEEGdeviceswillfoster
tooptimizetemporalorspatialfiltersforERPsaswell. suchapproachesandbenefittheHCIfield.