Table Of ContentHuman-Computer Interaction Series
Editors-in-chief
JohnKarat
IBMThomasJ.WatsonResearchCenter(USA)
JeanVanderdonckt
UniversitécatholiquedeLouvain(Belgium)
EditorialBoard
GaëlleCalvary,LIG-UniversityofGrenoble1,France
JohnCarroll,SchoolofInformationSciences&Technology,PennStateUniversity,USA
GilbertCockton,NorthumbriaUniversity,UK
LarryConstantine,UniversityofMadeira,Portugal,andConstantine&LockwoodLtd,
Rowley,MA,USA
StevenFeiner,ColumbiaUniversity,USA
PeterForbrig,UniversitätRostock,Germany
ElizabethFurtado,UniversityofFortaleza,Brazil
HansGellersen,LancasterUniversity,UK
RobertJacob,TuftsUniversity,USA
HilaryJohnson,UniversityofBath,UK
KumiyoNakakoji,UniversityofTokyo,Japan
PhilippePalanque,UniversitéPaulSabatier,France
OscarPastor,UniversityofValencia,Spain
FabioPianesi,BrunoKesslerFoundation(FBK),Italy
CostinPribeanu,NationalInstituteforResearch&DevelopmentinInformatics,Romania
GerdSzwillus,UniversitätPaderborn,Germany
ManfredTscheligi,UniversitätofSalzburg,Austria
GerritvanderVeer,UniversityofTwente,TheNetherlands
ShuminZhai,IBMAlmadenResearchCenter,USA
ThomasZiegert,SAPResearchCECDarmstadt,Germany
Human-Computer Interaction is a multidisciplinary field focused on human aspects of the
development of computer technology. As computer-based technology becomes increas-
ingly pervasive – not just in developed countries, but worldwide – the need to take a
human-centered approach in the design and development of this technology becomes ever
more important. For roughly 30 years now, researchers and practitioners in computational
and behavioral sciences have worked to identify theory and practice that influences the
directionofthesetechnologies,andthisdiverseworkmakesupthefieldofhuman-computer
interaction.Broadlyspeaking,itincludesthestudyofwhattechnologymightbeabletodo
forpeopleandhowpeoplemightinteractwiththetechnology.
Inthisseries,wepresentworkwhichadvancesthescienceandtechnologyofdeveloping
systemswhicharebotheffectiveandsatisfyingforpeopleinawidevarietyofcontexts.The
human-computer interaction series will focus on theoretical perspectives (such as formal
approachesdrawnfromavarietyofbehavioralsciences),practicalapproaches(suchasthe
techniquesforeffectivelyintegratinguserneedsinsystemdevelopment),andsocialissues
(suchasthedeterminantsofutility,usabilityandacceptability).
Forfurthervolumes:
http://www.springer.com/series/6033
Regina Bernhaupt
Editor
Evaluating User Experience
in Games
Concepts and Methods
123
Editor
Asst.Prof.ReginaBernhaupt
UniversitéPaulSabatier,ToulouseIII
InstitutdeRechercheen
InformatiquedeToulouse(IRIT)
118RoutedeNarbonne
31062ToulouseCedex9
France
[email protected]
ISSN1571-5035
ISBN978-1-84882-962-6 e-ISBN978-1-84882-963-3
DOI10.1007/978-1-84882-963-3
SpringerLondonDordrechtHeidelbergNewYork
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
LibraryofCongressControlNumber:2009943127
©Springer-VerlagLondonLimited2010
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced,
storedortransmitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublishers,
or in the case of reprographic reproduction in accordance with the terms of licenses issued by the
CopyrightLicensingAgency.Enquiriesconcerningreproductionoutsidethosetermsshouldbesentto
thepublishers.
Theuseofregisterednames,trademarks,etc.,inthispublicationdoesnotimply,evenintheabsenceofa
specificstatement,thatsuchnamesareexemptfromtherelevantlawsandregulationsandthereforefree
forgeneraluse.
Thepublishermakesnorepresentation,expressorimplied,withregardtotheaccuracyoftheinformation
containedinthisbookandcannotacceptanylegalresponsibilityorliabilityforanyerrorsoromissions
thatmaybemade.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
ForPhilippe
Foreword
It was a pleasure to provide an introduction to a new volume on user experience
evaluationingames.Thescope,depth,anddiversityoftheworkhereisamazing.It
atteststothegrowingpopularityofgamesandtheincreasingimportancedeveloping
arange oftheories,methods,and scalestoevaluate them.Thisevolution isdriven
bythecostandcomplexityofgamesbeingdevelopedtoday.Itisalsodrivenbythe
need to broaden the appeal of games. Many of the approaches described here are
enabledbynewtoolsandtechniques.Thisbook(alongwithafewothers)represents
awatershedingameevaluationandunderstanding.
Thefieldofgameevaluationhastruly“comeofage”.ThebroaderfieldofHCI
canbegintolooktowardgameevaluationforfresh,critical,andsophisticatedthink-
ingaboutdesignevaluationandproductdevelopment.Theycanalsolooktogames
forgroundbreakingcasestudiesofevaluationofproducts.
I’ll briefly summarize each chapter below and provide some commentary. In
conclusion,Iwillmentionafewcommonthemesandoffersomechallenges.
Discussion
In Chapter 1, User Experience Evaluation in Entertainment, Bernhaupt gives an
overview and presents a general framework on methods currently used for user
experience evaluation. The methods presented in the following chapters are sum-
marizedandthusallowthereadertoquicklyassesstherightsetofmethodsthatwill
helptoevaluatethegameunderdevelopment.
In Chapter 2, Enabling Social Play: A Framework for Design and Evaluation,
Isbisterexaminestheneglectedareaofsocialgames.Shereviewstheliteratureand
considers sample games and some of the primary factors affecting the experience
of social gaming. These include contextual factors, motivational factors, and the
conceptual andtheoreticalgroundwhichshapesuchevaluations.Isbisterpointsto
ecological validity being a primary focus when considering any testing method.
She stresses the importance of looking at the entire set of measures including
attitudinal and behavioral data. Her chapter creates a context for planning and
conductingevaluationsofsocialgames.
vii
viii Foreword
Chapter 3, Presence Involvement and Flow in Digital Games (Takatalo,
Hakkinen, Kaistinen, and Nyman) is a report on a massive study using multivari-
ateanalysistorevealthesubcomponentsofuserexperienceingames.Itreviewsthe
historyofclassificationsofuserexperiencegoingbacktotheclassicpsychological
categories ofthinking,feeling,andwill.Theauthorscompareandanalyze current
concepts:immersion,fun,presence,involvement,andflow.Theyalsoreviewprevi-
ousmultivariateanalysisofgamequestionnaires.Fromthisanalysis,theydevelop
thePresence-Involvement-Flowframework(PIFF)whichencompassesbothtechni-
calgamecomponentsandpsychologicaldeterminantofUX.Usingafactoranalytic
study, they evaluate this model. The final questionnaire was assessed by compar-
ing two different groups of gamers in two different games. In addition, the profile
for games was compared to Metacritic scores and user ratings. The results were
promising, with results of PIFF accounting for the important differences between
gamesandilluminatingthelearningcurvesofdifferentusersduringthefirsthourof
play.PIFFisverybroadinscopeandshowspotentialtoevaluateuserexperienceat
differentstagesofdevelopment.
InChapter4,AssessingtheCoreElementsoftheGamingExperience,Calvillo-
Gamez,Cairns,andCoxassumeanambitiousgoal–creatinganewtheoryofuser
experience for games. They adopt a novel process to achieve this end. They have
chosen to use a bottom-up approach. They build the theory by analyzing video
gamereviewsusingthemethodologyderivedfromgroundedtheory.Theiranalysis
iscomplexcontainingtwocoreelements:puppetryandvideogame,eachwiththree
elements. They term this theory the Core Elements of Game Experience (CEGE)
theory. This theory is integrated into a model and translated into a questionnaire.
The questionnaire is then applied to user experience in two versions of a popular
game (Tetris). The results are as predicted, i.e., scales show no difference for ele-
ments of thegame thatwerenotdifferent(visuals and sound) but aredifferentfor
thosethatrelatetocontrol–puppetry.Theambitiousgoalthatwasproposedatthe
beginningofthechapterismatchedbyanimpressiveresult–validationofatheory
for gaming. Since the theory includes measurement instruments, it represents an
advanceoverprevioustheories,inwhichthemeasurementisunspecified.
Emily Brown reports on the current practice in the gaming industry about the
LifeandToolsofaGameDesigner(Chapter5).
InChapter6,InvestigatingExperiencesandAttitudesTowardVideogamesUsing
a Semantic Differential Methodology, Lemay and Lessard explore the use of the
semanticdifferentialtoevaluatevideogamesasleisureactivities.Theyofferanew
toolforresearchonperceptionofgames.Astheypointout,havingastandardsetof
questions and metrics for attitudes about games would provide a foundation for
more research. This also could provide a foundation for analyzing the attitudes
towardpotentialmarketsegmentsforagame.Theirresultsarepromisingandcould
helpgamedesignersunderstandtheirintendedaudiences.
In Chapter 7, Video Game Development and User Experience, McAllister and
White provide a detailed overview of the testing done for three products. The
chapter gives a snapshot of how evaluation is currently being done in industry.
The methods and procedures are quite broad. The first case study recounts the
Foreword ix
development of Pure by Black Rock Studio. In that case study, three useful dis-
tinctionsintestmethodologyareintroduced.Theseareasfreeflow(playthegame
asyounormallywould)vs.narrowspecific(playonlyapartofthegameandplay
itrepeatedly)vs.broadspecific(userplaysmoreofthegamebutplaysrepeatedly).
Whilethesethreetypesoftestsareusedwidely,namingthemisastepforward.In
thesecondcasestudy,thedevelopmentofZoëModeillustratessomebasictruisms
oftesting,suchastheneedformultipleiterativetestsnotonlytorefinebutalsoto
developacompletelyalternativeandmoreeffectiveapproach–inthiscase,chang-
ingfromtextdescriptionstovisualimagestoillustratesuccessfulcalibration.The
third and final case study examines Relentless Software’s approach to evaluating
Buzz!QuizTV.Focusgrouptestingwasusedtocollectuserratingsofthegame.In
addition, telemetry from the game was captured remotely postlaunch (a technique
alsousedbyBlackRockStudio).Theauthorsseefuturedevelopmentsascapturing
moredatasuchasfacialexpressionsandphysiologicaldata.
InChapter8,UserExperienceDesignforInexperiencedGamers:GAP–Game
ApproachabilityPrinciples,DesurvireandWibergprovideaninsightfulanalysisand
usefulguidancetodesignstudiosthataredevelopinggamesforbroadaudiences.A
classic challenge that these studios face is creating a game that is “approachable”.
Thatis,thegamemustbeaccessibletonovicesandholdtheirinterestsothatthey
willpurchase itand continue to play it.Traditionally, game studios emphasize the
“playability”oftheirgames.Playabilityisusuallytakentomeanthegameisgood
for experienced players. While traditional methods like the members of the studio
playingthegameortestingforbalancebyaqualityassuranceteammightproduce
a “playable” game (good game balance, few bugs, elimination of “golden” paths,
etc.), “approachability” is often a bigger challenge since it addresses the needs of
newornoviceplayers.Desurvireetal.addressthisproblemusingtwo“traditional”
usabilitymethods:usabilitytesting(UT)andheuristicevaluation(HE).Theyenrich
theHEwithGameApproachabilityPrinciples(GAP).Heuristicevaluationisonly
asgoodastheprinciplesthatdriveitandtheexperienceoftheevaluators.Intheir
chapter, Desurvire et al. describe a study that tested several games using both HE
and UT. They found that using the methods together produced actionable insights
either method alone would not have uncovered. Their approach is unique, since
mostoftheresearchliteraturehasattemptedtoevaluatewhichmethodisbest.This
chapter shows that the combination of UT and HE produces more “value” for the
designstudio.Valueisdefinedasthemostactionabledatafortheinvestmentmade.
Anystudiohopingtoproduceanapproachablegamewouldbewisetoincorporate
bothHEandUTintotheirtoolkit.
In Chapter 9, Digital Games, the Aftermath, Poels, IJsselsteijn, de Kort, and
VanIerselexplorearelativelyunchartedareaofgaming–theshort-andlong-term
effects of the game on the player. While most past studies have focused on single
possible effects of gaming, e.g., gaming leads to isolation or gaming desensitizes
one fromviolence, Poels looks atmoreglobaleffects.The resultsshowboth tem-
poraryandlong-lastingeffectsthatarenotdifferentfromexposuretoothermedia
suchasmoviesorbooks.Theseresultsarenotunexpected,buttheirimplicationsare
consistentwithasignificantshiftintheconceptualizationofpersonality.Thatis,the
rejectionofafixed,trait-likeconceptionofpersonalityinfavorofamoremalleable
