Table Of ContentThe Wiley Handbook of
Human Computer Interaction
The Wiley Handbook of
Human Computer
Interaction
Volume 1
Edited by
Kent L. Norman and Jurek Kirakowski
This edition first published 2018
© 2018 John Wiley & Sons Ltd
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise,
except as permitted by law. Advice on how to obtain permission to reuse material from this title is
available at http://www.wiley.com/go/permissions.
The right of Kent L. Norman and Jurek Kirakowski to be identified as authors of the editorial material in
this work has been asserted in accordance with law.
Registered Office(s)
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
Editorial Office
The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
For details of our global editorial offices, customer services, and more information about Wiley products
visit us at www.wiley.com.
Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand. Some content
that appears in standard print versions of this book may not be available in other formats.
Limit of Liability/Disclaimer of Warranty
While the publisher and authors have used their best efforts in preparing this work, they make no
representations or warranties with respect to the accuracy or completeness of the contents of this work
and specifically disclaim all warranties, including without limitation any implied warranties of
merchantability or fitness for a particular purpose. No warranty may be created or extended by sales
representatives, written sales materials or promotional statements for this work. The fact that an
organization, website, or product is referred to in this work as a citation and/or potential source of
further information does not mean that the publisher and authors endorse the information or services the
organization, website, or product may provide or recommendations it may make. This work is sold with
the understanding that the publisher is not engaged in rendering professional services. The advice and
strategies contained herein may not be suitable for your situation. You should consult with a specialist
where appropriate. Further, readers should be aware that websites listed in this work may have changed
or disappeared between when this work was written and when it is read. Neither the publisher nor
authors shall be liable for any loss of profit or any other commercial damages, including but not limited
to special, incidental, consequential, or other damages.
Library of Congress Cataloging‐in‐Publication Data Is Available
9781118976135 – Hardback
9781118977262 – e‐PDF
9781118977279 – e‐Pub
Cover Design: Wiley
Cover Image: © Andrea Danti/Shutterstock
Set in 10/12pt Galliard by SPi Global, Pondicherry, India
10 9 8 7 6 5 4 3 2 1
Contents
Notes on Contributors vii
Acknowledgments xix
Introduction: Human‐Computer Interaction Yesterday,
Today, and Tomorrow 1
Kent L. Norman and Jurek Kirakowski
Part I Design Issues 7
1 Interactive Critical Systems and How to Build Them 9
Harold Thimbleby
2 Semiotics and Human‐Computer Interaction 33
Clarisse Sieckenius de Souza
3 Benefiting from ISO Standards 51
Nigel Bevan and Jonathan Earthy
4 Gender and Human‐Computer Interaction 71
Samantha Breslin and Bimlesh Wadhwa
5 Usability and Digital Typography 89
Peter Flynn
Part II Design Process 109
6 Agile User‐Centered Design 111
Gabriela Jurca, Theodore D. Hellmann, and Frank Maurer
7 Ethnographic Approach to Design 125
Dave Randall and Mark Rouncefield
8 User Modeling 143
Pradipta Biswas and Mark Springett
9 Kids and Design 171
Mona Leigh Guha and Jerry Alan Fails
vi Contents
Part III Evaluation Factors 191
10 User Experience 193
Jakob Grue Simonsen
11 Task Load and Stress 207
Julien Epps
12 Comparing Mobile Experience 225
Xiaoge Xu
13 Factors of Immersion 239
Noirin Curran
Part IV Evaluation Methods 255
14 Usability Testing 257
Sirpa Riihiaho
15 Remote Usability Testing 277
John Black and Marc Abrams
16 Applied User Research in Games 299
Randy J. Pagulayan, Daniel V. Gunn, Jerome R. Hagen,
Deborah J. O. Hendersen, Todd A. Kelley,
Bruce C. Phillips, J. J. Guajardo, and Tim A. Nichols
Part V Input / Output 347
17 Fitts’ Law 349
I. Scott MacKenzie
18 Principles for Designing Body‐Centered Auditory Feedback 371
Ana Tajadura‐Jiménez, Aleksander Väljamäe,
Frédéric Bevilacqua, and Nadia Bianchi‐Berthouze
19 Input Device—Motion Capture 405
Atsushi Nakazawa and Takaaki Shiratori
20 Applications of Intelligent and Multimodal Eye‐Gaze
Controlled Interfaces 421
Pradipta Biswas and Pat Langdon
Notes on Contributors
Marc Abrams serves as Harmonia’s president and chief technical officer. He provides
technical and business leadership to the company and manages all its technical pro-
jects. In the past, Dr. Abrams has been with the former U.S. Army Concepts Analysis
Agency, a postdoc in the Distributed Operating Systems group in Stanford’s Computer
Science Department, and a visiting scientist in the network protocol group at IBM’s
Zurich Research Laboratory in Switzerland. He has been the principal investigator for
over $30 million in research and development projects with the Air Force, the Army,
DARPA, DHS, DOE, DOT, NASA, the Navy, NIH, NSF, MDA, ONR, OSD, and
various companies including General Dynamics, IBM, Northrop Grumman,
Raytheon, and Leidos. He received his PhD from the University of Maryland at
College Park in computer science. Before Harmonia, Dr. Abrams was a tenured asso-
ciate professor at Virginia Tech, where his research on human‐computer interfaces
(HCI) led to the creation of User Interface Markup Language (UIML) and later the
co‐founding of Harmonia. UIML forms the basis for Harmonia’s LiquidApps®
product. At Virginia Tech, he also co‐founded the Center for Human Computer
Interaction, and worked with the HCI faculty in fields ranging from cognitive psy-
chology to human factors on scenario‐driven HCI design.
Nigel Bevan is an independent user experience (UX) consultant with wide industrial
and research experience. He has been the editor of several international standards
including both the original and revised versions of ISO 9241‐11 (usability), 9241‐210
(human‐centered design processes), 25010 and 25022 (software quality model and
measures), 20282‐2 (usability test method), and 25063 (context of use). He has
authored over 80 publications and was a member of the U.S. National Academy of
Science Committee on Human‐System Design Support for Changing Technology.
Frédéric Bevilacqua is the head of the Sound Music Movement Interaction team at
Institute for Research and Coordination in Acoustics/Music (IRCAM) in Paris,
which is part of the joint research lab Science and Technology for Music and Sound
(IRCAM—CNRS—Université Pierre et Marie Curie). He received his PhD in
Biomedical Optics from EPFL (Swiss Federal Institute of Technology in Lausanne),
in 1998. His research concerns the modeling and the design of interaction between
viii Notes on Contributors
movement and sound, and the development of gesture‐based interactive systems.
With his team, he developed several digital musical instruments such as the augmented
violin and the modular musical objects (First Prize of the Guthman Musical Instrument
Competition), and developed several systems to synchronize motion to sound, such
as the gesture follower. He coauthored more than 120 scientific publications and
coauthored five patents. He was keynote or invited speaker at several international
conferences such as the ACM TEI’13. As the coordinator of the “Interlude Project”
he received the ANR Digital Technology Prize (Societal Impact) in 2013.
Nadia Bianchi‐Berthouze is a full professor in affective computing and interaction at
the Interaction Centre of the University College London (UCL). She received her
PhD in computer science for biomedicine from the University of the Studies of Milan,
Italy. Her research focuses on designing technology that can sense the affective state
of its users and use that information to tailor the interaction process. She has pio-
neered the field of affective computing and for more than a decade she has investi-
gated body movement, and more recently touch behavior, as a means to recognize
and measure the quality of the user experience in full‐body computer games, physical
rehabilitation, and textile design. She also studies how full‐body technology and body
sensory feedback can be used to modulate people’s perception of themselves and of
their capabilities to improve self‐efficacy and copying capabilities. She has published
more than 170 papers in affective computing, HCI, and pattern recognition. She was
awarded the 2003 Technical Prize from the Japanese Society of Kansei Engineering
and she has given a TEDxStMartin talk (2012).
Pradipta Biswas is an assistant professor at the Centre for Product Design
and Manufacturing of the Indian Institute of Science. His research focuses on user
modeling and multimodal human‐machine interaction for aviation and automotive
environments and for assistive technology. He set up and leads the Interaction Design
Lab at CPDM, IISc. Earlier, he was a senior research associate in the Engineering
Department, a research fellow at Wolfson College, and a research associate at Trinity
Hall of the University of Cambridge. He completed PhD in computer science at the
Rainbow Group of the University of Cambridge Computer Laboratory and Trinity
College in 2010, and was awarded a Gates‐Cambridge Scholarship in 2006. He under-
took a masters degree at the Indian Institute of Technology, Kharagpur. He con-
ducted a course on HCI at the Indian Institute of Technology, Mandi, gave a guest
lecture at the Indian Institute of Technology, Madras, and was a vice chairman of
ITU‐T Focus Group on Smart TV.
John Black, after a youthful career as a fine artist, doing painting and sculpture, has
been developing software for more than 30 years. He has coded using a vast range of
programming languages, from Z‐80 assembly language to Prolog. He has founded
and run several startup businesses developing business software and worked at large
international corporations, such as Thomson Reuters, where he was both a software
architect and a software development manager. Always striving to keep abreast of new
technologies, he has run full‐scale Bitcoin and Ethereum block chain nodes for several
years, and worked with the Ethereum source code. As part of a commitment to fur-
thering software standards efforts, he has worked with the Object Management
Group (OMG, a worldwide standards organization) on a standard for employee
Notes on Contributors ix
time‐recording data and worked with the W3C during the development of the
resource description framework (RDF) standard for use on the Semantic Web. In
2006, he wrote the paper, “Creating a common ground for URI meaning using
socially constructed Web sites.”
Samantha Breslin is a PhD candidate in the Department of Anthropology at the
Memorial University of Newfoundland (MUN). Trained initially as a computer
scientist at the University of Waterloo, she then completed a master’s degree in
anthropology at MUN. Samantha’s doctoral research is an ethnography of under-
graduate computer science education in Singapore, exploring the “making” of com-
puter scientists as (trans)national citizens and subjects in relation to computer science,
world making, entrepreneurialism, and gender. Alongside this research, she has been
working with Dr. Bimlesh Wadhwa at the National University of Singapore towards
developing a gender and HCI curriculum to enable undergraduate computing
students to explore how gender—and values more generally—are embedded in their
programs, designs, and practices.
Noirin Curran received a PhD in applied psychology from University College Cork,
and within psychology, her specialized area of interest is immersion in games. As part
of her doctoral work, she used rigorous psychometric procedures to create the IMX
Questionnaire, which measures the level of immersive response as experienced in a
game session. Being fascinated by the experiences that can be prompted by modern
media and technology, her current work in the game industry falls under the banner
of HCI, where she investigates what game players do, and why, and promotes user‐
centered design based approaches. She has had the opportunity to carry out research
involving a variety of advanced research methods, statistical work, and human‐factors
and usability‐based methodologies in both academic and industry settings.
Clarisse Sieckenius de Souza is a full professor of the Department of Informatics of
the Pontifical Catholic University of Rio de Janeiro (PUC‐Rio). She received her
doctorate in applied linguistics in 1987 from PUC‐Rio’s Department of Letters,
with a thesis in natural language processing (NLP). In 1988, she joined the
Department of Informatics, first involved in teaching and research on NLP and text
generation and soon starting her lifetime work in semiotics and HCI. Clarisse is the
creator of semiotic engineering, a semiotic theory of HCI for which, along with her
colleagues and students, she developed specialized methods and models for interac-
tion design. In recognition to her contribution to the field, she received the ACM
SIGCHI CHI Academy Award in 2013 and the IFIP TC13 Pioneers of HCI Award
in 2014. Her work has been published in over a hundred papers and she is the author
or co‐author of four books on semiotic engineering: The Semiotic Engineering of
Human‐Computer Interaction (2005); Semiotic Engineering Methods for Scientific
Research in HCI (2009); A Journey through Cultures: Metaphors for Guiding the
Design of Cross‐Cultural Interactive Systems, and Software Developers as Users.
Semiotic Investigations on (2013) Human‐Centered Software Development (2016).
Jonathan Earthy works for Lloyd’s Register where he coordinates the introduc-
tion of human factors and a human‐centered approach into its products and the
marine industry in general. His technical specialty is the assurance of the quality of
x Notes on Contributors
human‐centered design. He is an adjunct associate professor at the Australian
Maritime Academy. He has participated in ergonomics and systems standards devel-
opment since the mid‐1980s and is convener of ISO TC159/SC4/WG6 human‐
centered design for interactive systems.
Julien Epps is an associate professor of signal processing at the School of Electrical
Engineering and Telecommunications at the University of New South Wales (UNSW),
Australia. He is also a contributed researcher with Data61, CSIRO, Australia. He
holds a PhD (2001) in signal processing, and is the author or coauthor of over 200
journal articles, refereed conference papers and book chapters. He is currently is serv-
ing as an associate editor for IEEE Transactions on Affective Computing and for the
human‐media interaction section of Frontiers in ICT and Frontiers in Psychology, and
is a member of the advisory board of the ACM International Conference on Multimodal
Interaction. His primary research areas are speech and behavioral signal processing,
cognitive workload, emotion and mental state recognition, and machine learning for
human‐computer interaction.
Jerry Alan Fails is an associate professor in the Computer Science Department at
Boise State University in Boise, Idaho. His primary area of research is Human-
Computer Interaction, with a focus on technologies that promote children’s creativity,
activity, mobility, collaboration, and exploration of the world around them. He has
been actively designing technologies with and for children utilizing—and further
developing—participatory design methods for children since 2003.
Peter Flynn manages the Academic and Collaborative Technologies Group in IT
Services at University College Cork (UCC), Ireland. He trained at the London
College of Printing and did his MA in computerized planning at Central London
Polytechnic (now the University of Westminster). He worked in the United Kingdom
for the Printing and Publishing Industry Training Board as a DP manager and for the
United Information Services of Kansas as an IT consultant before joining UCC as
project manager for academic and research computing. In 1990, he installed Ireland’s
first Web server and now concentrates on academic and research publishing support.
He has been Secretary of the TeX Users Group, deputy director for Ireland of
European Academic and Research Network (EARN), and a member both of the
Internet Engineering Task Force ( IETF) Working Group on HTML and of the W3C
XML SIG; and he has published books on HTML, SGML/XML, and LaTeX. Peter
also runs the markup and typesetting consultancy, Silmaril, and is editor of the XML
FAQ as well as an irregular contributor to conferences and journals in electronic pub-
lishing, markup, and humanities computing, and a regular speaker and session chair at
the XML Summer School in Oxford. He did his PhD in user interfaces to structured
documents with the Human Factors Research Group in Applied Psychology in UCC.
J. J. Guajardo has over 15 years of experience as a User Researcher. After earning a
bachelor’s degree in psychology from Northwestern University, he received his PhD
in developmental psychology from the University of Chicago in 2002. Immediately
after, J. J. came to Microsoft to work on Xbox games. Following his stint in the
gaming world, he worked with a number of products at Microsoft, including Encarta
and Office for Mac. From 2007–2009, J. J. lived and worked in Copenhagen,
Notes on Contributors xi
Denmark, conducting user research for Microsoft Dynamics. In 2009, he returned to
the United States and spent 2 years working in the Windows design and research
group. In 2011, he returned to Xbox Research to work on kid‐focused products and
nongame entertainment efforts. He currently supports the Turn 10 franchise, working
on the latest versions of the premier racing titles Forza Motorsport and Forza Horizon.
Mona Leigh Guha is the director of the University of Maryland’s Center for Young
Children. She has also been the interim director of the University of Maryland’s
Human‐Computer Interaction Lab (HCIL), as well as managing director of KidsTeam,
a team of adults and children who work together to design innovative technology for
children. Her research has focused on working with young children as design partners
and investigating the cognitive and social experiences of children who participate on
a design team.
Daniel V. Gunn is a senior user research operations lead within Xbox Research. His
team is responsible for driving the tech and facilities, people and process, and tools and
infrastructure that empower world‐class user research. Daniel’s background is firmly
seated in psychological research methods, statistics, and human behavior. He received
his PhD in experimental psychology with an emphasis on human factors from the
University of Cincinnati (UC) in 2002. Daniel has presented at several human‐factors‐
related conferences and has published in the American Journal of Psychology as well as
the Journal of the Human Factors and Ergonomics Society. In addition, he has coau-
thored articles and book chapters on the methodologies utilized within Xbox Research
to improve games in development. He has worked on several Microsoft Studios titles
across a variety of genres and platforms including installments in the Forza Motorsport
series across Xbox and Xbox 360 as well as PC titles such as Rise of Nations: Rise of
Legends and Viva Piñata PC. Daniel is always looking for ways to generate new types
of insight for game designers leveraging innovative research methods.
Jerome R. Hagen is a senior user researcher in Xbox Research at Microsoft. He cur-
rently leads research on Minecraft and has led research on game franchises including
Halo, Fable, Crackdown, Project Gotham Racing, and Phantom Dust. His back-
ground is in social / cognitive psychology and he also leads training for researchers on
the Xbox team. He has led Team Xbox LGBTQ and is part of Xbox’s focus on Gaming
for Everyone to help make Xbox a place where everyone is welcome, respected, and
supported.
Theodore D. Hellmann is a product manager at Splunk, where he works to support
and expand its developer ecosystem. His work focuses on making sure third‐party
developers are provided with the tools and guidance to build Splunk Apps that ingest
and store huge amounts of data, then make that data easy to use and understand. In
his previous life in academia, he was a member of the Agile Surface Engineering Lab
at the University of Calgary, where his research interests included test‐driven develop-
ment of graphical user interfaces, interactive and proxemic emergency operations
planning, and interaction with / visualization of large‐scale data.
Deborah J. O. Hendersen is a senior user researcher working on Xbox research at
Microsoft. She received her doctorate from Stanford University in 2008 in cognitive
