Things That Keep Us Busy: The Elements of Interaction

317 Pages·2017·0.855 MB·English

by  Lars-Erik Janlert, Erik Stolterman

Acknowledgments
The work on this book has been ongoing for more than twenty-five years.
Our engagement with the topic has sometimes been more focused and
at other times more in the background. Some of the ideas had already
emerged when we were PhD students, even though most of them have
been developed during the last five to ten years. The work we present here
is of a kind that does not necessarily require labs, students, equipment, or
any other resources except for time. Of course, there are students who have
influenced our ideas by taking our courses, reading our drafts, and giving
us feedback. To all of them we are grateful. There are also a small number of
colleagues we invited to read parts of this work who have really helped us.
We are particularly thankful for the comments we have received from Jonas
Löwgren and Johan Redström. We also want to thank Omar Sosa Tzec who
has done a great job in helping us with the design of our figures.
This book is composed partially of material from earlier published arti-
cles. The text from these articles has been rearranged and altered, edited
and rewritten:
Janlert, L.-E., and E. Stolterman. 1997. “The Character of Things.” Design Studies 18
(3): 297–314.
Janlert, L.-E., and E. Stolterman. 2010. “Complex Interaction.” ACM Transactions on
Computer-Human Interaction 17 (2): article 8.
Janlert, L.-E., and E. Stolterman. 2015. “Faceless Interaction—A Conceptual Exami-
nation of the Notion of Interface: Past, Present, and Future.” Human–Computer
Interaction 30 (6): 507–539.
Janlert, L.-E., and E. Stolterman. 2017. “The Meaning of Interactivity—Some Propos-
als for Definitions and Measures.” Human–Computer Interaction 32 (3): 103–138.

1 The Things That Keep Us Busy
Despite strong misgivings, private eye Eddie Valiant eventually ventures into the
city of Toontown (in Who Framed Roger Rabbit, 1988). It is a truly nerve-
racking experience: everything is throbbing with life, nervously responsive to
his every move, incessantly calling for his attention—not just the usual toon ani-
mals, but plants, cars, buildings, everyday things like the elevator button—even
the bullets in his toon revolver are alive. Everything is on speed as it were, inces-
santly making faces, quipping, jesting, collectively whipping up the environment
into a bedlam of interactions. Toontown, the viewer soon realizes, is a madhouse
where you would not maintain your sanity for long.
Is this our future?
Even though there are early examples of amazing constructions and
machines with interactive abilities, for most people, everyday interaction
with technical devices is a fairly recent phenomenon brought about by the
modern revolution in information technology. An avalanche of interactive
devices, artifacts and systems has followed in its path. With this change
come new questions and challenges.
It is hard to deny that our artifacts and environments are becoming more
and more complex, more and more “alive,” and as a consequence in many
cases more and more demanding. We have to interact more. Interactivity
seems to be everywhere. Why is this happening? There are of course many
answers to this question, some short and simple. Because it can be done:
One obvious cause is the extraordinary and powerful development of digital
technology that makes it possible to complexify and infuse everything in
our environment with computational and interactive capabilities. Because
we want it: Interactivity brings on many benefits that we would not want to
be without. Interactivity changes our everyday environments in ways that
previous generations would have seen as science fiction or magic. Today
2 Chapter 1
we are able to interact in advanced ways with a range of diverse artifacts
and systems, from the smallest device to our homes and with our environ-
ments. Interactivity promises that we can be in control of our lives and that
we can shape them in any way we desire.
These days, everybody seems to be talking confidently and comfortably
about interaction—you interact with web services, with apps, appliances,
vehicles, and any form of technical equipment, but also with people, and
even entire environments. To be interactive is generally considered good—a
positive feature or property associated with being modern, efficient, fast,
flexible, reasonable, dynamic, adaptable, controllable—perhaps even smart,
curious, caring, involved, engaged, informed, and democratic. Still, there
seems to be no very precise idea of what interaction is and what being inter-
active means, beyond a vague notion that it is some kind of interplay, usu-
ally optimistically understood as good-natured cooperation. This vagueness
would not be very surprising if it were just the idea of the general public,
but even among researchers and experts in human–computer interaction
(HCI)—a field of research and development with “interaction” in its very
heading—a deep, crisp, shared understanding is wanting. In this chapter,
we will stay close to the inclusive everyday understanding of “interaction.”
In the following chapters, we will approach a narrower and more precise
definition of “interaction” focused on interaction with digital artifacts and
systems, while still keeping in touch with and drawing inspiration from the
broader everyday sense.
The closely related notion of interface, which has a more technical fla-
vor in everyday parlance, has similar problems of depth, preciseness, and
shared understanding. Yet, it has over the years attracted more attention
than interaction from researchers and developers. From a design point of
view, this is understandable. Interaction, whatever exactly it is thought to
be, is something fluid, a dynamic relation played out in time, in use time,
not design time—whereas the interface appears as a stable property of the
artifact or system, which is there also when no user interaction is going on,
hence directly accessible to the designer at design time.
For these reasons, the interface appears more designable than the inter-
action. Even if a designer is really focusing on designing the interaction, it
is hard to see how this design can be effectively implemented except indi-
rectly via the design of the interface. To some extent, it is of course possible
to influence the user through education, training, or seeding behavioral
The Things That Keep Us Busy 3
patterns, for example, but this path to shaping interactions is not as direct
nor usually as potent as the concrete design of an interface and we will not
investigate it further in this book. Our examination of interactivity will
rather take as its point of departure the interface itself and how the way
we think about it has radically changed over time, from being a physical
surface with knobs and dials, to clickable symbols, to gestures, and finally
to its disappearance.
Today we can interact with some artifacts and environments without
there being any visible surface presenting controls or displays of any kind.
It is obvious that even if there is no interface, we still interact. We open
doors just by walking toward them, we turn on the light by clapping our
hands, we get the weather forecast spoken to us by just asking for it, the red
light turns green triggered by our car, and so on. Of course, as soon as we
move toward interaction without any visible surface the questions of what
an interface is and what interaction is become more complex.
This development combined with an ability and desire for interactivity
fosters a common feeling that the level of interactivity will just keep ris-
ing, inexorably. We feel that there is more interactivity, more interaction
between humans and things going on, than ever before in history, and it
just keeps increasing. There seems to be no retreat or escape from interactiv-
ity. Some well-informed critics worry that the proliferation of interactions
and interactive things has already gone too far. Their concerns raise many
questions. Does interactivity in fact increase? How can we know? What
does it really mean to claim that it does? And if indeed it is increasing, what
does it mean? Should something be done?
To be able to answer any of these questions requires a more careful and
penetrating examination of the concepts of “interactivity” and “interac-
tion” than has been common in research on human–computer interaction.
We believe that the answer to questions about rising interactivity and how
it affects us humans is not just a matter of belief and conviction about the
overall nature of technology and its influence, or how we experience it on
a personal and social level. We think it requires a careful investigation into
the aspects of artifacts and systems that causes interactivity with a purpose
to develop some common understanding that in turn can inform our opin-
ions and positions. This is also the purpose and ambition of this book.
But before we enter into such examinations, let’s first take a closer look
at some of the concerns that recently have been recognized in relation
4 Chapter 1
to the proliferation of interactivity, concerns that taken together paint
a picture with a lot of unknowns—unknowns that have inspired our
examinations.
1.1 Haunted by Interactivity
A common expectation seems to be that people over time will become
increasingly engaged in interactions as a result of increased exposure to
a growing number of interactive artifacts and systems. The sheer number
of interaction possibilities would allow us to interact more than with ear-
lier, less interactive technology, and perhaps not just allow us, but compel
or even force us. Some see this evolution as moving toward a situation
not unlike Toontown. They experience interactivity as overwhelming and
unwanted, maybe even haunting, and as a consequence as something that
needs to be limited, reduced, and maybe even removed.
Even if the number of interactive artifacts and systems did not increase
further, and the range of interaction possibilities offered by the different
artifacts and systems did not expand further on average, it seems quite pos-
sible that interactive artifacts and systems might still increase interactivity
by becoming more and more efficient in engaging people in interactions.
Or, could it be that interaction possibilities and demands on our attention
may start canceling each other out when they come together in greater
numbers?
As everyday users of interactive technology, we may sometimes feel that
we have become the unfortunate battleground for an increasing number
of artifacts and systems competing for a share of our attention (which we
assume is a limited resource). Is there any end in sight for this “war of atten-
tion”? This concern has been addressed in some recent writings in relation
to the concepts of conversation (Turkle 2015), attention (Birkerts 2011, 2015),
distraction (Crawford 2015) and focus (Goleman 2013), ambient commons
(McCullough 2013), and also in a slightly different context with the notion
of attention economy (Goldhaber 1997; Davenport and Beck 2001). We see
these writings as signs that there are a lot of thoughtful scholars who are
concerned with the development, and even though they do not make same
arguments they all argue that it will have serious consequences.
As a response to increasing interactivity, people will develop resistance
and counter-strategies to uninvited interaction attempts. If interacting is
The Things That Keep Us Busy 5
acting to satisfy different “goals” or “requests,” that is, striving toward some
temporary or final closure or satisfaction, what then should we call the act
of refusing engagement, avoiding taking part, mutely resisting attempts at
being interacted with? Let us call it disinteracting.
The existence and conditions of, as well as strategies for disinteracting
with people in social life are well known and investigated (Simmel 1903;
Goffman 1963). There are some modern-day reports of individuals’ attempts
to escape interactivity in different ways (Brende 2005; Sengers 2011)
and there are social experiments exploring the attraction and maybe
addiction to interactivity. For instance, in several studies, teenagers have
been asked to stay away from interactive devices for a week or so to see
what it means to live without digital interactions. The removal (or threat
of removal) of interactivity in the form of games or smartphones has also
become a common disciplinary tool for parents raising their children.
Absence of interaction can apparently be seen as both a blessing and a
punishment.
However, it is not clear what disinteracting really means. You could argue
that to disinteract is still a form of interacting (albeit disobligingly), that it
usually takes some effort not to respond, and that in some circumstances,
silence speaks loudest. Also, could it be that disinteracting in one situation
leads to increased interaction in other settings? Could it be that the interac-
tion needed to achieve certain outcomes remains constant, and disinteract-
ing only means that interaction is moved or shifted in time or place?
One reason why some people feel haunted by interactivity and search
for ways to limit interaction is that it takes time. Are we spending more and
more time interacting with our artifacts and environments; are we more
often engaged in interactive behavior than we used to be? On the surface
the answer obviously is yes. People do spend more time interacting with
artifacts and systems today than just a few years ago and of course a lot
more than in earlier times when interactive artifacts were rare. But it may
be that the answer is not that simple.
Perhaps a parallel can be drawn with traveling. Do modern people spend
more time traveling than people in earlier and technically less developed
societies? It is a debated issue, but a number of researchers (Szalai 1972;
Zahavi 1979; Schafer and Victor 1997) have propounded the idea that the
time expenditure for travel is fairly constant regardless of the available
means of transportation, be it by foot, by canoe, commuter train, jet plane,
6 Chapter 1
or whatever. The more money and technical resources you have, the faster
and wider you tend to travel, but the average time spent on traveling seems
to remain about 1.1 hours per day. The empirical evidence for a constant
“time budget” may be inconclusive, but we can transfer the idea to the area
of interaction, and ask similar questions as transportation research has: If
time expenditure is constant, why is it constant? And if it increases, why
does it increase?
So, suppose there is a fixed time budget for “interaction,” meaning
that regardless of the technological level of our environment we tend to
spend roughly the same amount of time per day interacting. To make a
technology-independent assumption like that we must assume a broad
understanding of “interaction,” not restricted to “digital things” and
“human–computer interaction,” and include interaction with other kinds
of artificial systems, with people, animals, and possibly other natural enti-
ties. Suppose that the average interaction time budget in such an analysis
indeed comes out as fairly constant. What kind of theory could explain
that? As in transportation research we might look for biological, cultural,
and economic explanations.
For example, there could be biological reasons for humans to have a
certain daily, natural need of or limited capacity for interaction, so that
when that quota is satisfied or exhausted humans go into “noninteractive
mode” (it remains unclear what that means, but people do sleep on a regu-
lar basis). If this were true, then technology would have nothing to do with
it and we could simply reject the suggestion that interactivity is increasing.
Still, if you think that there is an important difference between interacting
with digital artifacts and systems, and interacting with people, you might
be more concerned about the proportion of artifact interaction to people
interaction, and so still not be reassured. The broad, inclusive, and literally
more natural notion of interactivity inevitably raises the question: What
do we do when we are not interacting? And also: Are we perhaps always
interacting, and is it more a matter of what kind of interaction? Then what
kind of interaction is haunting us?
1.2 Caught in Complexity and Clutter
Things are getting dauntingly complex. We find ourselves caught in webs
of complex interactions. How can we think about that? There are three
questions we need to look into more closely. The first is: Where exactly is
The Things That Keep Us Busy 7
it? Where is this complexity located that seems to bother, worry, or even
threaten us? The second question is: Why is it there? What are the causes
and reasons for its presence? The third question is: Does increasing interac-
tion complexity mean that we are gaining or losing control?
With good answers to these questions we would be in a better
position to assess the situation and see what can sensibly be done to dis-
entangle us. As we will see in the following chapters, there are no simple
answers, the questions are interrelated in complex ways, and with each
partial or tentative answer a number of follow-up questions call for further
investigation.
It is well known that time and again new technologies have led to new
shapes of interfaces and new styles of interaction. These in turn have led to
changes in the overall understanding and theorizing about interaction and
complexity in general. The notion of what might be seen as complex, natu-
ral, or intuitive interaction has changed when technology has changed.
Today, new digital artifacts and systems are again unsettling the received
notions of interface and interaction, in a two-pronged push. On the one
hand, we have the complexification—the proliferation of artifacts and sys-
tems with increasingly complex functions. On the other hand, we have a
surge of advances in interaction technology, that is, communication and
interface technology such as wireless connections, tracking and identifi-
cation techniques, new, smaller, smarter, cheaper sensors, actuators and
motors, new, larger but lighter, thinner, more flexible displays, so-called
smart materials, and more, all run by ever-more powerful and compact
computational technology. Neither of these effects have come suddenly,
there have been improvements and advances at a steady rate over a long
time, but it seems we have now come to a point where they are causing a
qualitative game change.
We believe that the question of complexity when it comes to interaction
cannot be answered by simply referring to particular developments of tech-
nology, applications, or interfaces. Instead, what is needed is way of analyz-
ing complexity that makes it possible to explore, measure, and evaluate. We
will therefore engage in such an analysis in this book.
The Interface Bottleneck
Generally speaking, artifact complexity tends to increase in order to deliver
more and better functionality. At the same time the size of digital artifacts
tends rather to decrease or stay the same. This means that the interface in
8 Chapter 1
the long run cannot keep up with the functional improvements: the avail-
able surface of an artifact only has room for a limited number of controls
and displays given the limits of resolution of human perception and dexter-
ity. We call this the interface bottleneck.
In considering the effects of the artifact-downsizing and function-
upsizing trend on interaction, there are two basic relations to consider.
First, the smaller the size of an artifact the less surface area is available for
the user interface. Second, for a convex body, the ratio between surface
area and volume is roughly inversely proportional to size (diameter). Let
us assume that functionality is roughly proportional to volume (at a given
state of technological development). It may also seem reasonable to assume
that the more functionality, the more there is to control (but this assump-
tion will soon be adjusted and qualified in various way). Taken together,
this should have implications for the interface. With the larger surface-to-
volume ratio of smaller artifacts, they should give more room for control-
ling the functionality that is inside; in other words, be more controllable.
Good news for small artifacts, if controllability is what we want.
But there is obviously a lower limit to the size of artifacts that can be
controlled by a user through the surface of the artifact; they will stop
shrinking somewhere within a comfort zone short of this limit and not get
any smaller. (Imagine a smartphone so small you have trouble finding it in
your pocket or purse, so light it might blow away if you sneeze, with a dis-
play so small you need a magnifying glass. ... No, that is silly.) This means
there is a definite size-defined limit to the control benefits of small artifacts.
From that point on, the benefits of smallness have been fully spent and
further development is governed by the continued internal complexifica-
tion. Nothing stops artifacts from getting more and more complex, while
the control capabilities stay exhausted. Bad news if controllability is what
we want.
The basic technological premise underlying the complexification of arti-
facts and systems is of course that digital technology can be packed more
and more densely while keeping the cost per unit of volume almost con-
stant. Moore’s law (and its variations and extensions) is a long-term trend
that has held good for fifty years and we have reason to expect the trend to
continue, even if the rate of change might slow down a bit as photolitho-
graphic chips technology is approaching its theoretical limit and new basic