Eye Tracking and Performance Evaluation A Dissertation Proposal Allen VR Harper

171 Pages·2014·3.77 MB·English

Eye Tracking and Performance Evaluation
Automatic Detection of User Outcomes

A Dissertation Proposal

Allen V.R. Harper
[email protected]

Department of Computer Science
The City University of New York
Graduate Center

July 2014
ABSTRACT

While evaluating user task performance with eye tracking has been examined within the field of
Human-Computer Interaction (HCI), previous research has limited eye tracking to mainly one
role—explaining how users perform a task. This proposed dissertation research will demonstrate
a new role for eye tracking which focuses on a similar, but largely unanswered question: how
well users perform tasks. Our goal is to predict user outcomes on learning tasks based solely on
eye movement patterns. We pay special attention to both the design of the visual stimuli and
information content, which we predict are key factors in establishing a strong connection
between eye movement patterns and user outcomes. We propose to use a variety of machine-
learning approaches to explore the limits of this relationship. Our preliminary study demonstrates
that this novel approach has significant discriminative power in classifying users into
performance groups.

ii
Table of Contents
Chapter 1 Introduction ...........................................................................................1
Chapter 2 Background ............................................................................................5
2.1 Human-Computer Interaction ...............................................................................................5
2.2 Motivating Example ..............................................................................................................8
2.3 Overview of Eye-Tracking Technology .................................................................................. 10
Chapter 3 Survey of Related Work Connecting Eye Tracking and
Performance 17
3.1 Literature Evaluation Criteria ............................................................................................... 18
3.2 Paper Critiques ................................................................................................................... 20
3.2.1 ATC Study ............................................................................................................................... 21
3.2.2 IDE Study ................................................................................................................................ 28
3.2.3 PILOT Study ............................................................................................................................ 34
3.2.4 LINE Study .............................................................................................................................. 41
3.2.5 NEWS Study ........................................................................................................................... 46
3.3 Summing Up: Defining the Components of an EPC Experiment ............................................. 52
Chapter 4 Development of Research Questions .................................................56
Chapter 5 Preliminary Work: EPC Verification Study #1 ................................61
5.1 Design of Experimental Materials ........................................................................................ 61
5.1.1 Design of the Information Content ........................................................................................ 61
5.1.2 Design of the User Interface .................................................................................................. 63
5.1.3 Performance Measure Design ............................................................................................... 66
5.2 Experimental Design ........................................................................................................... 69
5.3 Subject Recruitment ........................................................................................................... 70
5.4 Questionnaires Administered .............................................................................................. 71
5.5 Experimental Procedure ...................................................................................................... 72
5.6 Eye-Tracking Calibration and Recording ............................................................................... 73
5.7 Summary of Data Collected ................................................................................................. 73
Chapter 6 Preliminary Work: Classification of Users from Study #1 .............74
6.1 Related Work on Machine Learning and Eye Movement Patterns ......................................... 74
6.2 Feature Extraction .............................................................................................................. 79
6.2.1 Content-Dependent Fixation-Based Features ....................................................................... 81
6.2.2 Content-Dependent Dwell-Based Features ........................................................................... 85
6.2.3 Content-Independent Fixation-Based Features ..................................................................... 87
6.2.4 Content-Independent Dwell-Based Features ........................................................................ 90
6.3 Tools................................................................................................................................... 91
6.4 Machine Learning Models ................................................................................................... 92
6.5 Results ............................................................................................................................... 94
iii
6.6 Discussion of RQ1 ............................................................................................................... 97
Chapter 7 Proposed Work ....................................................................................99
7.1 Research Question 2 ........................................................................................................... 99
7.2 Research Question 3 ......................................................................................................... 100
7.3 Research Question 4 ......................................................................................................... 101
7.4 Timeline for Proposed Dissertation Work .......................................................................... 102
Chapter 8 Discussion ...........................................................................................104
8.1 Expected Contributions of this Research Project ................................................................ 104
8.2 Future Applications of this Research Beyond the Proposed Work ....................................... 105
References
Appendix A Master List of Features ............................................................ 111

iv
List of Figures

FIGURE 1: MOCKUP OF AN EDUCATIONAL APPLICATION PROTOTYPE INTERFACE. ....................................................... 8
FIGURE 2: SCAN PATH SUPERIMPOSED ON AOIS [22].................................................................................................. 13
FIGURE 3: EXAMPLE HEAT MAP [23]. ........................................................................................................................... 14
FIGURE 4: ATC SIMULATOR INTERFACE [23]................................................................................................................. 22
FIGURE 5: A) TEXT UI B) COLOR UI C) PFT RESULTS [23]. ............................................................................................. 26
FIGURE 6: USER INTERFACE FOR IDE PAPER [24]. ........................................................................................................ 29
FIGURE 7: ORIGINAL AGGREGATED DATA FOR IDE STUDY [24].................................................................................... 32
FIGURE 8: SEGMENTED PRESENTATION OF PFT [24]. .................................................................................................. 32
FIGURE 9: LANDING SIMULATOR UI FOR PILOT STUDY [25]. ....................................................................................... 35
FIGURE 10: PILOT LANDINGS A) EXPERTS (ON LEFT) B) NOVICES (ON RIGHT). ............................................................ 37
FIGURE 11: A) EXPERT (LEFT) B) NOVICE (RIGHT) SAMPLE SCAN PATHS [25]. ............................................................. 39
FIGURE 12: TOTAL FIXATIONS/AVERAGE DWELL TIME EXPERTS AND NOVICES (KASARSKIS, STEHWIEN ET AL. 2001).
........................................................................................................................................................................... 39
FIGURE 13: FIXATIONS AND PERFORMANCE (KASARSKIS, STEHWIEN ET AL. 2001). ................................................... 40
FIGURE 14: EXAMPLE OF A MODIFIED SCAN PATH IN THE LINE STUDY [26]. .............................................................. 44
FIGURE 15: THREE VERSIONS OF THE NEWS STUDY UI A) BASE VERSION B) WITH CRAWLER C) BOTH CRAWLER AND
HEADLINE (JOSEPHSON 2006). ........................................................................................................................... 47
FIGURE 16: PERFORMANCE RESULTS FOR NEWS STUDY [22]. ..................................................................................... 49
FIGURE 17: PFT RESULTS FOR NEWS STUDY [22]. ........................................................................................................ 50
FIGURE 18: MOCK-UP OF EDUCATIONAL APPLICATION PROTOTYPE INTERFACE. ....................................................... 64
FIGURE 19: EPC VERIFICATION STUDY USER INTERFACE (VISUAL STIMULI#1 SLIDE#1 AT 22.5 SECONDS: SPEAKER IS
DELIVERING THE ASIDE WHICH IS ASSOCIATED WITH MATERIAL ON THE SLIDE.) ............................................. 65
FIGURE 20: PERFORMANCE MEASURE QUESTIONS FOR VISUAL STIMULI#1 SLIDE#1 AT TIME 22.5 SECONDS WHILE
SPEAKER IS DELIVERING THE ASIDE (REFER TO FIGURE 19). .............................................................................. 69

v
List of Tables

TABLE 1: EPC CANDIDATE EXPERIMENTAL FACTORS AND STUDY RATINGS. ................................................................ 53
TABLE 2: CONTENT RATIOS FOR VISUAL STIMULI #1 PILOT STORY (REFER TO FIGURE 19 BELOW FOR THE POSITION
OF THE AOIS ON THE USER INTERFACE). ............................................................................................................ 63
TABLE 3: PERFORMANCE MEASURE QUESTIONS SELECTED BY CONTENT TYPE AND AOI REFLECTING THE ORIGINAL
CONTENT RATIOS. .............................................................................................................................................. 68
TABLE 4: SUMMARY OF REVIEWED MACHINE LEARNING APPROACHES IN EYE TRACKING STUDIES. ......................... 79
TABLE 5: SUMMARY OF EYE-MOVEMENT METRICS USED IN ALL REVIEWED STUDIES (LITERATURE SURVEY AND
SECTION 6.1). ..................................................................................................................................................... 80
TABLE 6: ALL CONTENT-DEPENDENT FIXATION-BASED FEATURES............................................................................... 81
TABLE 7: ALL CONTENT-DEPENDENT DWELL-BASED FEATURES. ................................................................................. 86
TABLE 8: ALL CONTENT-INDEPENDENT FIXATION-BASED FEATURES. .......................................................................... 87
TABLE 9: ALL CONTENT-INDEPENDENT DWELL-BASED FEATURES. .............................................................................. 90
TABLE 10: PERFORMANCE RESULTS ON PRELIMINARY EYE MOVEMENT DATA. TABLE DISPLAYS WEIGHTED F-
MEASURE UNDER EACH DATA ARRANGEMENT. ................................................................................................. 98
TABLE 11: TIMELINE FOR PROPOSED DISSERTATION WORK. ..................................................................................... 103
TABLE 12: MASTER LIST OF ALL FEATURES INCLUDED IN THE STUDY......................................................................... 111

vi
Chapter 1 Introduction
Human interactions with computers can be considered from the perspective of two
information processors interacting across a user interface [1]. As such, the study of human
behaviors related to these interactions is of particular interest within the field of Human-
Computer Interaction as they can provide insight into human performance. One class of behavior
that is of interest within HCI research is eye movements. The reason for this interest is that many
groups of computer users can receive as much as 80% of their perceptual input via the visual
channel [2]. Therefore, techniques for studying eye movement behavior have been considered an
effective means of characterizing human-computer interactions.
The significance of eye movements to evaluating human performance is that they represent
an overt, observable and therefore, measurable quantity associated with the interactions which is
assumed to have a predictable relationship with the covert cognitive processes associated with
visual attention which are not thoroughly understood. As a result of this relationship, eye-
tracking measures, such as total number of fixations, gaze durations, and scan paths (defined in
Section 2.3) can provide detailed information about how users perform tasks—information that
would be difficult (if not impossible) to collect using other HCI methods. As Kowler explains:
Eye movements and attention are assumed to serve useful purposes connected to the visual task, an
assumption that has fueled decades of efforts to use eye movements to study how people search, read,
study pictures of scenes, or carry out all manner of visually-guided actions involving reaching,
pointing, manipulating objects, walking, or driving [3].

1
These efforts to use eye tracking as a means of explaining how experimental subjects
perform tasks are detailed in many surveys within this specialized area of the HCI usability
literature [4-7]. Due to the proliferation of the use of eye tracking across many disciplines this
literature can appear quite fragmented; however, one theme running throughout this literature is
the significant contribution to the understanding of how users perform tasks that is provided by
the analysis of eye movements. Leveraging eye tracking in this fashion has led to improvements
in the understanding of user behaviors during interactions as well as to improvements of
traditional HCI measures such as Think-Aloud protocols [8-10]. In this thesis proposal, we will
refer to this explanatory role of eye tracking as the HCI/Eye Tracking or HET perspective; the
goal of which is to employ eye tracking as a supplemental technique to the more traditional HCI
methods.
However, while the HET approach has proven useful for addressing the question of how
users perform tasks, other important questions have remained unaddressed. In particular, there
have been only tentative attempts within eye-tracking research to address the important related
question of how well users perform tasks.
We will argue in this dissertation proposal, that lack of progress in this direction is clearly
attributable to many difficulties associated with the design of eye-tracking experiments and the
subsequent analysis of eye-tracking datasets with regard to establishing a predictable relationship
between eye movement patterns and user performance outcomes—the necessary prerequisite if
eye movement measures are to stand in as proxies for user performance.

2
Recent literature has attributed these difficulties to a small number of factors. For
example, the lack of straightforward methods for working with the larger and more complex
experimental data sets typically generated in an eye-tracking experiment—some eye-tracking
experiments have collected data for approximately thirty minutes [11]; and the lack of standard
reporting practices and interpretation of eye-tracking results paralleling the more traditional HCI
methods such as reaction time (RT), speed of task execution and user error rates [12] [13].
However, we argue that the problem is more fundamental than currently believed and
requires a more thorough reworking of how eye-tracking experiments are designed if eye
tracking is to advance into new application areas. In our work we propose to address this
question by constructing an alternative framework to the HET perspective which we refer to as
the Eye Tracking / Performance Connection or EPC perspective. In contrast to the HET
approach, the goal of an eye-tracking experiment using an EPC design is to use the eye-tracking
record to predict the likely success of a user performing a task well.
We will argue that in order to demonstrate that a connection exists between eye
movements and user’s task-performance it will be necessary to conduct carefully designed EPC
verification experiments. Namely, an EPC verification experiment is an eye-tracking study in
which subjects attempt to perform a task with a given user interface with varying levels of
success. Following the collection of both performance and eye-tracking data, researchers conduct
an analysis in search of correlations between eye movement metrics and subject’s performance
scores. In this way, a researcher can verify that an eye-tracking metric can indeed be found which
relates to a user’s success at a particular task. These experiments will tightly control for factors

3
such as subject selection, user interface, visual content displayed to users, as well as both task
type and difficulty to be performed. The importance of providing stringent controls on these
factors is the great sensitivity of eye movements to almost any change in an experimental design.
By controlling for these factors it becomes possible to establish predictable associations between
eye-tracking metrics and user performance via some form of analysis of the eye movements of
users in the experiment and their performance on some task.
In summary, we propose to do the following:
• Design highly controlled visual stimuli and content
• Design a challenging task for the user to perform
• Conduct eye-tracking experiments
• Apply machine-learning techniques to predict user performance based solely upon
eye movement features
This proposal is structured in the following way: Chapter 2 discusses the background and
motivations of the proposed research as well as introducing the topic of eye tracking. Chapter 3
is a survey of a particular subset of papers taken from the eye-tracking literature that highlights
research attempts at connecting eye-tracking metrics and user performance. Chapter 4 develops
the research questions that stem from the summary of the analysis of the eye-tracking literature.
Chapter 5 presents our preliminary work related to EPC verification experiments, while Chapter
6 presents the results of this preliminary work with respect to the classification of users into
performance groups. In Chapter 7 we provide a detailed plan of our proposed dissertation work
and in Chapter 8 we discuss the expected contributions of this research project and future
applications of this work.

4