Table Of ContentPlease cite as the following:
Clinton, V., Cooper, J.L., Michaelis, J., Alibali, M.W., & Nathan, M.J. (2017). Revising visuals
based on instructional design principles: Effects on cognitive load and learning. In C.
Was, F.J. Sansosti, & B.J. Morris (Eds.) Eye-tracking technology applications in
educational research, (pp. 195-218). Hershey, PA: IGI Global.
Author Note
Address correspondence to Virginia Clinton, University of North Dakota, 231 Centennial
St., Grand Forks, ND, 58202, [email protected], phone 1 (701) 777-3920, and fax 1
(701) 777-3454.
This research was supported by the Institute of Education Sciences, U.S. Department of
Education, through Grant R305C100024 to the University of Wisconsin--Madison. The opinions
expressed are those of the authors and do not represent views of the Institute or the U.S.
Department of Education.
How Revisions to Mathematical Visuals Affect Cognition:
Evidence from Eye Tracking
Virginia Clinton
University of North Dakota, United States
Jennifer L. Cooper
Wesleyan University, United States
Joseph Michaelis, Martha W. Alibali, & Mitchell J. Nathan
University of Wisconsin – Madison, United States
ABSTRACT
Mathematics curricula are frequently rich with visuals, but these visuals are often not designed for
optimal use of students’ limited cognitive resources. The authors of this study revised the visuals in a
mathematics lesson based on instructional design principles. The purpose of this study is to examine the
effects of these revised visuals on students’ cognitive load, cognitive processing, learning, and interest.
Middle-school students (N = 62) read a lesson on early algebra with original or revised visuals while
their eye movements were recorded. Students in the low prior knowledge group had less cognitive load
and cognitive processing with the revised lesson than the original lesson. However, the reverse was true
for students in the middle prior knowledge group. There were no effects of the revisions on learning. The
findings are discussed in the context of the expertise reversal effect as well as the cognitive theory of
multimedia learning and cognitive load theory.
Keywords: Cognitive Load, Instructional Design Principles, Cognitive Processing, Multimedia Learning,
Multiple Representations
INTRODUCTION
Eye-tracking measures may provide important insight into the design of learning
materials (i.e., instructional design; Hyönä, 2010; Mayer, 2010; van Gog & Scheiter, 2010). This
view is based on the eye-mind assumption (Just & Carpenter, 1980), which states that the eye
fixates on what the mind is processing (Just & Carpenter, 1976; Rayner, 1998). By examining
what a student’s eyes fixate on, one can discern what that student is focusing on, and this
information may be useful for understanding how students use instructional materials.
One distinct benefit of eye-tracking measures is their spatial precision, which allows for
understanding how information in different regions of a lesson is processed (e.g., Chang & Choi,
2014; She & Chen, 2009). For this reason, eye-tracking measures are particularly valuable for
understanding how different representations, such as visuals and text, are processed (e.g., Mason
et al., 2013; Scheiter & van Gog, 2009; Schwonke, Berthold, & Renkl, 2009). In this research,
we used eye tracking to examine how variations in visuals affect students’ processing of a lesson.
Researchers and curriculum designers have articulated instructional design principles
(also called evidence-based principles and cognitive principles) that specify how visuals should
be integrated with text (Mayer & Moreno, 1999; Mayer, 2008). The broad aim of these principles
is to optimize learning (e.g., Mayer, 2009; Sweller, Ayres, & Kalyuga, 2011). In this work, we
used eye tracking to examine students’ processing of a lesson that was either well aligned or less
well aligned with these principles.
To address this issue, we used a lesson from Connected Mathematics 2 (CMP2; Lappan,
Fey, Fitzgerald, Friel, & Phillips, 2006), which is rich with visuals, such as pictures, diagrams,
and other spatial representations (Clinton, Cooper, Alibali & Nathan, 2012). However, the ways
visuals are used in the lessons and activities do not always make effective use of students’
cognitive resources. In a separate, large-scale study, a team of researchers has revised the visuals
based on instructional design principles, and is testing the revised version of the CMP2
curriculum in a nation-wide randomized control trial in order to determine the effectiveness of
the revisions on learning (Davenport, Kao, & Schneider, 2013).
Building on previous research findings on instructional design principles and eye tracking
(e.g., Johnson & Mayer, 2012; Ozcelik, Karakus, Kursun, & Cagiltay, 2009; Ozcelik, Arslan-Ari,
Cagiltay, 2010), we conducted an eye-tracking experiment with students who read a lesson
derived from the CMP2 curriculum with original visuals or with visuals that were revised on the
basis of instructional design principles. The aim was to assess the effects of the revised visuals
on students’ processing of the different representations and on their subsequent learning.
Specifically, we were interested in how eye-tracking measures could reveal the moment-by-
moment effort in working memory, referred to as cognitive load, as well as the amount of time
spent viewing representations, referred to as the amount of cognitive processing (see Ozcelik et
al., 2010 for a similar approach).
BACKGROUND
The instructional design principles that guided the revisions are grounded in the cognitive
theory of multimedia learning (Mayer, 2009; Mayer & Moreno, 2003) and in cognitive load
theory (e.g., Paas, Renkl & Sweller, 2003; Plass, Moreno, & Brünken, 2010). A central idea of
both theories is that the structure of the cognitive system imposes limits on the processing of
information presented to auditory, linguistic, and visual sensory processing channels that
influence how learners integrate information. The cognitive theory of multimedia learning holds
that visual and verbal information (i.e., text or speech) are processed in different pathways, and
the theory emphasizes the need for the information in these two pathways to be integrated
(Mayer, 2014a). In addition, the cognitive theory of multimedia learning prescribes guidance for
instructional design, namely the reduction of extraneous (i.e., unnecessary) processing to
improve learning (Mayer, 2009). In sum, implementing these theory-based principles in a lesson
should reduce the amount of cognitive processing necessary to understand a lesson.
In contrast to the cognitive theory of multimedia learning, cognitive load theory
emphasizes the different types of cognitive load (i.e., effort in working memory) a student may
experience (Sweller et al., 2011). Cognitive load theory builds on the idea that inherent (i.e.,
biological) constraints on working memory limit the amount of information one can process at a
given time (Chandler & Sweller, 1991). If a student has too much information to process or
information is difficult to understand, limited working memory capacity can be overloaded,
thereby impairing comprehension and diminishing learning (Sweller, 1994). Much of this
research has shown superior problem-solving performance and learning with manipulations that
reduce the cognitive load required to integrate information across visuals and text (e.g., Chandler
& Sweller, 1991; Kalyuga, Chandler, & Sweller, 1999; see Mayer & Moreno, 2009; Pashler et
al., 2007 for reviews).
Cognitive load can be intrinsic or extrinsic to the learning goal (Sweller et al., 2011).
Intrinsic cognitive load (this includes what used to be referred to as “germane load”; Kalyuga,
2011) consists of the information in working memory relevant to the task. In contrast, extrinsic
cognitive load consists of information in working memory irrelevant to the instructional task.
The general aim of instructional design principles is to reduce extrinsic cognitive load so that
students can focus their cognitive resources to best manage the lesson content (Mayer & Moreno,
2003).
Instructional design principles
Three instructional design principles guided the revisions to the curricular materials:
signaling, contiguity, and coherence.
According to the signaling principle, learning is promoted by cues, such as color codes
and labels (Mayer, 2009), to important information. Cues may promote learning both by
directing students’ attention to relevant information and by connecting corresponding
information across different representations (e.g., text and visuals; Berthold & Renkl, 2009;
Florax & Ploetzner, 2010; Kalyuga, Chandler & Sweller, 1999; de Koning, Tabbers, Rikers, &
Paas, 2009; for a meta-analysis see Richter, Scheiter & Eitel, 2016). These cues may reduce the
extraneous processing and the extrinsic cognitive load needed to discern the importance of
information or integrate corresponding information across representations (Lin & Lin, 2014;
Mayer & Moreno, 2003). The use of signaling may increase cognitive processing of the visual,
especially in the signaled areas of the visual (de Koning, Tabbers, Rikers, & Paas, 2010).
However, previous eye-tracking findings have not indicated a reduction in cognitive load due to
signaling (de Koning et al., 2010; Ozcelik et al., 2010).
The contiguity principle states that information should be arranged such that relevant
information in different representations is in close proximity. This reduces the cognitive load of
reading and connecting corresponding verbal and visual information (e.g., Ginns, 2006;
Renshaw, Finlay, Tyfa, & Ward, 2004). Labels place relevant text in close proximity to visuals,
making the information in the two representations more spatially contiguous, thereby facilitating
integration between the two representations (Florax & Ploetzner, 2010; Holsanova, Holmberg, &
Holmquist, 2009; Johnson & Mayer, 2012). Previous eye-tracking findings have indicated that
integrating relevant text with visuals does not affect the amount of cognitive processing of the
visual, but does reduce the amount of cognitive processing of the text, perhaps due to the
reduction in extraneous processing (Johnson & Mayer, 2012). However, there has been limited
empirical investigation of the contiguity principle on cognitive load (see Altan & Cagiltay, 2015,
for preliminary work with a small sample).
The coherence principle states that learning is fostered when interesting, but irrelevant
information, such as decorative pictures, is removed (Harp & Mayer, 1997; Mayer, 2009; for a
review, see Rey, 2012). This type of information has been found to distract learners and diminish
comprehension, a phenomenon referred to as the seductive details effect (e.g., Lehman, Schraw,
McCrudden, & Hartley, 2007). Irrelevant visuals may interfere with learning because they
increase the amount of information in the lesson (Sanchez & Wiley, 2006). This potential
increase in extraneous cognitive processing and extrinsic cognitive load appears to be
particularly problematic for learning from written lessons in which verbal information is
conveyed through text (visually) compared to oral presentations in which verbal information is
conveyed through narration (auditorily; Park, Moreno, Seufert, & Brünken, 2011). This
difference in modalities is presumably due to irrelevant visuals overloading cognitive resources
when all of the information in a lesson is presented visually (Park, Flowerday, & Brünken,
2015). Previous eye-tracking findings have indicated that irrelevant visuals in statistics lessons
caused less cognitive processing of instructional text and visuals compared to lessons without
irrelevant visuals, likely because of the extra information and distraction of the irrelevant visuals
(Rey, 2014). Surprisingly, one study found that self-reports of cognitive load were lower for
lessons with irrelevant visuals, despite lower comprehension scores (Park, Korbach, & Brünken,
2015. It is possible that the irrelevant visuals gave students the impression the lesson was easier
than it actually was. For this reason, an eye-tracking measure of cognitive load may be
particularly valuable (van Gog, Kester, Nievelstein, Giesbers, & Paas, 2009) for materials such
as these. An eye-tracking measure would be collected as the lesson is being viewed and would be
a more objective measure of the demand imposed by the lesson (e.g., Amadieu, Van Gog, Paas,
Tricot, & Mariné, 2009).
In past research, the signaling, contiguity, and coherence principles have typically been
examined in isolation (e.g., Florax & Ploetzner, 2010; Mason et al., 2013; Ozcelik et al., 2009,
2010; Scheiter & Eitel, 2015). However, in the revisions of the CMP curriculum, these principles
were all applied in the following ways: (1) additional signaling was added; (2) contiguity of
visuals and related text was increased, and (3) math-irrelevant visuals were removed. Multiple
principles were applied based on a “less is more” approach; less extrinsic cognitive load and
extraneous processing (through the removal of math-irrelevant visuals and increased contiguity
and signaling) was expected to yield more learning (see Mayer, 2014b). This approach is novel
and examines whether benefits to the process and products of learning can be maximized by
applying multiple principles simultaneously.
Instructional design principles can interact with prior knowledge
Generally speaking, the cognitive load and cognitive processing involved when learning
from lessons with text and visuals varies with both the design of the material (as previously
discussed) and the prior knowledge of the student (Kalyuga et al., 2003; Kirschner, Paas,
Kirschner, & Janssen, 2011; Moreno, 2004). Some evidence suggests that the implementation of
instructional design principles may be most effective for students with low levels of prior
knowledge (Mayer, 2001). This is because the less prior knowledge a student has, the more
intrinsic cognitive load the task imposes (Kalyuga, 2011; Leahy, Hanham, & Sweller, 2015). An
increase in intrinsic cognitive load could consume working memory capacity, leaving little
capacity for handling extrinsic cognitive load (Paas, Renkl, & Sweller, 2003). For these reasons,
if a task is high in intrinsic cognitive load, reductions in extrinsic load should yield more benefits
compared to tasks low in intrinsic load (Seufert, Jänen, & Brünken, 2007). Therefore, the
reduction of extraneous processing and extrinsic cognitive load through the application of the
instructional design principles may foster greater learning in students with low levels of prior
knowledge than in students with high levels of prior knowledge (e.g., Magner, Schwonke,
Aleven, Popescu & Renkl, 2014; see Moreno & Mayer, 2007; Schnotz, 2002, for discussions). It
is also possible that efforts to lower cognitive load may actually make learning more difficult for
students with higher levels of prior knowledge (Kalyuga, 2007). This phenomenon, known as the
expertise reversal effect, arises because the information added to guide processing is redundant
with what students with high prior knowledge already know, thereby increasing the cognitive
load of the lesson (Sweller et al., 2011).
Much of the previous work on prior knowledge and instructional design techniques has
involved separating students into two groups: high and low (e.g., Mayer & Gallini, 1990; Mayer,
Steinhoff, Bower, & Mars, 1995). However, recent research findings have indicated that
separating students into three groups allows a more nuanced understanding of interactions
between instructional design techniques and prior knowledge (Magner et al., 2014).
THE CURRENT STUDY
The purpose of the current study is to use eye-tracking methodology to examine the
effects of revisions to visuals based on instructional design principles (specifically, the signaling,
contiguity, and coherence principles) on the process of reading a mathematics lesson and on
subsequent learning from that lesson. Eye tracking was the methodology of choice because the
data from eye tracking can be used to infer the moment-by-moment processes involved in
reading (Just & Carpenter, 1980; Rayner, 1998). Because of the spatial precision of eye tracking,
the data afford valuable insight into the processing of different representations in written lessons
with visual representations (e.g., Mason et al., 2013; Rau, Michaelis, & Fay, 2015; Scheiter &
Eitel, 2015; see Hyönä, 2010 for review). In other words, eye tracking allows an examination of
how visuals and text are processed.
Specific to this study, we were interested in how eye tracking could yield information
about the cognitive load and amount of cognitive processing involved in viewing the text and
visuals. To assess cognitive load, average fixation length (i.e., pause in eye movement) was used.
Average fixation length is thought to be a positive indicator of cognitive load (i.e., as cognitive
load increases, average fixation length increases; Ozcelik et al. 2010; van Gog et al., 2009; Paas,
2009; van Gog & Scheiter, 2010). This is because cognitive load is essentially the mental effort
involved in working memory, and fixations typically increase as more effort is exerted while
viewing the fixated material (van Gog et al., 2009). Thus, average fixation length can be used to
infer the effectiveness of the revisions in reducing cognitive load. In contrast, the amount of
cognitive processing is how much time one spends thinking about something. Given the eye-
mind assumption (Just & Carpenter, 1980), the time spent viewing a representation, calculated
by summing the fixation durations on that representation, is considered to be the amount of
cognitive processing involved with the representation (e.g., Graesser, Lu, Olde, Cooper-Pye, &
Whitten, 2005; Rayner, 1998; Ozcelik et al., 2009). In other words, the amount of time a student
viewed a section of a lesson provides a measure of how much that student cognitively processed
that section (e.g., Kaakinen, Olkoniemi, Kinnari, & Hyönä, 2014). The amount of cognitive
processing differs from cognitive load because cognitive processing is the overall time spent on a
representation whereas cognitive load is the mental effort in working memory at a given moment
(Ozcelik et al., 2010).
We address three main research questions. First, what were the effects of the revisions on
cognitive load while reading the lesson? Because the revisions were designed to reduce cognitive
load, average fixation length was expected to be shorter for the revised lesson than the original
lesson (e.g., Amadieu et al., 2009). In this way, average fixation length can be used to evaluate
the effectiveness of the revisions. Further, this finding might be strongest for students with low
levels of prior knowledge, compared to students with higher levels of prior knowledge (Mayer,
2001). Moreover, because of the expertise reversal effect (Kalyuga, 2007), it is possible that the
revisions would increase the cognitive load for students with middle and high levels of prior
knowledge. It should be noted that some previous eye-tracking work examining a single
instructional design principle has not revealed effects on average fixation length (Altan &
Cagiltay, 2015; de Koning et al., 2010; Ozcelik et al., 2010). However, because our study applies
multiple principles, the cumulative effects of these principles may be powerful enough to reduce
average fixation length, at least for students with low levels of prior knowledge.
Second, what were the effects of the revisions on the amount of cognitive processing for
different representations? These amounts can be assessed via total fixation time, which is the
summed duration of all fixations within an area of interest (e.g., a visual or a section of text;).
With this measure, it can be determined whether students differed in how much they processed
the original and revised visuals. Given that the revised visuals provide additional guidance for
processing, students may need to engage in less cognitive processing with them, thereby needing
less total fixation time compared to the original visuals. In addition, it can be determined whether
students processed the text in the lesson differently depending on the type of visuals used.
Students with revised visuals may have less total fixation time on the text compared to students
with original visuals because students with revised visuals may need less help from the text to
understand the mathematical content. Finally, to assess the amount of cognitive processing of the
entire lesson, we examined the total fixation duration for the entire lesson. It was expected that
students with the revised lesson would spend less time with the lesson than students with the
original lesson.
As with cognitive load, the effects of the revisions on the amount of cognitive processing
would likely be most pronounced for students with low levels of prior knowledge. However,
prior knowledge is especially important to consider given that the amount of cognitive
processing of visuals is negatively associated with prior knowledge (i.e., the less prior
knowledge a student has, the longer the visual is viewed; Hegarty & Just, 1993; Schwonke et al.,
2009). This may be because students with high prior knowledge primarily use the visuals to
confirm what they already know, rather than to learn from them (Rasch & Schnotz, 2009).
Third, did the revisions of the visuals affect students’ learning from the lesson? The
instructional design principles were intended to reduce cognitive load and extraneous processing
so that students could focus their limited cognitive resources on the content presented in the
lesson (Mayer, 2009; Sweller et al., 2011). For these reasons, students may perform better on a
post-lesson test after reading the revised lesson than the original lesson. However, the learning
benefits from the revisions may be strongest for students with low levels of prior knowledge
(Mayer, 2001).
One benefit of eye tracking is that it can reveal whether the revisions affected how
students viewed the lesson, even if there is no observable effect on learning. For example,
previous eye-tracking findings have revealed that replacing text with narration in a multimedia
presentation reduced average fixation lengths when viewing the presentation, but did not affect
learning from the presentation (Liu, Lai, & Chuang, 2011). This is because eye-tracking
measures are more sensitive than many other measures. Furthermore, the amount of cognitive
processing of the representations in the lesson could be used to examine if students compensate
for the lack of guidance in the original visuals by spending more time on different
representations in the lesson as well on the lesson overall. Finally, if the revisions had no effect
on eye movements or learning, it can be assumed that the application of the instructional design
principles was ineffective and other techniques should be explored (e.g., Lowe & Boucheix,
2011).
These questions were addressed using a lesson about how to graph independent and
dependent variables on a coordinate grid. In addition to being visually rich, this topic is of
particular interest because of its importance in scientific literacy (Padilla, McKenzie, & Shaw,
1986) and algebraic reasoning (Nathan & Kim, 2007).
Method
Participants
Participants were 62 (26 female, 36 male) middle-school students entering sixth or
seventh grade (ages 10 – 12 years; M = 11.12 years, SD = .33 years). Because of apparatus
malfunction, eye-tracking data were collected for only 57 participants. However, all 62
participants read the lesson in the same manner and completed the self-report of prior knowledge
and the post-lesson test (see Measures). Participants were compensated with a $15 gift card for
an online retailer.
Apparatus
An EyeLink 1000 Desk-Mounted System, manufactured by SR Research Ltd. (Toronto,
Ontario, Canada), was used to collect eye movement data. This eye tracker uses an infra-red
video camera for monocular tracking, and the video camera was focused on the participant’s
pupil. The video camera sampled real-time fixations at a 1000 Hz sampling rate. Head position
was stabilized with a chin and forehead rest 70 cm from the computer monitor displaying the
lesson. Pupil diameter was recorded with centroid pupil tracking.
Materials
The lesson, derived from CMP2 (Lappan et al., 2006), covered the skills necessary to
record data with independent and dependent variables in a table and then construct a graph from
those data. The skills were presented in the context of a story in which a person was planning a
long-distance bike trip and needed to know her biking pace. The original lesson included a
variety of visuals including a map, graphs, tables, and math-irrelevant pictures.
The lesson consisted of nine pages, with identical text in both the original and revised
conditions but changes made to the visuals on eight of the revised pages. The first page was the
same for both groups, with text that reiterated the instructions for reading the lesson (e.g., read at
your own pace, please sit still). The introduction to the lesson discussed the person’s plans for
her trip and included a map that had decorative features in the original materials, but, based on
the coherence principle, did not have decorative features in the revised materials. In the
remaining 7 pages, there were 3 pages in which the signaling and contiguity principles were
applied to the revised visuals. Specifically, color coding, labels, and call-out boxes were added to
the math-relevant visuals of tables and graphs (see Figure 1). There were 2 pages in which the
math-relevant visuals were the same in the original and revised materials, but the original
materials also contained math-irrelevant visuals and, following the coherence principle, these
were deleted in the revised materials (see Figure 2). Finally there were 2 pages in which the
original materials contained a math-irrelevant visual and, following the signaling, contiguity, and
coherence principles, the revised materials contained a math-relevant visual with labeling and
call-out boxes (see Figure 3).
As one would likely see in CMP2, the arrangement of the visuals in relationship to the
text varied on each page. On some pages the visuals were below the text and on other pages the
visuals were beside the text. This created more variability in the design of the materials across
the lesson than is typically seen in eye-tracking experiments of lessons with visuals (e.g.,
Scheiter & Eitel, 2015). However, the aim was to make materials as authentic as possible to
enhance the ecological validity of the results (see Holsanova, 2014).
Figure 1. Example of revisions based on the signaling and contiguity principles
Figure 2. Example of revisions based on the coherence principle (image from Shutterstock©,
used with permission)
Figure 3. Example of revisions based on the coherence, signaling, and contiguity principles
(image in original lesson from Getty Images©, used with permission)
Procedure
Participants took part individually, and the experimenter calibrated the eye tracker for
each participant. Participants then read the lesson one page at a time, at their own pace, while
their eye movements were recorded. Prior to reading each page, participants gazed at a single dot
on the screen to correct for drifts in eye gaze that may have occurred since calibration. After
reading the lesson, participants reported their familiarity with the math content and then
completed the post-lesson test.
Measures
Similar to Mautone and Mayer (2001), prior knowledge was assessed through self-reports
of familiarity with the mathematical content of the lesson on a Likert scale from 1-5 (M = 3.74,
SD = 1.20). Based on these responses, participants were divided into three prior knowledge
groups: low (responses between 1-3; n = 22), middle (response of 4; n = 16), and high (response
of 5; n = 19). Self-reports were used instead of pretests to prevent pretest sensitization, in which
treatment effects may be inflated by priming knowledge with assessments prior to the lesson
(Willson & Putnam, 1982; Willson & Kim, 2010).
The post-lesson test consisted of three parts. The first part directly assessed individual
skills covered in the lesson, such as reading data values from a table and graph, identifying
independent and dependent variables, identifying appropriately scaled axes, using appropriate
axis units and scales, and locating the x- and y-axes. Answers on the first part were scored for
accuracy (0 for incorrect answers; 1 for correct answers). The second part asked participants to
construct a graph based on a table of data points. Students’ graph construction was scored out of
5 points based on variable and axis placement, consistent use of scale on each axis, variable
labels, and plotting points. The third part assessed preparation for future learning (see Bransford
& Schwartz, 1999) by presenting a short, novel lesson that students read and answered questions
about. The preparation for future learning questions asked students to take their reasoning a step
further and to match a graph’s pattern to a description of a data scenario. As with the first part,
