Opposing pupil responses to offered and anticipated reward values

41 Pages·2017·1.16 MB·English

bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
1
1
2
3 Opposing pupil responses to offered
4 and anticipated reward values
5
6
7
8
9
10
11
12 Tyler Cash-Padgett1, Habiba Azab2, Seng Bum Michael Yoo2, Benjamin Y.
13 Hayden1
14
15
16 1Department of Neuroscience and
17 Center for Magnetic Resonance Research,
18 University of Minnesota, Minneapolis, Minnesota
19
20 2Department of Brain and Cognitive Sciences and
21 Center for Visual Sciences,
22 Center for the Origins of Cognition,
23 University of Rochester, Rochester, New York
24
25 *Corresponding author:
26 Tyler Cash-Padgett
27 Department of Neuroscience
28 University of Minnesota
29 Minneapolis, MN 55455
30 Email address: [email protected]
31
32
33 Acknowledgments
34 This work is supported by a CAREER award from NSF (BCS1253576) and a R01
35 from NIH (DA038615) to BYH. We thank Meghan Castagno, Marc Mancarella
36 and Caleb Strait for assistance with data collection, and the rest of the Hayden lab
37 for valuable discussions.
38
39
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
2
40 ABSTRACT
41
42 Previous studies have shown that the pupils dilate more in anticipation of
43 larger rewards. This finding raises the possibility of a more general association
44 between reward amount and pupil size. We tested this idea by characterizing
45 macaque pupil responses to offered rewards during evaluation and comparison in
46 a binary choice task. To control attention, we made use of a design in which
47 offers occurred in sequence. By looking at pupil responses after choice but before
48 reward, we confirmed the previously observed positive association between pupil
49 size and anticipated reward values. Surprisingly, however, we find that pupil size
50 is negatively correlated with the value of offered gambles before choice, during
51 both evaluation and comparison stages of the task. These results demonstrate a
52 functional distinction between offered and anticipated rewards, and present
53 evidence against a narrow version of the simulation hypothesis, the idea that we
54 represent offers by reactivating states associated with anticipating them. They also
55 suggest that pupil size is correlated with relative, not absolute, values of offers,
56 suggestive of an accept-reject model of comparison.
57
58
59
60
61
62
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
3
63 INTRODUCTION
64
65 The pupils systematically dilate and constrict in response to ongoing
66 changes in mental state. Pupil diameter therefore provides a window into many
67 important mental functions, ranging from attention (Hoeks & Levelt, 1993; van
68 den Brink et al., 2016) and working memory (Kahneman & Beatty, 1966) to
69 mental effort (Just et al., 2003; Varazzani et al., 2015) and surprise (Lavín et al.,
70 2013; Preuschoff et al., 2011). Researchers have even used pupil size to gain
71 insight into the mechanisms of subjective time perception (Suzuki et al., 2016),
72 rate of learning (Nassar et al., 2012), and multi-sensory integration (Rigato et al.,
73 2016), as well as decision-making (de Gee et al., 2014; Einhauser et al., 2010;
74 Einhauser et al., 2008).
75 Previous research supports the idea that there is a positive relationship
76 between reward magnitude and pupil size. Specifically, pupil size increases in
77 anticipation of rewards and increases more in anticipation of larger primary
78 rewards (Rudebeck et al., 2014). The positive relationship between pupil size and
79 anticipated rewards is also observed in anticipation of conditioned reinforcers
80 (Rudebeck et al., 2014; Varazzani et al., 2015). These results suggest that there
81 may be a positive relationship between pupil size and reward amount that is
82 observed for types of rewards other than anticipated ones.
83 We are particularly interested in the relationship between the way the
84 brain encodes anticipated and offered rewards. Both types of reward are
85 imagined, not experienced, and both can be used to influence upcoming actions.
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
4
86 Despite these similarities, they are also somewhat conceptually distinct: offered
87 rewards are not certain (they are contingent on choice) while anticipated rewards
88 are generally certain. Offered rewards provide information that is used to directly
89 drive choice, while anticipated rewards generally drive other processes, including
90 preparation for reward receipt, savoring, and learning. One hypothesis about the
91 relationship between these reward types, the simulation hypothesis, holds that
92 when we choose, we represent offered values and we do so by reactivating a
93 domain-general representation of the experience of receiving the reward (Wang &
94 Hayden, 2017; Kahnt 2010 et al., Howard et al., 2015). There is some evidence in
95 favor of this hypothesis (Howard et al., 2015; Xie et al., 2016; Kahnt et al., 2010;
96 Schoenbaum et al., 2003; Stalnaker et al., 2006). However, at least some data
97 suggests that there are key qualitative differences in the way that offered rewards
98 are represented (McNamee et al., 2015; Farovik et al., 2015; Tsujimoto et al.,
99 2012; Wang & Hayden, 2017). These data then would predict that responses to
100 reward in key reward regions differ depending on the context in which the reward
101 was presented. We hypothesized that these contextual differences would show up
102 in other domains, such as pupil size.
103 In order to examine the relationship between pupillary encoding of offered
104 and anticipated values, we took advantage of an existing dataset based on two
105 macaques performing a sequential choice task with risky options. We found that
106 pupil size decreased in response to higher value offers – the opposite pattern
107 observed for anticipated values. Consistent with this observation, pupil size
108 following the second offer decreased less when the first was high value – a
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
5
109 finding that is parsimoniously explained by the idea that pupils encode relative
110 value, the key decision variable for accept-reject choices (Strait et al., 2014, Azab
111 & Hayden 2017). Following choice, but before reward, the relationship between
112 reward and pupil size reversed, replicating the findings of previous studies: it
113 increased on trials in which a large reward was anticipated, and on trials in which
114 a large reward was more likely. These findings indicate that anticipated and
115 offered rewards are disambiguated at the level of the pupillary reward response,
116 and suggest they are processed in distinct ways in the brain.
117
118
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
6
119 METHODS
120
121 Some of the data for dACC recordings were previously published (Azab &
122 Hayden, 2017; Strait et al., 2016); all data and analyses presented here are new.
123
124 Surgical procedures
125 All procedures were approved by the University Committee on Animal
126 Resources at the University of Rochester and were designed and conducted in
127 compliance with the Public Health Service’s Guide for the Care and Use of
128 Animals. Two male rhesus macaques (Macaca mulatta: subject B age 6; subject J
129 age 7) served as subjects. A small prosthesis for holding the head was used.
130 Animals were habituated to laboratory conditions and then trained to perform
131 oculomotor tasks for liquid reward. A Cilux recording chamber (Crist
132 Instruments) was placed over the dACC and sgACC. Position was verified by
133 magnetic resonance imaging with the aid of a Brainsight system (Rogue Research
134 Inc.). Animals received appropriate analgesics and antibiotics after all procedures.
135 Throughout both behavioral and physiological recording sessions, the chamber
136 was kept sterile with regular antibiotic washes and sealed with sterile caps. All
137 recordings were performed during the animals’ light cycle between 8 am and 5
138 pm.
139
140 Behavioral Task
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
7
141 Monkeys performed a two-option gambling task (Azab & Hayden, 2017).
142 The task was similar to one we have used previously (Strait et al., 2014; Strait et
143 al., 2015), with two major differences. First, monkeys gambled for virtual
144 tokens—rather than liquid—rewards. And, second, outcomes could be losses as
145 well as wins. Our previous research confirms that subjects’ behavior is consistent
146 with understanding of the link between colors and rewards and size and
147 probability in this task and in ones with similar structures, including more
148 complex foraging-like tasks – indicating that task understanding is not likely to be
149 a limiting factor here (Azab & Hayden, 2017; Blanchard & Hayden, 2015; Sleezer
150 et al., 2016).
151 Two offers were presented on each trial. Each offer was represented by a
152 rectangle 300 pixels tall and 80 pixels wide (11.35° of visual angle tall and 4.08°
153 of visual angle wide). 20% of options were safe (100% probability of either 0 or 1
154 token), while the remaining 80% were gambles. Safe offers were entirely red (0
155 tokens) or blue (1 token). The size of each portion indicated the probability of the
156 respective reward. Each gamble rectangle was divided horizontally into a top and
157 bottom portion, each colored according to the token reward offered. The size of
158 each portion indicated the probability of the respective reward. Gamble offers
159 were thus defined by three parameters: two possible token outcomes, and
160 probability of the top outcome (the probability of the bottom was strictly
161 determined by the probability of the top). The top outcome was 10%, 30%, 50%,
162 70% or 90% likely. The possible combinations of outcomes were: +3/0, +3/-1,
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
8
163 +3/-2, +2/+1, +2/0, +2/-1, +2/-2, +1/+1, +1/0, +1/-1, +1/-2, 0/0. Each non-safe
164 combination was equally likely to occur.
165 Six initially unfilled circles arranged horizontally at the bottom of the
166 screen indicated the number of tokens to be collected before the subject obtained
167 a liquid reward. These circles were filled appropriately at the end of each trial,
168 according to the outcome of that trial. When 6 or more tokens were collected, the
169 tokens were covered with a solid rectangle while a liquid reward was delivered.
170 Tokens beyond 6 did not carry over, nor could number of tokens fall below zero.
171 On each trial, one offer appeared on the left side of the screen and the
172 other appeared on the right. Offers were separated from the fixation point by 550
173 pixels (27.53° of visual angle). The side of the first offer (left and right) was
174 randomized by trial. Each offer appeared for 600 ms and was followed by a 150
175 ms blank period. Monkeys were free to fixate upon the offers when they appeared
176 (and in our observations almost always did so). After the offers were presented
177 separately, a central fixation spot appeared and the monkey fixated on it for 100
178 ms. Following this, both offers appeared simultaneously and the animal indicated
179 its choice by shifting gaze to its preferred offer and maintaining fixation on it for
180 200 ms. Failure to maintain gaze for 200 ms did not lead to the end of the trial,
181 but instead returned the monkey to a choice state; thus, monkeys were free to
182 change their mind if they did so within 200 ms (although in our observations, they
183 seldom did so). A successful 200 ms fixation was followed by a 750 ms delay,
184 after which the gamble was resolved and a small reward (100 μL) was
185 delivered—regardless of the outcome of the gamble—to sustain motivation. This
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
9
186 small reward was delivered within a 300 ms window. If 6 tokens were collected, a
187 delay of 500 ms was followed by a large liquid reward (300 μL) within a 300 ms
188 window, followed by a random inter-trial interval (ITI) between 0.5 and 1.5 s. If 6
189 tokens were not collected, subjects proceeded immediately to the ITI.
190 Each gamble included at least one positive or zero-outcome, ensuring that
191 every gamble carried the possibility of a win. This decreased the number of trivial
192 choices presented to subjects, and maintained motivation.
193 Eye position was sampled at 1,000 Hz by an infrared eye-monitoring
194 camera system (SR Research). Stimuli were controlled by a computer running
195 Matlab (Mathworks) with Psychtoolbox (Brainard, 1997) and Eyelink Toolbox
196 (Cornelissen et al., 2002). Visual stimuli were colored rectangles on a computer
197 monitor placed 57 cm from the animal and centered on its eyes (Figure 1A). A
198 standard solenoid valve controlled the duration of juice delivery. The relationship
199 between solenoid open time and juice volume was established and confirmed
200 before, during, and after recording.
201
202 Statistical Methods for Behavior
203 Subjective values for each gamble were estimated based on subjects’
204 choices in each test session according to the formula:
205
206 (1) SV = p * winα + (1 – p) * lossß
207
bioRxiv preprint first posted online Apr. 11, 2018; doi: http://dx.doi.org/10.1101/298844. The copyright holder for this preprint
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
All rights reserved. No reuse allowed without permission.
10
208 This formula comes from Yamada et al., (2013), although since our task
209 includes both wins and losses, we fit a parameter α for wins and another
210 parameter ß for losses. A value for α greater than 1 and a value for ß less than 1
211 both indicate risk-seeking. Both subjects were risk-seeking on average (values of
212 α > 1 or ß < 1 both indicate risk-seeking; subject B: average α = 1.21, average ß =
213 0.076; subject J: average α = 1.60, average ß = 0.022). For the remainder of this
214 study, “value” refers to subjective value.
215 We fit logistic regression models of behavior to predict choice of the first
216 vs. second offer. To ensure that subjects do, in fact, pay attention to both offers,
217 we fit a model where the value of the first and second offers were the predictors
218 of interest, while also including the number of tokens already accumulated, the
219 side the first offer appears on, and the choice eventually made to explain any
220 variance these variables might contribute to:
221
222 (2)
1
𝐹𝐹(𝑥𝑥) = 1+𝑒𝑒−(ß0+ß1(𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜1𝐸𝐸𝐸𝐸)+ ß2(𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜2𝐸𝐸𝐸𝐸)+ ß3𝑥𝑥3+ ß4𝑥𝑥4+ ß5𝑥𝑥5)
223
224 Where F(x) is the probability of choosing offer 1, x is the number of
3
225 tokens, x is the side of the first offer, and x is the choice that was made. To
4 5
226 determine whether subjects pay attention to all features of an offer, we use an
227 extended model with the three variables characterizing each offer (the two
228 possible outcomes, and the probability of the larger outcome) included as
229 predictors, controlling for the same variables mentioned above.
230