Prediction Errors Drive UCS Revaluation and not Classical Conditioning: Evidence and Neurophysiological Consequences

1.2 MB·

by  Luca Puviani

Prediction Errors Drive UCS Revaluation
and not Classical Conditioning: Evidence
and Neurophysiological Consequences
LucaPuviania,SiditaRamab,GiorgioVitettaa
aDepartmentofEngineering“EnzoFerrari”,UniversityofModena
andReggioEmilia,ViaVivarelli10,int.1-41125Modena,Italy
bMDatLocalHealthUnitofModena,ViaS.GiovannidelCantone
23,41121Modena,Italy
6
1
0
Abstract
2

n Nowadays,theexperimentalstudyofemotionallearningiscommonlybasedonclassicalconditioningparadigmsandmodels,
a whichhavebeenthoroughlyinvestigatedinthelastcentury. Onthecontrary,limitedattentionhasbeenpaidtotherevaluation
J
ofanunconditionedstimulus(UCS),which,asexperimentallyobservedbyvariousresearchersinthelastfourdecades,occurs

8 out of classical conditioning. For this reason, no analytical or quantitative theory has been developed for this phenomenon
2
anditsdynamics. Unluckily,modelsbasedonclassicalconditioningareunabletoexplainorpredictimportantpsychophysi-

ologicalphenomena,suchasthefailureoftheextinctionofemotionalresponsesincertaincircumstances. Inthismanuscript
]
C an analytical representation of UCS revaluation learning is developed; this allows us to identify the conditions determining
N the “inextinguishability” (or resistant-to-extinction) property of emotional responses and reactions (such as those observed
. inevaluativeconditioning,inthenonreinforcementpresentationofaconditionedinhibitor,inpost-traumaticstressdisorders
o
andinpanicattacks). Furthermore,ananalysisofthecausalrelationexistingbetweenclassicalconditioningandUCSreval-
i
b uation is provided. Starting from this result, a theory of implicit emotional learning and a novel interpretation of classical
-
conditioning are derived. Moreover, we discuss how the proposed theory can lead to the development of new methodolo-
q
[ giesforthedetectionandthetreatmentofundesiredorpathologicalemotionalresponses,andcaninspireanimalmodelsfor

resistant-to-extinctionresponsesandreactions.
1
v Keywords: Amygdala,classicalconditioning,conditionedinhibitor,emotionallearning,evaluativeconditioning,
6
misattribution,predictionerror,PTSD,resistant-to-extinction,UCSrevaluation,unconsciousemotion.
6
7
7
0
.
1
0
6
1
:
v
i
X
r
a
Emailaddress:[email protected](LucaPuviani)
Preprintsubmittedtoxxxx
Introduction
we also lay foundations for the development of new meth-
odsforthemodulationofemotionalreactiveresponses. Our
solutions to the above mentioned problems are based on the
Emotions are critical for the environmental adaptation of in-
analysisandtheinterpretationofvariousexperimentalresults
dividualsandfortheirsurvival. Infact,emotionsprepareany
acquired in different disciplines. In fact, we propose a uni-
organism to act in some specific ways without the need of
fyingframeworkthatprovidesacomprehensiveandcoherent
experiencing a physical effect that originates from a specific
descriptionoftheemotionallearningsystem.
sourceofstimulation; forinstance,theperceptionofasnake
evokesaninnatereaction[1,2,3]beforeexperiencing(orre-
The article begins by illustrating a multidimensional rep-
experiencing)aphysicalelicitation(e.g.,asnakeattack).This
resentation of the emotional response to a given stimulus.
is due to the fact that the representation of some stimuli and
The proposed representation, which is supported by experi-
the associated emotional reactions are innate [1, 2, 3], being
mentalevidenceshowingtheinvolvementofdistinctandspe-
shapedbyevolution.
cific neuronal populations in emotional responses [23, 24]
The emotional system plays also an important role in
(and is also supported by other experimental results based
many psychiatric and psychological diseases, in decision
on pharmacological conditioning [25, 26, 27]), accounts for
making [4] and in the clinical field (e.g., in drug treatment;
the different types of emotional response and, in particular,
[5, 6, 7, 8, 9]). Moreover, experimental evidence suggests for active responses (which are actively sustained by exter-
thatsubliminalemotionalstimulation(throughemotionalpic-
nal physical mechanisms, like pharmacological or mechani-
tures)canimprovesportorenduranceperformance[10,11], cal stimulation), reactive responses (which are not sustained
andthatemotionsinfluencepainperception[12,13,14]. This
by any physical mechanism, but “self-instantiated” by the
leads to the conclusion that any influence at emotional level emotional system) and passive residual responses (due to
maysubstantiallyaffecthumanbehaviorandperception.
theexcitatoryresidualsfrompreviousemotionalelicitations;
Nowadays,theexperimentalstudyofemotionsandoftheir [28, 29]). Moreover, in our representation plays an impor-
relatedphenomenaisusuallybasedonclassicalconditioning tant role the source attribution and misattribution phenom-
paradigms[15]. Classicalconditioningoccurswhenacondi- ena [30, 31, 32, 33, 34] and the modeling of contrast effects
tionedstimulus(CS)ispairedwithanunconditionedstimulus [35].Wethendevelopanovelmodeldescribingtheemotional
(UCS).Generallyspeaking,aCSisaneutral/innocuousstim- response of an organism elicited by a source of stimulation
ulus(e.g.,asoundoraneutralvisualcue),whereastheasso- (i.e., an UCS). The derivation of our model relies on the as-
ciatedUCSisasourceofstimulation(e.g.,anelectricshock sumption that, similarly as various mathematical models de-
or food). By repeated CS-UCS pairings, a CS can come to scribingclassicalconditioning(e.g.,Rescorla-Wagnermodel
elicit a conditioned response (CR), which is often similar to [36,37]ortemporaldifference(TD)models[38,39,40,41]),
the unconditioned response (UCR), i.e. to the response di- orprobabilistic(Bayesian)“perception”and“action”learning
rectly elicited by the paired UCS [16, 17, 15]; however, the models (i.e., the predictive coding (PC) [42] and the active
repetitive presentation of the CS without that of the paired inference models [43, 44]), coding of emotional and behav-
UCS leads to CR extinction, since the CS itself will not be ioral responses involves the computation of a specific error-
longerabletosignal(orpredict)theUCS. signal. In our work this error signal is defined as the dif-
Classical conditioning theory provides an explanation of ference between the response expected from the considered
the emotional learning mechanism involved in repeated CS- sourceofstimulationandtheresponseactuallyperceivedby
UCSpairings. However,thisisnottheonlyemotionallearn- the elicited organism. This definition is equivalent to that
ing mechanism, as originally understood by [18]. In fact, adoptedinTDandPCmodels,andreliesonexperimentalob-
Rescorla noticed the difference between the learning mech- servationsacquiredinfunctionalimagingstudies[45,38,46],
anisms involving two independent emotional memories, one or directly measured in dopaminergic circuits (e.g., in the
concerning CS-UCS pairing, the other one the UCS out- ventraltegmentalarea,VTA)orinotherfear-relatedcircuits
comeevaluation(andre-evaluation). Thisviewpointhasbeen [47, 48, 49, 50, 23, 51, 24, 52, 53]. The relationship be-
sharedbyotherresearchers(e.g.,see[19,20,21,22]). How- tweenthemechanismsofclassicalconditioninglearningand
ever,asfarasweknow,untilnownomathematical(orquanti- that of UCS revaluation learning is taken into consideration
tative)modelhasbeendevelopedforthesecondtypeofemo- next. In particular, we analyse the different mechanisms in-
tional learning, i.e. for UCS revaluation (in other words, volved in the encoding of a CS and of an UCS, and show
for the learning and the re-valuation of an UCS outcome). how the two resulting neural representations lead to specific
For this reason, no quantitative description can be given for differentproperties. Furthermore,weshowthata)whatisob-
many psychophysiological effects and phenomena driven by servedduringclassicalconditioningisactuallyaninterrelated
theemotionalsystem. Infact,thisrequirestheknowledgeof effectofbothassociativeandUCSrevaluationlearningmech-
a model able to porperly describe the dynamics of the inter- anisms, b) only the latter is driven by the emotional system
actionsbetweentheemotionalsystemandaprimarystimulus through the computation of emotional error signals (this im-
(i.e.,anUCS). portantresultissupportedbyexperimentalevidenceobtained
Inthismanuscriptwetackletheproblemsofdevelopinga fromoptogeneticmanipulations[54,55]).Aftershowinghow
noveltheoryandanewmodelforimplicitemotionallearning; the Rescorla-Wagner equation for classical conditioning can
2
be analitically derived from the proposed model for implicit theCNSitself. InthefollowingeachcomponentofaCNSre-
emotional learning under specific conditions, we propose a sponseisassociatedwithaspecificneuronal(e.g.,dopaminer-
morecompletemodelforPavlovianconditioning;indoingso, gic,serotonergic,opiodergic,etc.) populationwithinagiven
thestochasticHebbianplasticityrule[56,57,58,59,60,61] brainregion(e.g.,thepopulationofdopaminergicneuronsin
is exploited. The proposed model is able to predict specific the ventral tegmental area, VTA). For this reason, if N dis-
phenomenawhichcannotbeexplainedbythecurrentlyavail- tinctcomponentscanbeidentifiedintheCNS,theconsidered
able classical conditioning models (such as the dependence responsecanberepresentedasarowvectory=[y ,y ,...,y ]
1 2 N
of asymptotic responding on CS intensity and US intensity; belonging to a N-dimensional space, called CNS space in
[36,62]). Furthermore,wediscusshowtheconditionedinhi- the following; moreover, this vector can be expressed as
bition [63] and some related phenomena (such as the failure the linear combination of N versors (i.e., unit norm vectors)
of the extinction of conditioned inhibition through nonrein- {v;i=1,2,...,N}; this set of versors, being associated with
i
forcedpresentationsoftheinhibitor; [64,65]),whichcannot different neuronal populations, form a complete basis B for
be described in terms of classical conditioning [65, 36], can theconsideredspace. Then,wehavethat
bequantitativelypredictedbytheproposedtheory. Then,we
analysethecaseinwhichtheemotionalsystemiselicitedby N
y=∑yv, (1)
acontinuoustime-varyingsourceofstimulation(e.g.,atime- i i
i=1
varying acoustic stimulation, such as music; [66]), and we
showhowourmodelcanbeextendedtoprovidesomemath- where yi is a real quantity representing the product between
ematical and neurophysiological indications. Our model is the mean number of elicited neurons and their mean firing
thenexploitedtoquantitativelydescribesome(system-level) rates for the i-th neuronal population (with i = 1,2,...,N);
mechanisms leading to a resistance-to-extinction of an emo- consequently, yi takes on a positive (negative) value if the
tional response (i.e., inextinguishability over successive tri- response produces an increase of (a decrease or inhibition
als or over time) under certain conditions, and to illustrate of) the activity for the i-th population, and is equal to zero
some mechanisms through which an emotional reactive re- whenever the response does not involve any adjustment for
sponse associated with a primary stimulus can be artificially the baseline activity of the population. It is also important
strengthened/mitigated. This provides new insights on vari- topointoutthat,generallyspeaking,theversorsformingthe
ouspsychiatricpathologies,likepanicattacks[67],posttrau- basis B are not orthogonal, since a specific component can
matic stress disorder (PTSD; e.g., see [68, 69, 70]), and be (linearly or non-linearly, and directly or indirectly) re-
psychological phenomena like evaluative conditioning (EC) latedwithothercomponents. Thisfactisexemplifiedbythe
[71, 72, 73, 74, 75, 76, 77, 78, 79, 33, 80, 81] (which are dopaminergicnigrostriatalpopulation(associatedwithmotor
knowntoberesistanttoextinction[72,33]). functions)andthedopaminergicneuronswithinthemesolim-
Finally, we summarise our main findings and discuss the bicsystem(associatedwithmotivationandrewardfunctions);
potential implications of our results on future research on infact,thesepopulationsarenotsimplydifferentiatedfroman
emotionprocessing. anatomicviewpointandsignificantfunctionalinteractionsbe-
tweenthemhavebeenobserved(e.g.,see[82]andreferences
therein). Furthermore, given a specific region, a neuronal
Methods population can influence another population (e.g., the nora-
drenergicneuronalpopulationcaninteractwiththedopamine
neurons[83]).Fromaneurochemicalperspective,themotiva-
1. Responserepresentation tionfortheinterdependenciesbetweendifferentcomponents
is represented by the fact that a given neurotransmitter may
This Section is organized as follows. In Section 1.1 the
simultaneouslyinteractwithallthevariousisoformsofitsre-
emotionalandnon-emotionalcomponentsofthecentralner-
ceptoronneuronsthatalsoareundertheinfluenceofmultiple
voussystem(CNS)responsetoagivenstimulusareanalysed
otherafferentpathwaysandtheirtransmitters[84].
andvectorrepresentationsforsucharesponsearedeveloped.
The components forming the response elicited by a given
Then,inSection1.2theactiveandreactivecomponentsofan
stimulus can be classified on the basis of different criteria.
emotional response are introduced. Finally, the specific fea-
Afundamentalcriterionconsistsindistinguishingemotional
turesofthereactiveresponsearediscussedinSection1.3.
components from non-emotional ones. In practice, the emo-
tional component of a given response is due to all the emo-
1.1. Components and vector representation of a brain re-
tional and motivational neuronal systems and influences the
sponse
mesocortical limbic structures contributing to approach and
Generally speaking, in a mammalian CNS the response avoidancebehaviors[85]. Onthecontrary,thenon-emotional
(i.e., the effect) elicited by a given stimulus consists of dis- components originate from all the neuronal systems not be-
tinctcomponents,eachinvolvingamultitudeofneuronaland longingtotheemotional/motivationalsystems(e.g.,totheni-
receptorfamilies;thisresponse(oreffect)canbedetermined grostriataldopaminergicsystemorthesensorimotorsystem).
by hormonal, mechanical, acoustic, pharmacological and/or Note that, generally speaking, emotional and non-emotional
othereffectsoriginatingfromtheperiphericalsystemsorfrom componentscanbeinterdependent; forinstance, theycanbe
3
toasourcestimulusthroughclassicalconditioning;neverthe-
less,evenaCSpreviouslypairedwithaprimarystimulus(i.e.,
anUCS)isabletoelicitanemotionalresponse.Inthefollow-
ing we assume that a given UCS is always able to elicit an
Active Elicited emotionalcomponent(e.g.,anelectricshockdeviceorfood),
Response sothatboththetermsUCSandemotionalsourceofstimula-
tioncouldbeusedindiscriminately.
Emotional Definition2. Activestimulationandactiveresponse
Sub-Space CNS Space
An active emotional response is defined as any emotional
componentelicitedbyaprimarystimulusthroughdirectphys-
ical(e.g.mechanical,chemical,pharmacological,etc.)mech-
Figure1:BidimensionalrepresentationoftheCNSspaceandtheemotional
anisms. Furthermore,thestimuluselicitationisdefinedasac-
sub-space.AgenericelicitedresponseintheCNSspaceisalsosketched.
tivestimulation.
It is worth mentioning that the adjective “active” refers to
causallyrelated(e.g.,physicalandemotionalpainsarecorre-
thefactthattheconsideredresponseissubstainedthroughan
lated) or indirectly related (e.g., emotional components can
activeandphysicalactionoftheconsideredexternalsourceof
influence the immune response [86, 5, 87, 88, 89]). This
stimulation.
differentiationbetweenemotionalandnon-emotionalcompo-
Real world examples of a stimulus eliciting an active re-
nentscanbeincludedinthevectormodelsketchedaboveby
sponseareprovidedbyadrugabletostimulateanemotional
identifyinganemotionalsub-spacewithintheCNSspace,as
or motivational component (through pharmacological mech-
exemplifiedbyFig. 1.
anisms), a pain stimulus (originating, for instance, from an
Furthermore,theemotional(non-emotional)responsecom-
electricshockdelivery),orfood.
ponentsarecollectedinthevectory (y )andoverallCNS
em ne Generallyspeaking,theactiveresponsecanberepresented
canberepresentedastheconcatenationofthetwoaforemen-
asanon-linearandtime-varyingfunctionofthephysicalstim-
tionedvectors,i.e. as
ulation generating it and of the internal physiological states
(notethatthetime-varianceofthisfunctionaccounts, forin-
y=[y ,y ]. (2)
em ne stance,forhabituationeffectorforreceptorupregulationand
Finally, it is worth noting that our response representa- downregulation[84]). Moreover,theactiveresponsemaybe
tion is suited to properly model various experimental re- influenced by various physical features of the source stimu-
sultsevidencingthatthemammalianCNSisabletodiscrim- lus,suchasitsfrequency,rateofchangeandmagnitudeofthe
inatedifferentemotionalrewardingcomponents(e.g., differ- elicitation.
entdopaminergicpopulationsrelatedtoappetitiveorreward-
Definition3. Reactivestimulationandreactiveresponse
ing stimulations). In particular, some results illustrated in
[23, 24] have lead to the conclusion that neurons can dis- A reactive emotional response (or non-active response) is
criminate between cocaine and liquid rewards (or between defined as any emotional component “self-induced” by the
cocaine and heroin), possibly even better than between nat- perceptionofastimulus; sucharesponseisnotactivelysus-
ural rewards, and that in rats some neurons can be activated tained by any external physical (e.g. mechanical, chemical,
by lever pressing regardless of drug injection, whereas other pharmacological, etc.) mechanism. Furthermore, the stimu-
neurons by pressing a lever of a specific drug or natural re- luselicitationisdefinedasreactivestimulation.
wardsonly. Thisshowsthateveryexternalstimulus,suchas
Itisworthnotingthattheadjective“reactive”referstothe
food, a natural liquid reward or a rewarding drug, is able to
fact that the considered response is reactively eicited as the
elicitintheratCNSsomespecificemotionalcomponentsre-
stimulusisperceivedanditisnotsubstainedthroughanydi-
latedtospecificneuronalpopulations.
rectphysicalmechanism.
A real word example of a stimulus eliciting a reactive re-
1.2. Activeandreactiveemotionalresponses
sponseisprovidedbyaCSpreviouslypairedwithaprimary
Inthismanuscriptthefollowingdefinitionsareadoptedin stimulus; in this case the elicited reactive response is called
relationtoasourceofemotionalstimulation. coditionedreflex[15].Thenon-activeresponsecanbeelicited
alsothroughthemereperceptionofaprimarysourceofstimu-
Definition1. Sourceofemotionalstimulation lation,suchasathreateningstimulus(e.g.aspiderorasnake)
or food. Such a source of stimulation could be innate (e.g.,
Anemotionalsourceofstimulationisdefinedasanysource
represented by a biological/phylogenetic fear-related threat,
orprimarystimulusabletoelicitaresponseinvolvinganemo-
suchasaspider,asnakeoranangryface[1,2,90])orlearned
tionalormotivationalcomponent.
(e.g.,anontogeneticsourcelikeagun[1]).
Notethattheterm“primarystimulus”isemployedhereto Itisimportanttopointoutthat, fromanevolutionaryper-
excludeanysecondarystimulus(i.e.,CS[15,37])conditioned spective, emotional reactive responses are fundamental for
4
the survival of individuals since they can elicit behavioral placebo/nocebo effect can be interpreted as a conditioning
responses without the need for individuals to phisically re- process in which a primary stimulus (e.g., a drug) become
experiencing a given source elicitation (e.g., a snake bite), paired with a CS (e.g., the substance administration or other
or they act to turn individual attention to positive valenced cues) [5, 94, 95, 96, 97, 98, 99, 100, 101, 102, 8, 103]. Af-
sources, such as food. For this reason, reactive responses ter some effective pairings, a successive presentation of the
arelearnedthroughtheemotionalsysteminordertoinducea CS(e.g.,theadministrationofaninertesubstance),intheab-
properreactionwheneverindividualsfacesourcesofstimula- senceofanyactivephysical/pharmachologicalstimulation,is
tionorothercuessignallingaprimarystimulus. Asitwillbe abletoelicitapurelyreactiveresponsewhichmimicsthepre-
discussedinmoredetailinSection2.1,reactiveresponsesare viousactiveresponseduetotheprimarystimulus. Thisclaim
elicitedthroughtheamygdala,since,intheabsenceofanin- is confirmed by various brain imaging data, evidencing that
tactamygdala,noreactiveresponseorconditionedreflexcan placeboscanmimictheeffectofactivedrugsandactivatethe
occur[91].Itisalsoworthmentioningthataprimarystimulus samebrainareas;thisoccursforplacebo-dopamineinParkin-
isabletoelicitbothanactiveemotionalresponseandareac- son’s disease, for placebo-analgesics or antidepressants, and
tiveemotionalresponse;onthecontraryasecondarystimulus for placebo-caffeine in healthy subjects (see [27] and refer-
(i.e.,aCS)canelicitareactiveresponseonly. ences therein). Moreover, in [25] it is shown that pharma-
Given the definitions and the considerations illustrated cological conditioning, like conditioning with opioids, pro-
above,theemotionalvectory canbeexpressedas duces placebo analgesia mediated via opioid receptors ad-
em
ministering saline infusions, and that, if conditioning is per-
y =y +y (3) formed with nonopioid drugs, other nonopioid mechanisms
em aem rem
areinvolved, sothatconditioningactivatesthesamespecific
where y (y ) represents the active (reactive) portion of
aem rem neuronalpopulationsastheprimarystimulus. Similarresults
y .
em andconclusionshavebeenobtainedinmiceexperiments[26].
In the literature various results are available about the la-
This line of reasoning is also supported by a further experi-
tency and decay of emotional responses. First of all, the
ment [104] showing that an increase in dopamine release in
latency of an emotional response is usually deemed negligi-
theventralstriatum,measuredthroughmicrodialysis,areob-
ble; however, its decay is a time-consuming process for all
servednotonlywhenratsselfadministercocaine(represent-
nontrivial or pronunced emotional states [31, 92, 93, 28]. In
ingtheUCSintheconsideredexperiment),butalsowhenthey
analysing the decay of a response, it should be always kept
aresolelypresentedwithatone(representingtheCS)thathas
intoaccountthatanyemotionalresponsecanbedecomposed
beenpreviouslypairedwithcocaineadministration. Further-
in two main functional factors, namely the dominantly neu-
more, experimental verification of the influence of noncon-
rally controlled factor and the dominantly humorally con-
sciousconditionedstimulionplacebo/noceboeffects[98,99]
trolled factor. The former factor has a faster decay after the
showthatareactivestimulusisabletointerferewithagiven
end of the stimulation, whereas the latter one is relatively
active stimulation (e.g., an active drug or a painful stimula-
slower [28] and can elicit neural responses. For this reason,
tion), by increasing or decreasing the effect of the active re-
thedifferentfactorsofbothreactiveandactiveresponsesex-
sponse. This suggests that common active and reactive re-
tinguishaftertheirelicitationatdifferentspeeds. Forthisrea-
sponsecomponentscanbeadditiveorcompetingand,hence,
son, it can be always assumed that at least a portion of both
bothcontributetothedeterminationoftheoverallelicitedre-
activeandreactiveresponsesextinguishesquiteslowly(e.g.,
sponse. Thelastobservationisalsosupportedbyfurtherex-
this portion is related to hormones and neuromodulators de-
perimentalresults[12,13,14]whichshowthatemotionalre-
cay). Moreover,atagiveninstantagenericactiveorreactive
activestimulations(e.g.,theperceptionofemotionalpictures
response can be in its active state (e.g., instantiated, or dur-
orotherreactivestimuli)modulatepainperception. Forthese
inganactiveelicitation)orinitspassivestate(i.e.,duringthe
reasons, if the active and the reactive elicited responses in-
decayintervalfollowingtheendofanactiveelicitation).
volve some common components, these, in turn, add up in
ThemostimportantideasillustratedinthisSectioncanbe
an algebraic sense (see Eqs. (3) and (1)). A further support
summarizedinFig. 2,whichshowsthedifferenttypesofthe
ofthisconclusioncomefromtheexperimentalevidencesde-
emotionalresponsethatcanbeelicitedbyagenericsourceof
scribedin[31,92,28,29]whichshowthat,undersomecon-
stimulationandthemathematicalnotationadoptedtodenote
ditions,anemotionalexcitationcanbetransferredtoasucces-
them.
siveindependentsourceofstimulation,energizingit,because
of the residual excitation due to the incomplete decay of the
1.3. Onthenatureofreactiveemotionalresponses
previousemotionalelicitation. Thisphenomenoniscalledex-
AsalreadystatedinthepreviousSection,agenericCScan citationtransfer[28,29].
elicit a reactive emotional response only, unless it also rep-
resents a primary source of stimulation. For this reason, the Additionalresultsillustratedin[105]revealthedifferenti-
observationoftheresponseselicitedbydifferentCSscanun- ationbetweenemotionalandnon-emotionalcomponents,and
veilthespecificpropertiesofreactiveresponses. Variousre- reactive and active responses in nociceptive stimulation (in
sults are already available about this issue in the literature. thiscasethepainnetworkinvolvesbothaffectiveoremotional
In particular, it is well known that the unconscious/implicit components, and sensory components). In particular, func-
5
NON EMOTIONAL
RESPONSE ACTIVE STATE
RESPO NSE ( y n e ) A(PChTyIEsViMcEaO RllTyEI OsSuNPbOAsLNta SinEe d
( y ) EMOTI ONAL by( ay Sa e om u r c) e) (P HAuSmSoIVraEl DSTecAaTyE)
RESPONSE
( y e m )
EMOTIONAL
REACTIVE ACTIVE STATE
RESPONSE
( Innate or Learned )
( y r e m ) PASSIVE STATE
( Humoral Decay)

Figure2:RepresentationofthedifferenttypesofCNSresponseelicitedbyagenericsourceofstimulation.
tional imaging experiments have evidenced that both phys- experimentalresultsrelatedtoconditionedimmuneresponse
ically receiving a pain stimulus (i.e., an active stimulation) and pharmacological conditioning with immunosuppressive
orobservingasignalindicatingthatalovedperson-present drugs [5, 87, 108]. Note also that Remark 4 is fundamental
in the same room - is receiving a similar pain stimulus (i.e., onthedescriptionoftheemotionaldynamics(seeSection3).
a reactive emotional pain-related stimulus mediated through
Remark5.
empathy)activatesthebilateralanteriorinsula(AI),theros-
Ifareactiveandanactiveresponsesinvolvethesamecom-
tral anterior cingulate cortex (ACC), the brainstem and and
ponents, the reactive response can strenghten/weaken (ener-
the cerebellum. These results have also shown that, on the
gize/inhibit) the emotional components of the primary ac-
contrary,activityintheposteriorinsula/secondarysomatosen-
tive response since they are qualitatively indistinguishable.
sory cortex, the sensorimotor cortex (SI/MI), and the caudal
Hence,Eq. (3)canbeapplied.
ACCwasspecifictophysicallyreceivingpain(i.e.,thatnon-
AsitwillbecomeclearerinSection3,thisremarkwillplay
emotionalcomponentswereexclusivelyduetotheactivere-
afundamentalroleinthedescriptionofemotionaldynamics.
sponse)[105].
All the results mentioned above evidence that a CS (or a One can argue that, in some cases, a reactive response
generic reactive stimulus) can elicit specific neural popula- learned from its active counterpart does not mimic the orig-
tionsandthatthesepopulationsreflectormimicthesamesys- inalemotionalresponse. Forinstance,thisoccursinaversive
tems elicited by the primary stimulus paired with the condi- conditioning,whenanelectricshockisusedasunconditioned
tionedCS;inotherwords,asfarastheemotionalcomponents stimulus; in particular, it has been observed that heart rate
are concerned, a reactive response mimics the original ac- decreases during the presentation of a CS and, on the con-
tiveresponse. Moreover,activeandreactiveresponseswhich trary,itincreaseswhenanactiveelectricshockisgiven[109].
involve the same emotional components add up in algebraic For this reason, at first glance reactive and active responses
sense. Thisobservationleadsustothefollowingtwoimpor- seem to be qualitatively different as they lead to distinct be-
tantremarks: haviors. Actually, a different interpretation of these results
canbeformulated. Infact,itshouldnotbeforgottenthatthe
Remark4.
activeresponsetoapainstimulusconsistsoftwocomponents
Areactiveresponsecanmimic,atleast,theemotionalsub- (one emotional, the other one non-emotional) and the non-
spaceofapreviouslyassociatedelicitedactiveresponse. emotionalcomponentcouldresultinabehaviorsubstantially
Nonetheless, we argue that a reactive response can cer- differentthanthatproducedbytheelicitationoftheemotional
tainly determine also some non-emotional components pro- componentalone. Moreover,generallyspeaking,inthecom-
vided that these are causally related to the emotional ones plexmammalianCNS,differentlevelofaspecificneurotrans-
(e.g., the emotional system can influence the immune re- mitterwithinagivenbrainregion(determinedbytheelicita-
sponse [86, 5, 87, 88, 89], or the dopaminergic mesolimbic tionofaspecificemotionalcomponent)couldleadtodifferent
system can interact with the nigrostriatal dopaminergic sys- (even opposite) behavioral observations [84] (this last asser-
tem[106,107,82]). Furthermore,wearguethateventhehu- tionwillbediscussedinmoredetailinSection6.2).
moral immune response sub-space (in particular the compo-
nentsoftheCNSsuchasthehypothalamic-pituitary-adrenal
2. Error-drivenlearning
axis, HPA, or the sympathetic nervous system, SNS; [87]),
otherthantheemotionalsub-space,canbemimickedfroman In this Section the basic mechanisms involved in emo-
activeresponseelicitation; thislastassertionissupportedby tional learning are reviewed; in particular we focus on the
6
roleplayedbytheamygdala(andotherimportantsystems)in whereas in the latter one onto the centromedial amygdala
emotionallearningandonthemechanismsforthegeneration (CeM)[112].
oftheerrorsignalsonwhichsuchalearningisbased.
2.2. Onthegenerationoferrorsignalsinemotionallearning
2.1. Ontheroleoftheamygdalainemotionallearning
Neuronsinseveralbrainstructuresappeartocodespecific
In complex vertebrates the amygdala represents the core signals, that are called error signals and, generally speak-
centerintheformationandstorageofemotionaleventsandin ing,representthedifferencebetweenareallyexperiencedre-
theelicitationofemotionalresponses. Inparticular,itiswell sponseanditsexpectedcounterpart[51,24]. Intheliterature
knownthattheamygdalaplaysafundamentalroleinencod- anumberofresultsareavailableabouttheroleplayedbyspe-
ingemotionalmemories,infearresponsesandinskinconduc- cificneuronalpopulationsincodingerrorsignalsandthena-
tanceresponse(SCR)classicalconditioning. Infact,various tureofsuchsignals.Inparticulartheerrorsignalsarecodedin
results are available in the literature about the involvement relationtorewards,punishments,externalstimuli,andbehav-
of the amygdala in the acquisition and encoding of relevant ioralreactions[48,51,53].Insomecases,dopamineneurons,
emotionalmemories[110,111,112,113,114],andaboutthe norepinephrine neurons, and nucleus basalis neurons broad-
fact that these mechanisms are based on its synaptic plastic- cast prediction errors as teaching signals to large postsynap-
ity [115, 112, 116]. In addition, in a growing body of liter- tic structures; in other cases, error signals are coded by se-
ature [117, 118, 119, 120, 121, 122] it is shown that amyg- lected neurons in the cerebellum, superior colliculus, frontal
dala is necessary for fear responses, and no reactive fear re- eyefields,parietalcortex,striatum,andvisualsystem,where
sponsesareinstantiatedintheabsenceofanintactamygdala theyinfluencespecificsubgroupsofneurons. Ingeneral,pre-
[118]. Finally, theanalysisofpatientswithadamagedbrain diction errors can be used in postsynaptic structures for the
has evidenced the importance of amygdala in SCR classical immediate selection of behavior or for synaptic changes un-
conditioning. Thisemerges, forinstance, fromtheresultsil- derlying emotional and behavioral learning [46, 51]. Evi-
lustrated in [91], where the case of a patient with selective dences of coding of error signals during learning have been
bilateraldestructionofhisamygdalaisanalysed;infact,this foundinvariousneuroimagingstudies[45,38,46].
patient shows an unconditioned SCR to a UCS, but no SCR Morespecifically,asevidencedbyagrowingbodyofliter-
conditioning to the paired CS, although he is well aware of ature[133,48,134,23,24,53],inemotionallearning,popu-
theCS-UCSrelation. lationsofdopaminergicneuronsencodetheerrorsignaleval-
Alltheresultsillustratedaboveleadtotheconclusionthat uating the difference between what is expected (i.e., the ex-
theamygdalaisnecessaryfortheelicitationofanemotional pectedreward)andwhatisreallyoccuring;furthermore,this
reactive response; note, however, that, if the amygdala is error signal is exploited to correct and modulate the individ-
damaged, an active elicited response (e.g., an unconditioned ual’s emotional and behavioral response. The error signal
painfulstimulus)canbestillelicited.Furtherresultsavailable computed in these dopaminergic regions can be positive or
intheliteratureevidencethattheamygdalamediatestheemo- negativeandcandriveappetitiveoraversiveemotionalreac-
tional reaction, and that directly and indirectly elicits emo- tions[48].
tional and motivationalareas of human brain [123,54, 124]. On the basis of all the above illustrated results, it can be
Inparticular,onthebasisofinformationcomingfromthesub- statedthaterrorsignalsdrivingemotionalresponsesareeval-
cortical (i.e. thalamus) and cortical pathways (from which uated in different brain regions, depending on the nature of
perceptionandtherepresentationofthefeaturesofanysource the involved emotional components; however, in emotional
originates)theamygdalaarousesboththecortexandtheemo- learning (and in the computation of the associated error sig-
tional and motivational brain regions directly and indirectly nal),afundamentalroleisplayedbytheorbitofrontalcortex
through different systems [125, 126], that is the nucleus ac- (OFC)[135], whichrepresentsakeystructureincodingand
cumbens(NAcc),theprefrontalcortex,themidbrain,thehy- maintaining the representations of a stimulus response (i.e.,
pothalamus, the autonomic nervous system, the endocrine the representation of the expected outcome associated with
system and others). In the following analysis we briefly re- a stimulus) [136, 135]. In fact, various experimental results
fer to this group of systems as system chain. In fact, the have evidenced that the OFC generates information about
amygdalasendsprojectionstoavarietyofsystems,anditcon- expected outcomes (e.g., see [137] and references therein),
sistsofseveralinteractingsubnucleithatmayprovidespecific which are demeed critical in the computation of prediction
individual contribution to the overall emotional computation errors; these results are consistent with the relation between
[127,126,125](inparticular,differentsubnucleioftheamyg- the reward-related activity in OFC and VTA dopamine neu-
dala can process and elicit specific emotional components rons [137]. Experimental results have also evidenced that,
[128, 129, 130, 54, 127]). Research activities in this field whenOFCandmidbraindataarejuxtaposed,anticipatoryac-
havealsoevidencedthatthattherepresentationofanyUCSis tivity observed in the OFC is inversely related to dopamin-
storedwithinthebasolateralamygdala(BLA)[54,55]. Note ergic error signaling downstream [138]. This suggests that
alsothattheamygdalamediatesbothappetitive(i.e. reward- the error signals in other brain areas might depend partly on
ing)andaversivestimuli[117,54,131,120,121,122,132]; OFCinputforproperlycalculatingtheerrors[139,138].This
in the former case the BLA neurons project onto the NAcc, ideahasbeenpartiallyconfirmedforerrorsignalsinmidbrain
7
dopamine neurons (at least in rats; see [140]). The OFC re- humanbrainisassumedtolearnfromapredictionerror,de-
gion has been also shown to respond to both appetitive and finedasthediscrepancybetweenareferencevalueandwhat
aversive outcomes [141], to integrate multiple sources of in- is actually perceived by the considered subject [36, 37]. In
formation regarding outcome signals, to code outcomes and particular,learningoccursthroughamechanismthatupdates
supervise the amygdala (which, as already explained in the theexpectationsabouttheoutcomeinproportiontoapredic-
previous Section, is the key element for both positive and tion error, so that, across trials, the expected outcome con-
negative-valence emotions) [142, 143, 144], and to integrate vergestotheactualoutcome[37]. AvariantoftheRescorla-
cognitive information [145] (also coming from the dorsolat- Wagnertheoryisrepresentedbythesocalledtemporaldiffer-
eral prefrontal cortex [146]) in its evaluation and coding of ence (TD) learning [38, 39, 40, 41], which accounts for the
emotional outcomes [135]. Finally, it is worth pointing out timeevolutionoftheresponsewithineachtrial. Thegoalof
that OFC is not necessary for Pavlovian conditioning, since TDlearningisprovidingaprediction,foreachinstantt inthe
thiscanbedrivenbypriorexperience;however,itiscertainly trialduringwhichaCSispresented,ofthetotalfuturereward
neededformodifyingtheresponseifthepredictedoutcomeis tobegainedinthattrialfromtimettotheendofthetrialitself
revaluated(i.e.,UCSinflationanddevaluation)[138,147]. [38].
A more recent theory (known as predictive coding theory
2.3. Onerror-basedemotionallearning
[42]) formalizes the notion of the Bayesian brain, in which
Onthebasisoftheexperimentalresultssummarisedinthe neural representations in the higher levels of cortical hierar-
previous Section, it can be assumed that, during emotional chies generate predictions of representations in lower levels.
learning, an error signal is computed in different brain re- These top-down predictions are compared with representa-
gions(e.g.,theVTA),dependingonthenatureoftheelicited tions at the lower level to compute a prediction error. The
emotional response. In this process the OFC plays an im- resulting error-signal is passed back up the hierarchy to up-
portant role since it interacts with these brain regions and date higher representations; this recursive exchange of sig-
codestheexpectedresponsetoagivenstimulus(i.e.,theex- nalsleadtotheminimization(ideallythesuppression)ofthe
pectedUCR).Moreover,theerrorsignalwhichisbroadcasted prediction error at each and every level to provide a hierar-
to other brain regions including the OFC, may undergo pro- chical explanation for sensory inputs that enter at the low-
cessing and fusion with other information (even at a higher est (sensory) level. In the Bayesian jargon neuronal activity
cognitive level) and is certainly sent to the amygdala, since encodes beliefs or probability distributions over states in the
thispartofthebrainisnecessaryfortheelicitationofanemo- world that causes sensations [42]. In predictive coding the-
tional reactive response, for conditioning and for the storing ory the notion of precision (or confidence, which is the in-
processofemotionalstimuli. verse of the variance) of the error signals is also formalised,
It is also important to point out that the extended network and the mechanism through which the brain has to estimate
forcomputingandcodingtheemotionalerror-signals(which andencodetheprecisionassociatedwiththepredictionerrors
we call error computing distributed network) can be repre- is explained. The prediction errors are then weighted with
sented as a distributed system consisting of different sub- their precision before being assimilated at a high hierarchi-
systems and, in particular, the OFC, the prefrontal cortex callevel. Generallyspeaking,predictivecodingassumesthat
(PFC), the VTA, the midbrain, the striatum and others; all organisms minimize an upper bound on the entropy of sen-
these interact in an iterative fashion. However, the study of sory signals (the free energy), which, under certain simpli-
these multiple neural interactions is behind the scope of our fying (Gaussian) assumptions is equivalent to the prediction
analysis. For this reason, in the following we assume that, error. The minimization of the error-signal (at the different
whenever a stimulus elicits an emotional response, an error hierarchicallevelsofaneuralnetwork)isgenerallycomputed
signal is computed and transmitted to the amygdala for up- through a generalized gradient descent [44]. Hence, predic-
dating the corresponding reactive emotional response; note tivecodingtheoryleadstomodelthebrainactivitybymeans
that this error-driven mechanism ensures adaptivity in emo- of a neural network with multiple neuronal layers (from the
tionallearning. Asitwillbeshowninthefollowingsections, tophighhierarchicallevel,tothebottomsensorylevel)which
thismechanismcanbequantitativelydescribedandanalysed is governed by the backpropagation algorithm [148]; this al-
by representing the emotional learning system as a dynamic gorithmleadstotheminimizationoftheerrorsignal,adjust-
system,characterisedbymemoryelements(duetotheamyg- ing,iteratively,theweightsofthepredictionerrorsatdifferent
dalaandtotheOFC),fedbyatime-variyngerrorsignal,and network layers (such weights correspond to the precisions).
generatingtheemotionalresponses(i.e.,thevectoryeminEq. In machine learning and engineering fields artificial neural
(3)). networkshavebeenextensivelyappliedtoalargenumberof
Itisusefultomentionthattheideasofrelatingbrainlearn- learning problems [148] (for classification, regression or pa-
ing to an error-prediction signal and of modelling this pro- rametersestimation). Hencethroughaneuralnetworkarchi-
cessasadynamicadaptivecontrolsystemhavebeenalready tecture,inprinciple,anynonlinearfunctionordistributioncan
developedinvarioustheories. Themostrelevantexampleof belearned.Predictivecodingtheoryandtheabovementioned
this approach is provided by Rescorla and Wagner in their conceptshavebeensuccesfullyappliedtoperception[42]and
theory about classical conditioning. In fact, in this case the motion (action) learning (the so called active inference the-
8
ory; [43, 44]) to communication learning [149]; moreover, 1.3,amathematicalrepresentationbasedonmultidimensional
apredictivecodingversionofinteroception(i.e.,amodelfor vectorsshould beadoptedtoproperly describetheerrorsig-
theautonomic,metabolic,immunologicalregulationsso,gen- nal involved in emotional learning and defined as the differ-
erally speaking, for homeostasis and allostasis, and, poten- encebetweentheactuallyperceivedresponseanditsexpected
tially, even for emotional regulations) has been proposed in counterpart;thisisduetothefactthatmultiple(andspecific)
[150, 151, 152]. Nonetheless, we will show (see Section 5) emotional components could be elicited by aprimary stimu-
that the interoceptive predictive coding model, which could lus. However,inthefollowingwefocusonthedynamicsofa
holdforinteroceptivevisceromotorfunctions(i.e.,homeosta- singlecomponent toeasethereading. Thischoice, however,
sisandallostasis),cannotbedirectlyappliedtoimplicitemo- doesnotentailanylossofgenerality,sinceourmodelcanbe
tional learning, since it would lead to the instability of the appliedtoanycomponent.
emotionalsystem. Indistinguishability between the emotional component of
an active response and the corresponding reactive learned
2.4. Anewmodelforerror-basedemotionallearning response - As illustrated in Section 1.3, a learned reactive
response mimics the emotional components of the active re-
Inthefollowinganerror-basedmathematicalmodelforde-
sponse elicited by a primary stimulus. For this reason, we
scribing the dynamics of implicit emotional learning is de-
assume that the reactive and the active responses add up, so
veloped. The main features of the proposed model, and its
generatingtheemotionalcomponenty (seeEq. (2)andRe-
em
similaritiesanddifferenceswiththemodelsmentionedatthe
mark5).
endofSection2.3aresummarisedbelow.
The essential neural network and the related neural func-
A model for primary stimulus learning - The proposed
tions - The considered error signal is computed by a dis-
modeldescribestheimplicitlearningofaUCSoutcome(i.e.,
tribuited and dynamically interconnected neural network
the UCS acquisition, inflation and devaluation: the UCS
whose master element is supposed to be the OFC (see Sec-
revaluation) and, consequently, does not immediately refer,
tions2.2and2.3). Thiserror-signalistransmittedbythenet-
unlike the classical conditioning models (like the Rescorla-
worktotheamygdala, which, inturn, updatestheemotional
Wagner and TD models), to the learning of the associative
reactiveresponseassociatedwiththerepresentationofthepri-
connection between a UCS and a paired CS [18]. Further
mary stimulus. The processing of the error signal inside the
detailsaboutthisareprovidedinSection6.2,wheretherela-
amygdala is represented by the (unknown) amygdala func-
tionshipbetweenthesetwolearningmechanismsisanalysed
tion,denotedF (·)inthefollowing.
indetail. A
Implicit emotional evaluation - In the development of our As already discussed in Section 2.1, whenever a primary
modelitisassumedthatnocognitiveorsuggestionprocesses stimulus is perceived, the amygdala elicits the reactive re-
occur during the interactions between a subject and the con- sponse previously coded for that stimulus. In particular, the
sideredUCS;forthisreason,ourattentionfocusesononlyim- amygdala is be able to arouse the cortex, and the emotional
plicit(orautomatic)emotionalevaluation(inotherwords,in- and motivational brain regions directly and indirectly; this
formationsuchasverbalsuggestions,cognitiveexpectations, process involves the system chain described in Section 2.3
beliefs and so on, are avoided). Nevertheless, the source of andoperatingbetweentheamygdalaoutputandtheemotional
stimulation has to brought to the attention of the subject. In brain systems. In the proposed model, the biological func-
particularithastobeevidentthattheUCSisresponsiblefor tionality of this system chain is described by the (unknown)
theresponseelicitation,sothattheoutcomewillbecorrectly systemchainfunctionFCh(·). Notethatthisfunctiondoesnot
attribuitedtothatsource. Thisassumptionexludesthesitua- lenditselftoasimpledescription,sinceitisinfluencedbyiter-
tioninwhichtheoutcomeismisattribuitedtoanotherinsignif- ativemechanismsinvolvingneuronalandhormonalsystems,
icant stimulus [31, 32, 33, 34], or even not attributed (like and various brain regions. In practice, FCh(·) represents the
inthecaseofhiddendrugadministration). Furthermore, the processing which turns all the amygdala output signals into
stimulusremainsunchangedinthesequenceoftrials. Under emotionaland motivationalresponses. These ideasaresum-
suchassumptions,withoutanylossofgenerality,itcanbeas- marisedinFig. 3,whichshowsthefunctionalrepresentation
sumedthatthepredictedresponse(i.e.,theexpectedoutcome) proposedfortheimplicitemotionallearningsystem.
inagiventrial,forthegivenstimulus,willcoincidewiththe Functionalapproximations-Generallyspeaking,thefunc-
response experienced during the last source-subject interac- tions F (·) and F (·) may change over time, and are ex-
A Ch
tion. Suchanassumptionsimplifiesthefollowingmathemati- pected to be influenced by the internal physiological states
calcomputationsanddoesnotleadtoanylossofthegeneral- atthetimethestimulusisencountered[153,154,155,156],
ity,since,generallyspeaking,onemightevenassumethatthe external circumstances and various features of the eliciting
predictedoutcomeconvergesduringtrials(orduringtime)to stimulus. Moreover, it is expected that the amygdala func-
therealexperiencedoutcomethroughalearningmechanism. tiondependsalsoontheelicitedsensorymodality(e.g.,audi-
In the latter case the convergence of the emotional response tory, visual, olfactory). Finally, the behavior of these func-
towardsasteadystatewillsimplybeslower. tions change from component to component. However, in
Multidimensional nature of the error-signal and evalua- the following it is always assumed, for simplicity, that these
tionofasinglecomponent-AsalreadymentionedinSection functions are static; consequently, the influence of all the
9
NON EMOTIONAL
COMPONENTS
NON EMOTIONAL NON EMOTIONAL
( somato-sensory cortex…)
ELICITATION RESPONSE

ACTIVE
STIMULATION
RESPONSE
( Physical stimulation ) y
ACTIVE EMOTIONAL EMOTIONAL
ELICITATION ( x n ) RESPONSE
EMOTIONAL COMPONENTS
iR,nFCh(iA,n) ( lymbic syste m, insula…) yem,nxniR,n1
ORBITO FRONTAL
CORTEX
y
SYSTEM CHAIN expected
NETWORK
STIMULUS
F ()
Ch
ERROR COMPUTATION
DISTRIBUTED
NETWORK
( midbrain, striatum..)
AMYGDALA
iA,niA,n1FA(en) FA () en  yem,n yexpected , n

REACTIVE
STIMULATION TALAMUS CORTEX HIPPOCAMPUS
(visual, uditory…)
(Perceptive stimulation)
Figure3: Discrete-timerepresentationofthemechanismsonwhichtheimplicitemotionallearningsystemisbased. Thefunctionalconnectionsbetween
theinvolvedactorsandtheprocessingtaskaccomplishedbyeachblockareshown. Inparticular,astimuluscouldexertanactivestimulation,areactive
stimulationorboth;incaseofareactivestimulation,thestimulusisperceivedand,successively,theamygdalaelicitstheassociatedreactiveresponse. The
amygdalaoutputisbeingprocessedbythesystemchainnetwork,whichinvolveallthesystemsbetweentheamygdalaoutputandtheemotional/motivational
brainregions(e.g.,thenucleusaccumbens,thesympatheticsystem,thehypothalamusandothers). Theorbitofrontalcortexupdatestheexpectedresponse
associatedtoagivenstimulusanddrivestheerrorcomputationnetworkforthecomputationoftheemotionalerrorsignals.Theerrorsignalsaresenttothe
amygdalawhichupdatesthereactiveresponsesassociatedtothegivenstimulus.Theprocessesinvolvedinthediscrete-timeemotionalsystemarediscussin
detailinthefollowingsections.
10