Dynamics of Neural Networks with Continuous Attractors

0.15 MB·English

by  C. C. Alan Fung

Dynamics of Neural Networks with Continuous Attractors
C. C. Alan Fung1, K. Y. Michael Wong1, and Si Wu2
1Department of Physics, Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong, China
2Department of Informatics, University of Sussex, Brighton, United Kingdom
(Dated: February 3, 2008)
We investigate the dynamics of continuous attractor neural networks (CANNs). Due to the
translational invariance of their neuronal interactions, CANNs can hold a continuous family of
stationary states. We systematically explore how their neutral stability facilitates the tracking
performanceofaCANN,whichisbelievedtohavewideapplicationsinbrainfunctions. Wedevelop
8
aperturbativeapproach thatutilizes thedominating movementof thenetwork stationary states in
0
the state space. We obtain results on the maximum speed for a moving stimulus to be trackable,
0
and thereaction time to catch up an abrupt change in stimulus.
2

n PACSnumbers: 87.10.-e,05.45.-a
a
J

9 Understanding how the dynamics of a neural network dynamics,thatpermitssystematicperturbativeimprove-
2 is shaped by the network structure, and consequently ment. We display clearly how the dynamics of a CANN
facilitates the functions implemented by the neural sys- is decomposed into different modes, and which of them
]
n tem, is at the core of using physical models to elucidate dominates the tracking behaviors of the network. The
n brain functions [1]. Traditional models of attractor neu- solution is reminiscent of those of solitons in nonlinear
- ral networks (ANNs), such as those based on the Hop- dynamics, and by using the eigenfunctions of the quan-
s
i field model [2], are powerful for elucidating the compu- tum harmonic oscillator as the basis, a time-dependent
d tation andthe memory retrieving processesin the brain. perturbation analysis can be applied systematically. We
.
t When external noisy inputs are presented, the dynamics expect our method provides a general tool for the the-
a
of ANNs will reachan attractorthat is highly correlated oretical study of CANNs. Our work also generates new
m
withthe memoriesstoredinthenetwork. Eachattractor predictionsonthetrackingbehaviorsofCANNs,namely,
-
d has its own basin of attraction well separated from the the maximum tracking speed to moving stimuli, and the
n others. These models successfully describe the behavior reactiontimetosuddenchangesinexternalstimuli,both
o of associative memories, and stimulated an upsurge of are testable by experiments.
c studies on the retrievalbehaviorof the models [3]. How- Specifically, we consider a one-dimensional continuous
[
ever,otherthanmemoryretrieval,manyothercomputa- stimulus being encoded by an ensemble of neurons. For

1 tional issues of the brain have not been analyzed to the example, the stimulus may represent the moving direc-
v same extent. tion,theorientation,orageneralcontinuousfeatureofan
1 Recently,anewtypeofattractornetworks,calledcon- externalobject. LetU(x,t)bethesynapticinputattime
6
tinuousattractorneuralnetworks(CANNs),hasreceived t to the neurons with preferred stimulus of real-valued
4
considerable attention [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, x. We will consider stimuli and responses with correla-
4
. 15]. These networkspossess a translationalinvarianceof tion length a much less than the range of x, so that the
1 the neuronal interactions. As a result, they can hold a rangecanbe effectivelytakentobe ( , ). Thefiring
0 −∞ ∞
family of stationary states which can be translated into rate r(x,t) of these neurons increases with the synaptic
8
0 each other without the need to overcome any barriers. input, but saturates in the presence of a global activity-
: Thus, in the continuum limit, they form a continuous dependent inhibition. A solvable model that captures
v manifold in which the system is neutrally stable, and these features is given by
i
X the network state can translate easily when the external
U(x,t)2
r stimulus changescontinuously. Beyondpure memory re- r(x,t)= , (1)
a trieval, this endows the neural system with a tracking 1+kρ dx′U(x′,t)2
capability.
where ρ is the neural densitRy, and k is a small posi-
However, existing literature mainly focused on using
tiveconstantcontrollingthestrengthofglobalinhibition.
CANNs to illustrate pattern formation, movement con-
ThedynamicsofthesynapticinputU(x,t)isdetermined
trol, orientation tuning, spatial navigation, head direc-
by the external input Iext(x,t), the network input from
tion, population decoding and working memory [5, 6, 8,
other neurons, and its own relaxation. It is given by
11, 16]. Despite these initial successes, the conclusions
are often based on simulation results, and more rigorous dU(x,t)
′ ′ ′
analyses of the tracking behavior are still lacking. The τ =Iext(x,t)+ρ dxJ(x,x)r(x ,t) U(x,t),
dt −
relation between the tracking behavior and the energy Z (2)
landscape was not well understood. where τ is the time constant, which is typically of the
′
In this Letter we propose, as far as we know, the first order 1 ms, and J(x,x) is the neural interaction from
′
analytical solution of a model of CANNs to its tracking x to x. The key characteristic of CANNs is the transla-
2
2 The eigenfunctionsofF correspondto the variousdis-
1.5 tortion modes of the bump, the first two of which are
U(x) 1 particularly important. (1) The eigenfunction for the
0.5 eigenvalue λ0 is v0(xz), and represents a distortion of
|
0 -2 0 2 the amplitude of the bump. As we shall see, amplitude
x
changesof the bump affect its trackingperformance. (2)
Central to the tracking capability of CANNs, the eigen-
function for the eigenvalue 1 is v1(xz) and is neutrally
|
stable. We note thatv1(xz) ∂v0(xz)/∂z,correspond-
FIG.1: Thecanyonformedbythestationarystatesprojected | ∝ |
ing to the shift of the bump position among the station-
ontothesubspace formed byb10 and b0 0. Motion along the
| | ary states. This neutral stability is the consequence of
canyon corresponds tothe displacement of the bump(inset).
the translational invariance of the network. It implies
that when there are external inputs, however small, the
bump will move continuously. This is a unique prop-
tionalinvarianceoftheirneuralinteractions. Inoursolv-
erty associated with the special structure of a CANN,
ablemodel,wechooseGaussianinteractionswitharange
a, namely, J(x,x′) = exp[ (x x′)2/(2a2)]J/√2πa2. not shared by other attractor models. Other eigenfunc-
− − tionscorrespondtodistortionsoftheshapeofthebump.
CANN models with other neural interactions and inhi-
For example, the eigenfunction for the eigenvalue 1/4 is
bitionmechanisms havebeenstudy [4, 5,6, 8, 10]. How-
ever, our model has the advantage of permitting a sys- 6/7v3(xz)+ 1/7v1(xz),andcorrespondstoaskewed
| |
distortion of the bump.
tematic perturbative improvement. Nevertheless, the fi-
p p
nal conclusions of our model are qualitatively applicable It is instructive to consider the energy landscape in
to general cases. the state space of a CANN. Since Fmn is not symmet-
ric, a Lyapunov function cannot be derived for Eq. (4).
We first consider the intrinsic dynamics of the CANN
Nevertheless,for eachpeak position z, one can define an
model in the absence of external stimuli. For 0 < k <
kc ρJ2/(8√2πa), the network holds a continuous fam- effectiveenergyfunctionE|z = n(1−λn)bn|2z/2,where
ily≡of stationary states, which are bn|z is the overlap between U(xP)−U˜(x|z) and the nth
eigenfunction of F centered at z. Then the dynamics in
(x z)2 Eq.(4)canbelocallydescribedbythegradientdescentof
U˜(x|z)=U0exp − 4−a2 , (3) E z in the space of bn z. Since the set of points bn z =0
(cid:20) (cid:21) fo|rn=1tracesoutal|inewithE =0inthestate|space
z
6 |
whereU0 =[1+(1 k/kc)1/2]J/(4√πak). Thesestation- whenzvaries,onecanenvisageacanyonsurroundingthe
− line andfacilitating the localgradientdescent dynamics,
arystatesaretranslationallyinvariantamongthemselves
as shown in Fig. 1. A small force along the tangent of
andhavetheGaussianbumpedshapepeakedatarbitrary
thecanyoncanmovethenetworkstateeasily. Thisillus-
positions z.
trates how the landscape of the state space of a CANN
The stability of the Gaussian bumps can be studied
is shaped by the network structure, leading to the neu-
by considering the dynamics of fluctuations. Consider
the network state U(x,t) = U˜(xz)+δU(x,t). Then we tralstabilityofthesystem,andhowthisneutralstability
| shapes the network dynamics.
obtain
Next, we consider the network dynamics in the pres-
d ′ ′ ′ ence of a weak external stimulus. Suppose the neural
τ δU(x,t)= dxF(x,x)δU(x,t) δU(x,t), (4)
dt − response at time t is peaked at z(t). Since the dynam-
Z
ics is primarily dominated by the translational motion
′
where the interaction kernel is given by F(x,x) = of the bump, with secondary distortions in shape, we
′′ ′′ ′′ ′
ρ dx J(x,x )∂r(x )/∂U(x). To compute the eigen- may develop a time-dependent perturbation analysis us-
′
functions and eigenvalues of the kernel F(x,x), we ing v (xz(t)) as the basis, andconsider perturbations
n
R { | }
choose the wave functions of the quantum har- in increasing orders of n. This is done by considering
monic oscillators as the basis, namely, v (xz) = solutions of the form
n
|
exp( ξ2/2)H (ξ)/ (2π)1/2an!2n, where ξ (x
n ∞
a)/(√−2a) and Hn(pξ) is the nth order Hermite≡polyno−- U(x,t)=U˜(xz(t))+ an(t)vn(xz(t)). (5)
mial function. Indeed, the ground state of the quantum | |
n=0
harmonic oscillator corresponds to the Gaussian bump, X
andthefirst,second,andthirdexcitedstatescorrespond Furthermore, since the Gaussian bump is the steady-
tofluctuationsinthepeakposition,width,andskewness statesolutionofthedynamicalequationintheabsenceof
of the bump respectively. The eigenvalues of the matrix externalstimuli,theneuronalinteractionterminEq.(2)
Fmnaregivenbyλ0 1 (1 k/kc)1/2,andλn =1/2n−1 can be linearized for weak stimuli. Making use of the
≡ − −
for n 1. Since λ1 =1 andall other eigenvaluesare less orthonormalityandcompleteness of vn(xz(t)) , we ob-
≥ { | }
than 1, the stationary state is neutrally stable in one tainfromEq.(2) expressionsforda /dt foreachordern
n
component, and stable in all other components. of perturbation, each being driven by I (t), the overlap
n
3
between Iext(x,t) and the nth eigenfunction. Determin- recurrent interactions, which tends to reduce the lag.
ing z(t) by the center of mass of U(x,t), we obtain the Tracking is maintained when these two factors match
self-consistent condition each other, i.e., v =g(s); otherwise, s diverges.
Fig.2(a) shows that the function g(s) is concave, and
∞
dz = 2a I1+ n=3,odd n!!/(n−1)!!(In−an) . has the maximum value of gmax =2αa/(τ√e) at s=2a.
dt τ U0 (2πP)1/2a+ p∞n=0,even (n−1)!!/n!!a(n6!) Ttrhaicskmtheaensstitmhautluisf.vT>hugsm,axg,matxhednefietnweosrtkhiesmunaaxbimleutmo
p P p trackable speed of a moving stimulus. Notably, g in-
Inpractice,loworderperturbationsalreadyyieldveryac- max
creases with the strength of the external signal and the
curateresults. Below,weconsiderthenetworkdynamics
range of neuronal recurrent interactions. This is reason-
for several kinds of external stimuli.
able since it is the neuronal interactions that induce the
1) Tracking a stationary stimulus: Consider the ex-
ternal stimulus consisting of a Gaussian bump, namely, movement of the bump. gmax decreases with the time
Iext(x,t)=αU0exp[ (x z0)2/4a2]. Perturbationup to constant of the network, as this reflects the responsive-
− − ness of the network to external inputs.
the order n=1 yields a1(t)=0, [d/dt+(1 λ0)/τ]a0 =
αU0 (2π)1/2aexp[ (z0 z)2/8a2]/τ, and− On the other hand, for v <gmax, there is a stable and
− − unstablefixedpoint,respectivelydenotedbys1 ands2 in
p dz α (z0 z)2 −1 Fig.2(a). Whentheinitialdistanceislessthans2,itwill
dt = τ(z0−z)exp − 8−a2 R(t) , (7) converge to s1. Otherwise, the tracking of the stimulus
(cid:20) (cid:21) will be lost. Figs. 2(b) and (c) show that the analytical
results of Eq. (8) well agree with the simulation results.
t ′ ′
where R(t)=1+α −∞(dt/τ)exp[−(1−λ0)(t−t)/τ− 3) Trackingan abrupt changeof thestimulus: Suppose
(z0 z(t′))2/8a2]. Hence, the dynamics is driven by a the network has reached a steady state with an external
− R
pull of the bump position towards the stimulus posi- stimulus stationary at t < 0, and the stimulus position
tion z0. When compared with the limiting expression jumps from 0 to z0 suddenly at t = 0. This is a typical
for weak stimuli, which was considered in the dynamics scenarioin experiments studying mental rotationbehav-
of the moving bump in [15], the pull is reduced by the iors. We first consider the case that the jump size z0 is
factor R(t). This is due to the increase in amplitude of smallcomparedwiththerangeaofneuronalinteractions.
the bump, which slows down its response. In the limit of weak stimulus, the dynamics is described
Suppose, in addition to the bumped signal, there is byEq.(7)withR(t)=1. Weareinterestedinestimating
alsotheI1 componentconsistingofawhite noiseoftem- thereactiontimeT,whichisthetimetakenbythebump
perature Tn. Then the noises shift the bump position to move to a small distance θ from the stimulus. In the
randomly. Intheabsenceofthebumpedsignalintheex- limitoflownoise,thereactiontimeincreaseslogarithmi-
ternalstimulus(α=0),thepeakpositionofthebumped cally with the jump size, namely, T (τ/α)ln(z0 /θ).
response undergoes a random walk with z(t)2 = 2Dt Whenthestrengthαoftheextern≈alstimulus|isl|arger,
in the low noise limit, where D 4aTn/[(2hπ)1/2iU02τ2] is improvement using a perturbation analysis up to n = 1
≡
thediffusionconstant. Inthepresenceofaweakbumped is requiredwhen the jump size z0 is large. This amounts
signal (α > 0), the peak of the bumped response expe- to taking into account the change of the bump height
riences a pull which can be written as the gradient of a during its movement from the old to new position. The
potential V(z) = 4a2αexp[ (z z0)2/8a2]/τ. Thus, result is identical to Eq. (7), with R(t) replaced by
− − −
the probabilitydistributionofz canbe approximatedby
a Boltzmann distribution P(z) exp[ V(z)/D] in the α (1 λ0) t dt′
low noise limit. ∝ − R(t)=1+ exp − t +α
1 λ0 − τ 0 τ
2) Tracking a moving stimulus: The tracking perfor- − (cid:20) (cid:21) Z
mance of a CANN is a key property that is believed to exp (1−λ0)(t t′) (z0−z(t′))2 . (9)
have wide applications in neural systems. Suppose the × − τ − − 8a2
(cid:20) (cid:21)
stimulusismovingataconstantvelocityv,andthenoise
Indeed, R(t) represents the change in height during the
is negligible. The dynamical equation becomes identical
movement of the bump. Contributions from the second
to Eq. (7), with z0 replaced by vt. Denoting the lag of
and third terms show that it is highest at the initial and
the bump behind the stimulus by s = z0 z we have,
− final positions respectively, and lowest at some point in
after the transients,
between,agreeingwithsimulationresultsshowninFig.3
ds αse−s2/8a2 αe−s2/8a2 −1 inset. Fig.3showsthatthefirstorderperturbationover-
=v g(s); g(s) v 1+ . comes the insufficiency of the logarithmic estimate, and
dt − ≡ − τ " 1 λ0 # has an excellent agreement with simulation results for
−
(8) z0 up to the order of 2a. We also compute the reaction
The value of s is determined by two competing factors: time up to the fifth order perturbation, and the agree-
the first term represents the movement of the stimu- ment with simulations remains excellent even when z0
lus, which tends to enlarge the separation, and the sec- goes beyond 2a. This implies that beyond the range of
ond termrepresents the collective effects ofthe neuronal neuronalinteraction,trackingisinfluencedbythedistor-
4
0.04 4 1
(a) (b) (c)
0.03vmax 3 0.8
v
0.6
g(s)0.02 s2 s
0.4
0.01 1
0.2
00 0.5s1 1 1.s52 2 2.5 3 00 50 100 150 200 250 00 0.01 0.02 0.0v3max 0.04
s t v
FIG. 2: (a) The function g(s) and the stable and unstable fixed points of Eq. (8). (b) The time dependence of the separation
s starting from different initial values. Symbols: simulations with N =200. Lines: fifth order perturbation. Dashed lines: s1
(bottom) and s2 (top). (c) The dependence of the terminal separation s on the stimulus speed v. Symbols: simulations with
N =200. Solid line: prediction of [15]. Dashed line: first order perturbation. Parameters: α=0.05, a=0.5, τ =1, k =0.5,
ρ=N/(2π), J =√2πa2.
400 logarithmic for small changes and increasing rapidly be-
2
yond the neuronal range. These two properties are as-
1.5
300 U(x) 1 saoreciatteesdtabwliethinthperauctniicqeueanddyncaanmiscesrvoef aasCgAenNeNra.l Tclhueeys
0.5
T200 in for checking the existence of a CANN in neural sys-
0
-2 x0 2 tems. In order to solve the dynamics which is otherwise
100 extremely complicated for a large recurrent network, we
have developed a perturbative analysis to simplify the
0 dynamics of a CANN. Geometrically, it is equivalent to
0 0.5 1 1.5 2 2.5 3
z0 projecting the network state on its dominant directions
of the state space. This method works efficiently and
FIG. 3: The dependence of the reaction time T on the new
may be widely used in the study of CANNs. Further-
stimulus position z0. Symbols: simulations. Solid line: pre-
diction of [15]. Dotted line: first order perturbation. Dashed more, the special structure of a CANN may have other
line: fifth order perturbation. Parameters: as in Fig. 2. In- applications in brain functions, for instance, the highly
set: profiles of the bump between the old and new positions structured state space of a CANN may provide a neural
at z0 =π/2 in thesimulation. basis for encoding the topologicalrelationship of objects
in a feature space, as suggested by recent psychophys-
tion of the width and skewed shape of the bump. ical experiments [17, 18]. We expect the mathematical
To conclude, we have systematically investigated how framework developed in this study will be very valuable
the neutral stability of a CANN facilitates the tracking to explore these issues.
performance of the network, a capability which is be-
lieved to have wide applications in brain functions. Two ThisworkispartiallysupportedbytheResearchGrant
interestingbehaviorsareobserved,namely,themaximum Council of Hong Kong (Grant No. HKUST 603606
trackable speed for a moving stimulus and the reaction and HKUST 603607),BBSRC (BB/E017436/1)and the
time for catching up an abrupt change of a stimulus, Royal Society.
[1] P. Dayan and L. Abbott, Theoretical Neuroscience: 66, 041902 (2002).
Computational and Mathematical Modelling of Neural [10] A.Renart,P.SongandX.Wang,Neuron38,473(2003).
Systems,(MIT Press, Cambridge MA, 2001). [11] S. Wu and S. Amari, Neural Computation 17, 2215
[2] J.Hopfield,Proc.Natl.Acad.Sci.USA,792554(1982). (2005).
[3] E.Domany,J.vanHemmenandK.Schulteneds.,Models [12] S.CoombesandM.R.Owen,Phys.Rev.Lett.94,148102
of Neural Networks III,(Springer, Berlin, 1995). (2005).
[4] S.Amari, Biological Cybernetics 27, 77 (1977). [13] B. Blumenfeld, S. Preminger, D. Sagi and M. Tsodyks,
[5] R.Ben-Yishai,R.LevBar-OrandH.Sompolinsky,Proc. Neuron 52, 383 (2006).
Natl. Acad.Sci. USA,92 3844-3848. (1995). [14] S. Coombes, Scholarpedia 1, 1373 (2006).
[6] K.-C. Zhang, J. Neurosicence 16, 2112 (1996). [15] S. Wu, K. Hamaguchi and S. Amari, Neural Computa-
[7] B. Ermentrout, Reports on Progress in Physics 61, 353 tion, in press (2008).
(1998). [16] A.SamsonovichandB.L.McNaughton,J.Neurosci.17,
[8] S. Deneve, P. Latham and A. Pouget, Nature Neuro- 5900 (1997).
science, 2, 740-745 (1999). [17] J. Jastorff, Z. Kourtzi and M. Giese, J. Vision 6, 791
[9] X. Xie, R. H. R. Hahnloser and S. Seung, Phys. Rev. E (2006).
5
[18] A. B. A. Graf, F. A. Wichmann, H. H. Bu¨lthoff, and B.
Sch¨olkopf, NeuralComputation 18, 143 (2006).