On the Equivalence of Restricted Boltzmann Machines and Tensor Network States

1.4 MB·

by  Jing Chen

On the Equivalence of Restricted Boltzmann Machines and Tensor Network States
Jing Chen,1 Song Cheng,1 Haidong Xie,1 Lei Wang,1,∗ and Tao Xiang1,2,†
1Institute of Physics, Chinese Academy of Sciences, P.O. Box 603, Beijing 100190, China
2Collaborative Innovation Center of Quantum Matter, Beijing 100190, China
RestrictedBoltzmannmachine(RBM)isoneofthefundamentalbuildingblocksofdeeplearning.
RBMfindswideapplicationsindimensionalreduction,featureextraction,andrecommendersystems
viamodelingtheprobabilitydistributionsofavarietyofinputdataincludingnaturalimages,speech
signals,andcustomerratings,etc. WebuildabridgebetweenRBMandtensornetworkstates(TNS)
widely used in quantum many-body physics research. We devise efficient algorithms to translate
an RBM into the commonly used TNS. Conversely, we give sufficient and necessary conditions to
determinewhetheraTNScanbetransformedintoanRBMofgivenarchitectures. Revealingthese
7 generalandconstructiveconnectionscancross-fertilizebothdeeplearningandquantum-manybody
1 physics. Notably,byexploitingtheentanglemententropyboundofTNS,wecanrigorouslyquantify
0 theexpressivepowerofRBMoncomplexdatasets. InsightsintoTNSanditsentanglementcapacity
2 can guide the design of more powerful deep learning architectures. On the other hand, RBM can
n represent quantum many-body states with fewer parameters compared to TNS, which may allow
a more efficient classical simulations.
J

7
I. INTRODUCTION (a)
1
v v v v v v
1 2 3 4 5 6
] Deep learning is transforming the world with its far-
l
e reachingapplicationsincomputervision,speechrecogni-
-
tion,naturallanguageprocessing,recommendersystems,
r
st etc [1, 2]. At the core of many of these applications are h1 h2 h3 h4
t. the artificial neural networks which can be used to rec- (b)
a ognize or even discover interesting patterns in the input
m data [3–5]. In a nutshell, the neural nets act as train-
- ablefunctionalmappingsofmultiplevariables. Todesign
d A(1) A(2) A(3) A(4) A(5) A(6)
evenmorepowerfulandintelligentmachinesrequiresone
n
toquantifyandextendtheexpressivepoweroftheneural
o
nets. However, there is a gap between the mathemati-
c
(c)
[ cal foundation and the real-world applications which are

largelydrivenbytheengineeringpractices[2], becauseit
1 SecII
has long been a difficult endeavor to rigorously quantify RBM TNS
v
1 the neural nets over complicated datasets. SecIII
3 Insights into the physical rules governing the neural
8 networks may offer an answer to the great success of
4 deep learning and guide its more fruitful development Figure1. (a)TheRBMfunctiondefinedinEq.(2). Theblue
0 inthefuture. Forexample,statisticalphysicshasalong- dotsrepresentvisibleunitsv andthemagentadotsrepresent
. thehiddenunitsh. Theyarecoupledthroughtheconnections
1 standingimpactonmachinelearning[4,6],becauseboth
indicated by the solid lines. (b) The MPS defined in Eq. (3).
0 fields concern about collective behavior emerged from a
7 large amount of microscopic degree of freedoms. More- Each red dot denotes a three index tensor A(i). Throughout
1 this paper, we use flat dots to represent units of an RBM,
over, as suggested by Mehta and Schwab [7], there is
: andusestereoscopicdotstorepresenttensors. (c)Ourpaper
v a connection between the deep learning and the renor-
buildsexplicitconnectionsbetweenRBMandTNS.SeeSec.II
Xi malization group. Lin and Tegmark [8] also argued and Sec. III for detailed discussions on the mapping between
that the “unreasonable success” of deep learning can them.
r
a be traced back to the law of physics, which often im-
poses symmetry, locality, compositionality, polynomial
understood as an energy-based model. As illustrated in
log-probability, and other properties on the input data.
Fig.1(a),itconsistsofasetofinterconnectedvisibleand
RBMisavividexampleoftheintrinsicconnectionbe-
hidden binary variables. These variables are assumed to
tweenstatisticalphysicsandmachinelearning. AnRBM
satisfy the Boltzmann distribution whose energy func-
is a special type of neural networks which can be better
tional is defined by
(cid:88) (cid:88) (cid:88)
E(v,h)=− a v − b h − v W h , (1)
i i j j i ij j
i j i,j
∗ [email protected]
† [email protected] where v ={v } and h={h } are the visible and hidden
i j
2
binary variables. We denote their number as n and n In fact, the quest for more expressive wavefunction
v h
respectively. Parameters a ,b are the biases applied to is actually central to the quantum many-body physics
i j
the visible and hidden units, respectively. W is the research. Ideally, those ansatz should accurately de-
ij
coupling matrix between these two units. scribe exponentially large degree of freedoms with poly-
By integrating out the hidden units, the RBM repre- nomial resources. Tensor network states (TNS) [22] are
sents the marginal distribution of the visible variables promising candidates to meet this demand. Fig. 1(b)
(omitting an irrelevant normalization factor) showsoneofthesimplestTNS,thematrixproductstate
(MPS) [23], as an example. The MPS parametrizes a
(cid:88)
ΨRBM(v)= e−E(v,h) wavefunction of nv physical variables as,
=(cid:89)h eaivi(cid:16)1+ebj+(cid:80)iviWij(cid:17), (2) ΨMPS(v)=Tr(cid:32)(cid:89)nv A(i)[vi](cid:33), (3)
i,j i=1
In the RBM there is no direct connection between the where each A(i) is a 3-index tensor indicated by a red
visibleunits. Howeverthehiddenunitsgenerateeffective dot in Fig. 1(b). For fixed physical variables on the ver-
connections or interactions among them. By increasing tical lines, each A(i)[v ] is a matrix. We denote their
i
the number of hidden units and connections, the RBM matrix sizes as the virtual bond dimensions indicated by
can in principle parameterize more complex functions of the thickness of the horizontal bonds in Fig. 1(b). Con-
the visible units [9–11]. necting these horizontal bonds means tensor contraction
One can train an RBM by specifying its parameters ofthevirtualdegreeoffreedoms. Byincreasingthebond
such that the probability distribution of the visible units dimension of the virtual bonds, MPS could represent in-
reproduces that of the input data [12, 13]. The hidden creasingly complex multivariable functions [23]. MPS
units of a trained RBM may also reveal correlations in representation is equivalent to the tensor train decom-
thedatawithappealingphysicalmeanings. Forexample, positions in the applied math community [24]. Similarly,
in an RBM trained with a dataset of images containing onecanconnecthigherordertensorsinatwodimensional
handwritten digits, the connection weight contains the network to represent the physical degree of freedoms lies
information about the pen stokes [14]. These learning in two dimensions more naturally. This construction is
features can be used either for discriminative tasks, such namedprojectedentangledpairstate(PEPS)[25]. Inthe
as pattern recognition, or for generative tasks, such as past decades, quantum physicists have developed solid
generating more samples according to the learned distri- theoretical understanding and efficient numerical tech-
bution. RBM has played an important role in the recent niques for TNS. See [22, 26] for pedagogical reviews on
renaissance of deep learning [15, 16], because of its ver- TNS.
satile abilities in feature extraction and dimensionality The physics community also has an answer for TNS’s
reduction of complex data sets. “unreasonable effectiveness”: the entanglement area
Recently, RBM has attracted great attention in the law [27]. Entanglement entropy [28] characterizes the
field of quantum many-body physics. For example, Car- information content of a bipartition of the system. And
leoandTroyer[17]proposedanRBMinspiredvariational the entanglement area law means that the entanglement
wavefunction to study quantum many-body systems at entropy increases only proportional to the boundary be-
or away from equilibrium. Deng et al. [18] constructed tweenthetwosubsystems. Althoughthereareextremely
exact RBM representations for several interesting topo- large number of possible quantum states, many physi-
logical states. Torlai and Melko [19] trained an RBM cal states of practical interests fulfill the entanglement
to reproduce the thermodynamics of a statistical physics area law [27]. TNS are designed to efficiently represent
model. Huang and Wang [20] used RBM as a recom- these quantum states with relatively low entanglement
mender system to accelerate Monte Carlo simulation of entropy and have achieved remarkable successes in the
quantum many-body systems. Liu et al. [21] reported past decades [29].
similar ideas using classical spin models instead of the The success of TNS for quantum many-body systems
RBM. makes one wonder whether is there anything similar to
These developments have raised several critical ques- quantify the expressive power of deep learning architec-
tions about the expressive power of neural nets. Is RBM tures. As Refs. [2, 8] suggest, we only care about a tiny
more expressive than the standard variational wavefunc- fractionoftheinputdataamonginfinitenumberofpossi-
tions of quantum states [17]? Can RBM efficiently de- bleinputsinthepracticalmachinelearningapplications.
scribe the probability distribution of physical models at This hints for certain guiding principle similar to the en-
criticality [19, 20]? Unfortunately, the existing univer- tanglement area law to quantify the required resources.
sal approximation theorem [9–11] established in the ma- In this paper, we present a general and constructive
chine learning community is not particularly instructive connectionbetweenRBMandTNS.Withthiscorrespon-
because it involves exponentially large resources, and it dence,manyconceptsandtechniquesfromdeeplearning
cannotbeusedasaguidingprincipletosolvephysicalor andquantumphysicscanbeexchanged. Bytransforming
industrial problems. an RBM to a TNS and exploiting its entanglement en-
3
tropybound, onecanquantifytheexpressibilityofRBM (a)
(cid:47)(1) (cid:47)(2) (cid:47)(3) (cid:47)(4) (cid:47)(5) (cid:47)(6)
for quantum states, for statistical physics models, and v v v v v v
for industrial datasets. On the other hand, we present
sufficientandnecessaryconditionsonTNSsothatitcan
be transformed into an RBM of a given structure. RBM
canserveasaveryefficientclassicalrepresentationofthe (cid:47)(1) (cid:47)(2) (cid:47)(3) (cid:47)(4)
h h h h
quantum states.
Theorganizationofthispaperisasfollows: InSec.II, (b)
we present algorithms to transform the RBM into TNS
such as MPS and PEPS and discuss their consequences.
In Sec. III, we discuss the sufficient and necessary condi-
tionsofaTNStohaveanRBMrepresentationsofagiven
architecture. InSec.IV,weillustratetheRBM-TNScon-
nection by showing the four ground states of the toric
(c)
code model in the RBM form and in the corresponding
TNSform. InSec.V,wediscusstheredundancyofRBM D=2 D=8 D=8 D=4 D=2
parametrizationandillustratedhowtosimplifythemus-
A(1) A(2) A(3) A(4) A(5) A(6)
ing TNS methods. In Sec. VI, we discuss implications of
our results both for machine learning and for quantum
many-body physics. In Appendix A we show the equiva- Figure2. TransformanRBMtoanMPS.(a)ATNSwiththe
lenceoftheshift-invariantRBMemployedinRef.[17]to same network structure as the RBM in Fig. 1(b). The blue
an MPS and discuss their entanglement entropy bounds. dots represent the diagonal tensors Λ(i) at the visible units
v
In Appendix B, we give a sufficient condition to find the and the magenta dots represent the Λ(j) at the hidden units,
h
RBMrepresentationofsomespecificTNSforthestatisti- seeEqs.(5,6). ThebondsdenotethematrixM(ij) definedin
calphysicsmodelsandsimplequantumstates. Finally,in Eq. (4). (b) The TNS is cut into n pieces. The long range
v
Appendix C we discuss the general equivalence between connection is mediated by a cross gate (red square) attached
more general Boltzmann machines [30, 31] and TNS. to the vertical connection. (c) The MPS representation. The
3 index tensor A(i)(red dot) is determined by contraction of
theinternalindicesofeachpiecein(b). Thebonddimension
II. TNS REPRESENTATION OF RBM: THE DoftheMPSisdeterminedbythenumberofcutconnections
ROUTE TO MPS AND PEPS of the RBM.
In this section we show the general equivalence of an
to view it as a function approximator such as a feedfor-
RBM and an MPS via a constructive approach. Namely,
ward neural net [20, 33]. Thirdly, although the standard
giventheRBMarchitectureandparameters{W ,a ,b },
ij i j RBMhasdenseall-to-allconnectionsbetweenthevisible
wepresentalgorithmstoobtainthetensorsexplicitly. We
and the hidden units, for generality and clarity we illus-
firstpresentasimpleandintuitiveapproach,thenamore
trated the transformation using the RBM with sparser
sophisticated approach which provides a tighter bound
connections shown in Fig. 1(a). Our results are never-
on the tensor bond dimension. For both algorithms we
theless general and apply to the RBM with dense inter
provide software implementations in the supplementary
connections between the visible and the hidden units.
materials [32].
To start with, we rewrite the RBM in terms of a TNS
Before proceeding, we first clarify a few notations
with external (visible) and internal (hidden) degree of
about the RBM. Firstly, in the standard machine learn-
freedomswhilekeepingthenetworkstructureunchanged,
ing applications and in [19, 20], one tunes the real pa- seeFig.2(a). WedefineadiagonaltensorΛ(i)andΛ(j)on
rameters{W ,a ,b }tolettheRBM[Eq.(2)]modelthe v h
ij i j each visible and hidden unit indicated by the dots. And
probability distributions of the input data. While refer-
weintroducea2×2matrixM(ij) oneachconnectionline
ences[17,18]interpretEq.(2)astheamplitudeofquan-
between the visible unit v and the hidden unit h . The
tum mechanical wavefunctions. Therefore, these authors i j
tensor elements are related to the weights and biases of
generalized the weights and biases to complex numbers.
the RBM
In this paper, we consider complex RBM parameters for
(cid:18) (cid:19)
generality.1 Secondly,conventionallyoneviewstheRBM 1 1
M(ij) = , (4)
as an energy based generative model. While for our dis- 1 eWij
cussion about the expressibility of Eq. (2) it is sufficient
Λ(i) =diag(1,eai), (5)
v
Λ(j) =diag(cid:0)1,ebj(cid:1), (6)
h
1 Thereferences[17,18]definesRBMwithunitswith±1instead where diag(·,·) denotes a diagonal tensor with nonzero
of binary values. This amounts to a simple rescaling and offset elements given in the bracket.
oftheweightsandbiases. To translate the TNS in Fig. 2(a) into an MPS, we
4
(a) (cid:47)(v3) (cid:47)(v4) (cid:47)(v3) (cid:47)(v4) (a) A B
B B
1 2
Q
M(41) =
(cid:47)(1) M(32) (cid:47)(1) P M(32) v1 v2 v3 v4 v5 v6
h h
(cid:47)(2) (cid:47)(2)
h h
(b) (b) A B
i A2 A1
l
= = (cid:71) (cid:71)
M(41) P Q k j ij kl
v v v v v v
1 2 3 4 5 6
Figure 3. (a) The cross gate indicated by the red square
mediatesthelongrangeconnectionsbetweenv andh . The
4 1 (c)
greenconnectiondenotesthematrixM(41) definedinEq.(4).
D=4
ThecyanconnectionsdenotematricesP,Qwhichdoublethe
virtualbonddimensions. (b)Thecrossgatesplitsthematrix A(1) A(2) A(3) A(4) A(5) A(6)
M(41) into the product of two arbitrary matrices P and Q.
Figure 4. Estimating a tighter entanglement entropy bound.
(a)WeshowonlythevisibleunitsoftheRBMandtheconnec-
next divide it into n pieces as shown in Fig. 2(b). Con- tions mediated by the hidden units as the arcs. A division of
v
tracting all the internal indices within each piece is then thesystemintoAandB =B1∪B2,whereB1 istheinterface
equivalent to summing over the hidden units. Finally, region can be reached from A. (b) A division of the system
we merge the external connections between pieces into into A = A1 ∪A2 and B, where A1 is the interface region
can be reached fromB. (c) Theresulting MPS hasa smaller
the virtual bonds of the MPS shown in Fig. 2(c). The
bond dimension at the bipartition compared to Fig. 2(c).
thicknessofthevirtualbondsindicatesthecorresponding
bond dimension of the MPS.
From the step Fig. 2(b) to (c), we need to pay spe-
and define the reduced density matrix
cial attention to the “long-range” connections indicated
bythecyanlinewhichcrossestheinbetweenpieces. We ρ= (cid:88) Ψ∗ ({v(cid:48) ,v }) Ψ ({v ,v }).
RBM i∈A i∈B RBM i∈A i∈B
introduce a “cross gate” denoted by the red square to
{vi∈B}
break this long range connection into segments. As ex-
(7)
plained in Fig. 3, the cross gate splits the matrix M(41)
The von Neumann entanglement entropy is defined as
intotheproductoftwoarbitrarymatricesM(41) =P·Q.
S = −Tr(ρlnρ) [28]. The entanglement entropy charac-
Afterwards,thematricesP andQareabsorbedintoA(2)
terizestheinformationcontentofthedistributionΨ .
RBM
and A(3) respectively. The “long-range” connection dou-
The entanglement entropy can be viewed as a proxy for
bles the bond dimension of A(3) to D = 8. In general,
thecorrelationsbetweenthetwopartitions. Uncorrelated
longrangeconnectionswilldoublethebonddimensionof
data has zero entanglement entropy at any bipartition.
all tensors they pass by. Following the scheme of Fig. 2
While long range correlations typically increases the en-
onecanobtainanMPSrepresentationofarbitraryRBM.
tanglement. We conjecture that the realistic dataset in
The MPS bond dimensions are given by the number of
most successful machine learning applications has rela-
connections n cut at the bipartition D =2n.
tively low entanglement entropies. Computationally, the
The mapping from the RBM to the MPS shown in entanglemententropyoftheMPSateachbipartitioncan
Fig. 2 is not unique because one can assign the hidden be obtained as a byproduct of the canonicalization. The
units into different subregions. The resulting MPS will maximal entanglement entropy is bounded by the MPS
have different bond dimensions even though they repre- bonddimensionlnD,inwhichcasealltheeigenvaluesof
sent the same function. One can bring these equivalent the reduced density matrix are 1/D [36].
MPSs into a unique canonical form [34, 35] by perform- The bond dimensions obtained by the simple and in-
ingtheSchmidtdecompositionforeachbond. SeeSec.V tuitive approach Fig. 2 is only an upper bound. For ex-
formorediscussionsabouttheredundanciesintheRBM ample, the bond dimension of the second bond D = 8
and TNS representations. in Fig. 2(c) is more than enough to capture the entan-
The explicit connection between the RBM and the glement since there are only two sites on the left. Next
MPSallowsonetoquantifytheRBM’sexpressibilityus- we present a more sophisticated approach to provide a
ing the entanglement entropy because it is convenient to tighter bound on the bond dimensions than Fig. 2(c).
readouttheentanglementinformationinTNSlanguage. This approach is also independent of the assignment of
Wedividethevisibleunitsintotwosetsv ={v ,v } the hidden units. Figure 4(a) shows an RBM with the
i∈A i∈B
5
hidden units been traced out, where the arcs indicate TNS representation. On the other hand, one should also
the hidden units mediated connection between the vis- note that although the TNS representation has a nom-
ible units. The bipartition of the RBM cut the visible inal larger number of parameters, they may still be ad-
units into two parts A and B =B ∪B , where B con- vantageous in practice for solving quantum many-body
1 2 1
tains the visible units can be reached directly from A, physics problems thanks to more efficient optimization
and B contains the remaining part. The RBM function methods [22].
2
Eq. (2) has a product form
To conform with the effective long range connections
Ψ (v)=ψ({v ,v })φ({v ,v }). (8) in Fig. 4, one can first arrange the visible units into a
RBM i∈A i∈B1 i∈B1 i∈B2 two dimensional lattice, then map it into a PEPS [25].
Once the visible units in B are given the function re- ThisisparticularlyrelevantiftheoriginalRBMarchitec-
1
duced to a product state of v and v . The tensor ture already accommodate for the dimensionality of the
i∈A i∈B2
rank[37,38]ofEq.(8)isdeterminedbythevisibledegree data, for example, pixels of images or physical degree of
of freedoms of B . freedoms reside in two dimension .
1
Alternatively, one can also consider a subdivision of
In general, suppose the interface region size m does
A = A ∪A depending on whether it can be reached
1 2 not scale with the system size. The width of the region
directly from the units in the part B. In the case of
Fig. 4(b), A = A and A = ∅. The MPS bond dimen- is in the order of md1. The entanglement entropy bound
1 2 scales as
sion at the A,B bipartition is determined by the smaller
one of the two interface regions m = min(|A |,|B |).
1 1
When mapping to an MPS, the physical state of A or
B is copied to the virtual bond. Compared to Fig1. 2, Smax ∼md1Ld−1, (11)
1
the approach of Fig. 4 always results into an MPS with
smaller bond dimensions since m≤n. For example, the
where d is the spatial dimension, L is the linear size of
connectionsoutoftheunitv arecountedindependently
4 the subregion. Thus, the maximum entanglement of a
in Fig. 2(a), but some of them end into the same visi-
sparsely connected RBM [18] still follows the area law,
ble units. As a consequence, the bond dimension of the
similar to the conventional TNS such as MPS or PEPS.
center bond in Fig. 4(c) is D = 4 instead of D = 8 in
While for densely connected RBM each visible unit can
Fig. 2(c).
reach all other visible units through the hidden units,
Ingeneral,iftherearemunitsintheinterfaceregionof
thus m = min(|A|,|B|) for any bipartition. The entan-
a bipartition of the RBM, the corresponding MPS bond
glement entropy bound can therefore reach the volume
dimension is
law scaling. The dense RBM provides a compact repre-
D =2m. (9) sentation of highly entangled quantum states. The num-
ber of parameters of the dense RBM scales polynomially
The MPS bond dimension of the Fig. 4(c) is D = with the system size. On the other hand, to describe the
2,4,4,4,2 respectively, which is smaller than Fig. 2(c). state with same entanglement one needs exponentially
The bond dimension determines the maximal entangle- large number of parameters in the MPS or PEPS, or
ment entropy which can be captured by the RBM at the needstousetheTNSwithhierarchicalstructures[39,40].
bipartition The RBM-TNS correspondence justifies the excellent
accuracy achieved in the variational study [17]. Ap-
Smax =mln2. (10) pendix A discusses about the TNS representation and
entanglement capacity of the shift-invariant RBM state
The entanglement entropy bound Eq. (10) only depends
employed in Ref. [17].
on the connectivity structure of the RBM. Employing
this central result one can quantify the expressive power In fact, the discovered RBM-TNS correspondence is
ofvariousRMBfordatasuchasthenaturalimagesorthe not limited by the “restricted” feature (i.e. no coupling
quantum many-body states [17, 18]. Note that Eq. (10) within the visible or the hidden units) of the RBM. In
is only an upper bound of the entanglement entropy, to Appendix C, we proved a general TNS can be always
obtain the precise value one needs to construct the MPS mapped to a Boltzmann machine [30]. Moreover, the
representation explicitly. We refer the interested readers mappingsshowninFig.2andFig.4applyaswelltothe
to the code implementation in the supplementary mate- deep Boltzmann machines [31] with multilayers of the
rials [32]. hiddenunits. ThevisibleunitsinadeepBoltzmannma-
It is also instructive to compare the number of chine can have longer ranged effective connections medi-
parameters of the RBM and the corresponding TNS ated by the deep hidden units, therefore achieve a larger
parametrization of the same function. Assuming each entanglement capacity compared to the RBM with the
visible unit has m effectively connections to the sur- samenumberofhiddenunitsandconnections. Thus,the
roundingvisibleunits,thecorrespondingtensorwillhave entanglement entropy bound Eq. (10) offers a new per-
2m parameters on its virtual bonds. Thus, in principle spectiveonthepowerfulexpressibilityofthedeepneural
the RBM can be exponentially more compact than the architectures.
6
(a) RBMofthegivenstructureinFig.1supposeEq.(12)has
no solution. In that case, one can increase the number
of variables by enlarging the hidden units or including
A(1) A(2) A(3) A(4) A(5) A(6) more connectivities in the RBM. The linear equations
willeventuallybecomeunderdeterminedwithmorecom-
= plex RBM.
If a valid solution of Eq. (12) exists, we next aim to
decompose each tensor T(j) into the components defined
in Eqs. (4,5,6). See Fig. 5(b) for an example,
T(2) = (cid:88) eh2b2+(cid:80)i∈{2,3,4}vi(Wi2h2+ai2), (13)
T(1) T(2) T(3) T(4) v2v3v4 h2∈{0,1}
where a are partial contributions to the biases of the
a a a ij
(b) 22 32 42 visible units. The overall biases sum up contributions
Q from all the connections a =(cid:80) a . For a 3-index ten-
W i j ij
T(2)= W22 32 W 42 = P I R stoor7Tv(a2)r,iatbhleerseuapretionatontoarlm23al=iza8tieoqnufaatciotonrsiwnitFhigr.es5p(ebc)t.
b 3
2 The number of variables grows linearly with the order
of T, while the number of equations grows exponentially
Figure5. Thesufficientandnecessaryconditionstotransform
instead. The Eq. (13) is in general over determined and
anMPStoanRBMwithagivenarchitecture. (a)TheRBM
the solution exists only in special cases.
factorizes into n isolated tensors T(1,2,3,4) for given visible
h Inpractice,theEq.(13)canbeconsideredasthetensor
variables, see Eq. (12). (b) Decomposition of the tensor T(2)
rank decomposition (CP decomposition) [37, 38] of T(2),
Eq.(13)canbecarriedoutviathetensorrankdecomposition
Eq. (14). (cid:88)
T(2) = P Q R . (14)
v2v3v4 v2h2 v3h2 v4h2
h2∈{0,1}
III. RBM REPRESENTATION OF TNS:
NotethatinaccordancetotheRBMwiththebinaryhid-
SUFFICIENT AND NECESSARY CONDITIONS
den units, the rank of the tensor decomposition should
be two, i.e. P,Q,R are all 2×2 matrices 2 For 3-index
We now address the reverse question about how to 2×2×2 tensor, the decomposition of rank 2 is always
transform a TNS to an RBM with a given architecture. possiblebytheKronecker-Weierstrassnormalformofthe
It is meaningful to seek an RBM representation with linear matrix pencil that the tensor represents [42]. For
given architectures because otherwise one can reproduce an arbitrary tensor, it is difficult to find the rank of ten-
any function using an RBM with exponentially large re- sor [37]. But it is much easier to judge whether it can
sources [9–11]. We present a constructive approach to be decomposed within a given rank. The rank is not
determine the RBM parameters from a TNS represen- smallerthanthedimensionofcoretensorinhigherorder
tation. Appendix B contains a sufficient condition with singularvaluedecomposition[43]. Ifthedimensionofthe
examples including the statistical Ising model and the coretensorislargerthan2, thebinaryconditioncannot
cluster state [41]. be satisfied. After the decomposition, we split each of
Tomakethefollowingdiscussionsconcrete,wepresent these matrices into product of three matrices according
the general procedure using the MPS in Fig. 1(b) as to Eqs. (4-6) for the weight and biases of the RBM, for
an example. Assuming we seek the RBM parameters example
{W ,a ,b }oftheRBMarchitectureshowninFig.1(a),
ij i j (cid:18) (cid:19) (cid:18) (cid:19)(cid:18) (cid:19)(cid:18) (cid:19)
we first require the MPS can be transformed into a TNS p q 1 1 1 1
of structure shown in Fig. 5(a). The TNS factorizes into P = r s =p r 1 ps q . (15)
p qr p
n =4 isolated pieces with given physical indices. It is a
h Comparing to Eqs. (4-6) we obtain
product of the elements of one 2×2×2×2 tensor T(1)
andthree2×2×2tensorsT(2,3,4). Requiringtheproduct ps
W =ln , (16)
equal to the MPS, we have ij qr
r
(cid:32)(cid:89)nv (cid:33) aij =lnp, (17)
Tr A(i)[v ] =T(1) T(2) T(3) T(4) .
i v1v2v3v4 v2v3v4 v3v4v5 v4v5v6 q
b =ln , (18)
i=1 ij p
(12)
Taking the logarithm of the above equation, we obtain
26 = 64 linear equations for the 24 +23 +23 +23 = 40
tensor elements. These equations are therefore over de- 2 Ingeneral,iftherankoftensordecompositionislargerthantwo,
termined. ItisimpossibletotransformtheMPSintothe oneobtainsanRBMwithmultistatehiddenunits.
7
where Wij is the connection weight, aij,bij are partial (a) D2 C2 (b)
contributions to the biases at the two ends of the con-
nection. The overall biases sum up contributions from I3 W=i(cid:83)
all the connections ai = (cid:80)jaij and bj = (cid:80)ibij. In this D1 I4 b=0
way, each tensorofEq. (12)is writtenin theRBM form, C U a=0
1
for example
a=i(cid:83)
Eqs.(12,13)arethesufficientandnecessaryconditions
to translate a TNS into an RBM of a given architecture.
In the case of n = 1, the Eq. (12) is just rephrases the
h (c) D (d) (e) D
MPSasthewavefunctiontensor. Itishoweveringeneral 2 2
impossible to satisfy the subsequent rank two factoriza-
D D
tion requirement Eq. (13) or Eq. (14). By having more 1 1
hidden units and connections one increases the degree of
freedomsinEqs.(12)and(13)untiltheybothhavesolu-
tions, whichisinlinewiththemathematicalresultssug-
gesting that the RBM can represent any function given
sufficientlargehiddenunitsanddenseconnections[9–11].
Inpractice,aconvenientwaytoquicklycheckwhether
a state has a particular RBM representation is to con- Figure 6. (a) The TNS and (b) the RBM representations of
the toric code states. The meaning of each symbol is given
sider the factorization property Eq. (8), i.e. the RBM
in the figure. The dashed red and orange lines are the paths
factorizes into a product state if one fixes a sequence of
of Wilson loops defined in Eq. (26) and Eq. (27). (c-e) The
long enough visible units. For example, this necessary
RBM representations of the toric code states in the other
condition tells us that the Affleck-Kennedy-Lieb-Tasaki
topological sectors. The topological sectors are labelled by
state [44] can not be written as an RBM (of ternary vis-
theeigenvaluesoftheWilsonloopoperators(X ,X )defined
1 2
ible unit) with short range connections because of the in Eq. (26). The cyan dots denote the visible units with bias
existence of the hidden string order. a=iπ .
Finally, it is natural to map a TNS with translational
invariancetoanRBMwithauniformstructure. Forthis
purpose one can assemble tensors on a finite size torus representation of one of them via solving equations for
and seek for the solutions of Eqs. (12,13). thewavefunctioncoefficients. Now,giventheRBM-TNS
correspondence, we can readily construct the RBM rep-
resentation of the toric code wavefunction in all of the
IV. EXAMPLE: THE TORIC CODE STATES four topological sectors based on their elegant TNS rep-
resentations [22, 48].
AsaconcreteexampleofthediscussedRBM-TNScor- WestartwiththebonddimensionD =2PEPSrepre-
respondence, we present the RBM representation of the sentation of the toric code ground state [22, 48] shown
toriccodestates[45]. Moreexamplesincludingthestatis- inFig.6. The4indexidentitytensorI isdefinedinver-
4
tical Ising model and the cluster state [41] are presented tices. The 3 index identity tensor I resides on edges of
3
(cid:18) (cid:19)
in the Appendix B. 1 1
lattice. U = √1 resides on the bond connect-
The toric code model [45] has caught much attention 2 1 −1
ofcondensedmatterandquantuminformationphysicists ing I and I . We identify the bond centered I tensors
4 3 3
becauseitisoneofthesimplestmodelsthathaspossesses as the visible units and the tensors I on the vertices as
4
of topological order [46] and holds the promise for quan- the hidden units of the RBM. Following the recipes of
tum information processing [47]. The toric code Hamil- Eqs. (16-18), we decompose the matrix
tonian reads
(cid:18) (cid:19)(cid:18) (cid:19)(cid:18) (cid:19)
(cid:88) (cid:88) 1 1 1 1 1
H=− A+− B(cid:3), (19) U = √2 e0 1 eiπ e0 , (21)
+ (cid:3)
(cid:89) (cid:89)
A+ = σiz, B(cid:3) = σix, (20) corresponding to the weights and biases of the RBM,
i∈+ i∈(cid:3)
W =iπ, (22)
where Pauli matrices σx,σz reside on the bonds of the
i i
squarelatticewithperiodicboundarycondition. A con- a=0, (23)
+
sists of the product of four σz operators connecting to b=0. (24)
i
each vertex denoted by +, B(cid:3) consists of the product of
four σx operators on each square plaquette (cid:3). All the The resulting RBM representation of the toric state is
i
A+ and B(cid:3) terms commute. shown in Fig. 6(b). The magenta hidden unit couples to
The toric code ground state on a torus has four topo- thefourvisiblevariablesaroundeachvertex. Tracingout
logically sectors. Reference [18] already gave the RBM the hidden units, the RBM function defined in Eq. (2) is
8
(a) (b)
E D1' RBM SecⅡ MPS
Canonicalize
Figure 7. The symbols are the same as Fig. 6. (a) Acting RBM MPS
fourA operators[Eq.(20)]inthecenterchangesthevisible
+ Sec Ⅲ
biases along the path E to iπ. The RBM function is still
in the (+,+) sector identical to Fig. 6(b). (c) The RBM
wavefunction remains in the (+,−) sector as Fig. 6(d) after
four A operations. The operation moves the path D in
+ 1 Figure 8. Simplify an RBM by mapping it into an MPS and
Fig. 6(d) to D(cid:48).
1 then transforming the canonicalized MPS back. For detailed
discussions of the mappings, see Sec. II and Sec. III.
a product over all the vertices
ΨTC(v)=(cid:89)(cid:16)1+eiπ(cid:80)i∈+vi(cid:17), (25) ianndFiDg.2(cid:48)7a(bre).reIltatisedthbeyparoldouocptooffσthizeoAperaotopresraatloorngsuEr-
+
+ rounded by the path E. Since the ground state is the
whichrequiresanevennumberofvisibleunitswithv =1 eigenstate of every term in the toric code Hamiltonian
i
at each vertex. The RBM state thus represents an equal Eq. (19), acting A+ to arbitrary vertex will not change
weight superposition of the closed loop configurations as the wavefunction. In Fig. 7(a), the visible bias along the
the toric ground state [45]. This construction is simper closed loop E is a = iπ. The wavefunction is the same
than Ref. [18] which introduced hidden units on the ver- as Fig. 6(b) because the path does not winds around the
tices as well as in the plaquette centers. torus. Similarly, the state in Fig. 7(b) is identical to
The topological property of the toric code state can Fig. 6(c). The redundancy of the RBM parametrization
also be demonstrated in the RBM language. We first will be discussed in detail in Sec. V.
define two Wilson loop operators The PEPS wavefunctions can be decomposed into
products of translationally invariant tensors in the form
(cid:89) (cid:89)
X = σx, X = σx, (26) ofEq.(12)inSec.III.Conversely,startingfromtheRBM
1 i 2 i
representationEq.(25)wewillobtainthePEPSbackfol-
i∈C1 i∈C2
lowing the instructions of Sec. II.
where C ,C denote the paths along the x/y directions
1 2
indicated by the red lines in Fig. 6(a). The Wilson loop
operatorsX ,X aremutuallycommutingandtheycom-
1 2 V. SIMPLIFYING THE RBM USING TNS
mute with the Hamiltonian (19). So we can label the
METHODS
toriccodegroundstatesusingtheeigenvaluesofEq.(26).
Therearetotallyfourtopologicalsectors. Thewavefunc-
SimilartoTNS,itisknownthatthereareredundancy
tionEq.(25)illustratedinFig.6(b)belongstothe(+,+)
sector, we denote it by (cid:10){v }|Ψ(+,+)(cid:11)=Ψ ({v }). in the neural network functions [3]. A network with dif-
i TC i
ferent connection weights and biases may result into the
Next, we define anothertwooperators alongthe paths
same function. The knowledge about the redundancy in
D ,D indicated by the dashed orange lines in Fig. 6(a),
1 2
TNS can help simplify the RBMs.
Z = (cid:89) σz, Z = (cid:89) σz, (27) Figure 8 illustrated the idea with diagrams. Given an
1 i 2 i
RBM, we first obtain its MPS representation following
i∈D1 i∈D2
the algorithms of Sec. II. We then transform the MPS
These two operators do not commute with X1,X2. They into its canonical form by performing the Schmidt de-
translatethestatesintodifferenttopologicalsectors[49]. composition at each bond [34, 35]. This will reduce the
(cid:12)(cid:12)Ψ(−,+)(cid:11) = Z2(cid:12)(cid:12)Ψ(+,+)(cid:11), (cid:12)(cid:12)Ψ(+,−)(cid:11) = Z1(cid:12)(cid:12)Ψ(+,+)(cid:11), bond dimensions to the minimal requirement and fix the
(cid:12)(cid:12)Ψ(−,−)(cid:11) = Z1Z2(cid:12)(cid:12)Ψ(+,+)(cid:11). In the PEPS representa- gauge degree of freedoms. Finally, we map the canon-
tion,theoperatorsZ ,Z modifytheidentitytensorI in icalized MPS to another RBM following the procedure
1 2 3
Fig.6(a)intoadiagonaltensordiag(1,−1). Accordingly, of Sec. III. Because the canonicalized MPS has smaller
this changes the bias of the visible units in Eq. (23) to bonddimensions,thefinalRBMwillbesimplerthanthe
a=ln(−1)=iπalongtheD andD lines[Fig.6(c,d,e)]. original one.
1 2
Note that the Wilson lines D do not need to be As an example of the above procedure, we consider
1,2
straight lines. It is only crucial that they wind around the RBM representation of the 1D cluster state given in
thetorus,seeFig.7. TheX definedalongD [Fig.6(d)] Ref.[18]. TheRBMstatehasn =n hiddenunitswith
1 1 h v
9
(a) (b) cause the ground state is a Z spin liquid state [52, 53].
2
Since the redundancies in non-injective case are richer,
I3 I4 it offers us more possibility of simplifying the RBM via
I different TNS constructions.
I 4
4
U A
U B VI. SUMMARY AND DISCUSSION
The constructive connection between RBM and TNS
Figure 9. Equivalent TNS representations of the toric code functions bridges between the worlds of deep learning
ground states. (a) The simplex is defined in the plaquette. andquantummany-bodyphysics. TheRBM-TNScorre-
(cid:18) (cid:19)
1 1
The green dot is the matrix U = √1 . (b) The spondence has broad implications for both communities.
2 1 −1 The entanglement entropy bound of the TNS can rigor-
simplex is defined both in vertex and plaquette. The 5 index
ously quantify the expressive power of the RBM func-
tensor A,B are given by projectors acting on an arbitrary
tions. This physics perspective provides tighter bounds
state.
on the number of hidden units and the connectivities of
the RBMs than previously known results solely based
each hidden unit connecting to three visible units. Ac- on mathematical considerations [9–11]. Akin to the suc-
cording to Fig. 2 it maps to a D =4 MPS because each cess of the TNS methods in the quantum physics, our
bipartition will cut n=2 connections. However, canoni- finding suggests that the success of RBM learning may
calization of the MPS or using the scheme of Fig. 4 will be related to the relatively low entanglement entropy in
reduce the bond dimension to D =2. Mapping the sim- the typical datasets fulfilling the physics laws such as
plified MPS back, we obtain an RBM with each hidden the natural images and speech signals. Exploiting the
unitconnectingonlytotwoneighboringvisibleunits. See RBM-TNS correspondence one may put the previously
Appendix B for detailed discussions of this RBM archi- argued operational similarity between the renormaliza-
tecture and the explicit parameters in Tab. I. MPS rep- tion group and deep learning [7] to a more rigorous and
resentation of the RBM function, and in particular its constructive manner. On the technical side, TNS meth-
canonicalform, isveryusefultorevealtheessence ofthe ods have already been used for pattern recognition [54].
state dictated by the entanglement entropy requirement. Given the deeper connection between the two fields sug-
Thesituationishowevermorecomplicatedifonemaps gested by the RBM-TNS correspondence, more insights
an RBM to a higher dimensional TNS such as PEPS. andcomputationaltoolsofbothfieldscanbeexchanged.
There is no general numerical approach to remove the For example, the canonical MPS [34, 35] and the tensor
redundancy in the PEPS. There are two cases, renormalization group methods [55] can be employed to
1)Thetrivialcase,wherethetwoPEPSscanbetrans- transform and simplify the RBM architectures.
formed into each other by gauge transform in the bonds. On the other hand, the RBM is a more compact vari-
This corresponds to the different assignment of the cross ational wavefunctions [17] because it can represent the
gate in Fig. 2. TNS with much fewer parameters, see discussions in
2) The non-trivial case. The two PEPSs are so dif- Sec. II. Deep learning algorithms and industrial soft-
ferent that we can’t transform one into the other with ware/hardware [56–58] may thus be beneficial for quan-
only gates acting on the bonds. This is due to the non- tum many-body physics researches through this connec-
injectivity property [50] of the wavefunction. Take the tion.
toriccodestateinSec.IVasanexample. Wecanactthe For future research directions, it is interesting to fur-
projectionoperatorP =(cid:81) 1+A+ (cid:81) 1+B(cid:3) toarbitrary ther explore the connection between deep learning archi-
+ 2 (cid:3) 2 tectures such as the deep Boltzmann machines [31] and
wavefunctions to obtain the ground state. The construc-
the TNS with hierarchical structures such as tree tensor
tion is therefore not unique and the TNS of toric code
networks[59–62]andmulti-scaleentanglementrenormal-
is not injective [51] either. The state shown in Fig. 6(a)
is a result of projection on the state (cid:16)|↑(cid:105)√+|↓(cid:105)(cid:17)⊗nv. If iuznadtieornstaannsdaintzg[t3h9e].eIxnprpeassssiivnegpwoewearlsoofntohteedtheeepefffeoerdtfsoor-f
2
weinsteadprojectontothestateP|↑(cid:105)⊗nv wewillobtain ward neural networks [63, 64]. We believe that the in-
anotherPEPSwiththe4indextensordefinedinthepla- sightsofthequantumentanglementandtensornetworks
quette, see Fig. 9(a). More generally, we can apply the canprovideevendeeperunderstandingofthedeeplearn-
projector P to an arbitrary product state to obtain the ing and guides better neural nets design.
toric code ground state. For example, the TNS can have
4 index tensors defined both at the vertices and at pla-
quette center as shown in Fig. 9(b). The corresponding VII. ACKNOWLEDGMENTS
RBMrepresentationistheonediscussedintheRef.[18].
Althoughtheyareallequivalentstates,itisimpossibleto We thank Dongling Deng, Xiaopeng Li, Chen Fang,
findalocaltransformationtoconnectthelocaltensorbe- Hong-Hao Tu and Yi-feng Yang for helpful discussions.
10
This work is supported by the National Natural Science 11190024) and the Ministry of Science and Technology
Foundation of China (Grants No. 11474331 and No. of China (Grant No. 2016YFA0302400).
[1] Y. LeCun, Y. Bengio, and G. Hinton, Nature 521, 436 [29] E.StoudenmireandS.R.White,AnnualReviewofCon-
(2015). densed Matter Physics 3, 111 (2012).
[2] I.Goodfellow,Y.Bengio, andA.Courville,“Deeplearn- [30] G.E.HintonandT.J.Sejnowski,inParallel distributed
ing,” (2016), book in preparation for MIT Press. processing: Explorations in the microstructure of cogni-
[3] C. M. Bishop, Neural Networks for Pattern Recognition, tion, Vol. 1 (1986) pp. 282–317.
1st ed. (Clarendon Press, Oxford : New York, 1996). [31] R. Salakhutdinov and G. E. Hinton, JMLR Workshop
[4] S. S. Haykin, Neural networks and learning machines, Conf. Proc. 5, 448 (2009).
Vol. 3 (Pearson Upper Saddle River, NJ, USA, 2009). [32] Seehttps://github.com/yzcj105/rbm2mpsfortheMat-
[5] M.A.Nielsen,Neural Networks and Deep Learning (De- labimplementationsoftheRBMtoMPStransformation
termination Press, 2015). corresponding to Fig. 2 and Fig. 4 respectively.
[6] D. J. Amit, H. Gutfreund, and H. Sompolinsky, Phys. [33] B. M. Marlin, K. Swersky, B. Chen, and N. de Freitas,
Rev. A 32, 1007 (1985). JMLR Worshop Conf. Proc. 9, 509 (2010).
[7] P. Mehta and D. J. Schwab, ArXiv e-prints (2014), [34] G. Vidal, Phys. Rev. Lett. 91, 147902 (2003).
arXiv:1410.3831 [stat.ML]. [35] G. Vidal, Phys. Rev. Lett. 93, 040502 (2004).
[8] H. W. Lin and M. Tegmark, ArXiv e-prints (2016), [36] U. Schollwöck, Annals of Physics 326, 96 (2011),
arXiv:1608.08225 [cond-mat.dis-nn]. arXiv:1008.3477 [cond-mat.str-el].
[9] Y. Freund and D. Haussler, “Unsupervised learning of [37] https://en.wikipedia.org/wiki/Tensor_rank_
distributionsofbinaryvectorsusingtwolayernetworks,” decomposition.
(1994). [38] F. L. Hitchcock, J. Math. Phys 6, 164 (1927).
[10] N.LeRouxandY.Bengio,Neuralcomputation20,1631 [39] G. Vidal, Phys. Rev. Lett. 99, 220405 (2007).
(2008). [40] G. Evenbly and G. Vidal, Physical review letters 112,
[11] G. Montufar and N. Ay, Neural computation 23, 1306 240502 (2014).
(2011). [41] H. J. Briegel and R. Raussendorf, Phys. Rev. Lett. 86,
[12] G. E. Hinton, Neural computation 14, 1771 (2002). 910 (2001).
[13] T.Tieleman,TrainingrestrictedBoltzmannmachinesus- [42] J. M. Landsberg, Tensors: geometry and applications,
ingapproximationstothelikelihoodgradient (ACM,New Vol. 128 (American Mathematical Society Providence,
York, New York, USA, 2008). RI, USA, 2012).
[14] http://www.deeplearning.net/tutorial/rbm.html. [43] T. G. Kolda and B. W. Bader, SIAM review 51, 455
[15] G.E.HintonandR.R.Salakhutdinov,Science313,504 (2009).
(2006). [44] I.Affleck,T.Kennedy,E.H.Lieb, andH.Tasaki,Phys.
[16] G.E.Hinton,S.Osindero, andY.W.Teh,Neuralcom- Rev. Lett. 59, 799 (1987).
putation 18, 1527 (2006). [45] A. Kitaev, Annals of Physics 303, 2 (2003).
[17] G. Carleo and M. Troyer, ArXiv e-prints (2016), [46] X.G.Wen,QuantumFieldTheoryofMany-BodySystems
arXiv:1606.02318 [cond-mat.dis-nn]. (Oxford University Press, 2007).
[18] D.-L. Deng, X. Li, and S. Das Sarma, ArXiv eprint [47] S. B. Bravyi and A. Y. Kitaev, eprint arXiv:quant-
(2016), arXiv:1609.09060 [cond-mat.dis-nn]. ph/9811052 (1998), quant-ph/9811052.
[19] G.TorlaiandR.G.Melko,PhysicalReviewB94,165134 [48] F. Verstraete, M. M. Wolf, D. Perez-Garcia, and J. I.
(2016). Cirac, Phys. Rev. Lett. 96, 220601 (2006).
[20] L.HuangandL.Wang,Phys.Rev.B95,035105(2017). [49] A. Kitaev, A. Shen, and M. Vyalyi, Classical and
[21] J. Liu, Y. Qi, Z. Y. Meng, and L. Fu, Phys. Rev. B 95, Quantum Computation, Graduate studies in mathemat-
041101 (2017). ics (American Mathematical Society, 2002).
[22] R. Orús, Annals of Physics 349, 117 (2014). [50] N. Schuch, I. Cirac, and D. PšŠrez-Garcšaa, Annals of
[23] D. Perez-Garcia, F. Verstraete, M. M. Wolf, and J. I. Physics 325, 2153 (2010).
Cirac, eprint arXiv:quant-ph/0608197 (2006), quant- [51] N.Schuch,D.Poilblanc,J.I.Cirac, andD.Pérez-García,
ph/0608197. Phys. Rev. B 86, 115108 (2012).
[24] I. V. Oseledets, SIAM Journal on Scientific Computing [52] N. Read and S. Sachdev, Phys. Rev. Lett. 66, 1773
33, 2295 (2011). (1991).
[25] F. Verstraete and J. I. Cirac, eprint arXiv:cond- [53] X. G. Wen, Phys. Rev. B 44, 2664 (1991).
mat/0407066 (2004), cond-mat/0407066. [54] E. Miles Stoudenmire and D. J. Schwab, ArXiv e-prints
[26] F. Verstraete, V. Murg, and J. I. Cirac, Advances in (2016), arXiv:1605.05775 [stat.ML].
Physics 57, 143 (2008). [55] M. Levin and C. P. Nave, Phys. Rev. Lett. 99, 120601
[27] J.Eisert,M.Cramer, andM.B.Plenio,Rev.Mod.Phys. (2007).
82, 277 (2010). [56] https://www.nvidia.com/en-us/deep-learning-ai/
[28] M. A. Nielsen and I. L. Chuang, Quantum computation developer.
and quantum information (Cambridge university press, [57] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis,
2010). J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard,
M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G.