Table Of ContentStructural Attention Neural Networks for improved sentiment analysis
FilipposKokkinos AlexandrosPotamianos
SchoolofE.C.E., SchoolofE.C.E.,
NationalTechnicalUniversityofAthens,NationalTechnicalUniversityofAthens,
15773Athens,Greece 15773Athens,Greece
[email protected] [email protected]
Abstract applied to various Natural Language Processing
tasks. Tree structured neural networks, which
We introduce a tree-structured attention
are found in literature as Recursive Neural Net-
neural network for sentences and small
works, hold a linguistic interest due to their close
7
phrasesandapplyittotheproblemofsen-
1 relation to syntactic structures of sentences being
0 timent classification. Our model expands abletocapturedistributedinformationofstructure
2 the current recursive models by incor-
such as logical terms(Socher et al., 2012). These
n porating structural information around a
syntactic structures are N-ary trees which repre-
a
nodeofasyntactictreeusingbothbottom-
J sent either the underlying structure of a sentence,
upandtop-downinformationpropagation.
7 known as constituency trees or the relations be-
Also, the model utilizes structural atten- tweenwordsknownasdependencytrees.
] tion to identify the most salient represen-
L
tations during the construction of the syn-
C
. tactic tree. To our knowledge, the pro- This paper focuses on sentence-level sentiment
s
c posed models achieve state of the art per- classification of movie reviews using syntactic
[ formanceontheStanfordSentimentTree-
parse trees as input for the proposed networks. In
1 bankdataset.
order to solve the task of sentiment analysis of
v
1 1 Introduction sentences, we work upon a variant of Recursive
1 Neural Networks which recursively create repre-
8
Sentiment analysis deals with the assessment sentation following the syntactic structure. The
1
0 of opinions, speculations, and emotions in text proposedcomputationmodelexploitsinformation
. (Zhang et al., 2012; Pang and Lee, 2008). It is fromsubnodesaswellasparentnodesofthenode
1
0 a relatively recent research area that has attracted under examination. This neural network is re-
7
greatinterestasdemonstratedbyaseriesofshared ferred to as Bidirectional Recursive Network (Ir-
1
: evaluation tasks, e.g., analysis of tweets (Nakov soy and Cardie, 2013). The model is further en-
v
et al., 2016). In (Turney and Littman, 2002), hancedwithmemoryunitsandtheproposedstruc-
i
X the affective ratings of unknown words were pre- tural attention mechanism. It is observed that dif-
r dicted utilizing the affective ratings of a small set ferent nodes of a tree structure hold information
a
of words (seeds) and the semantic relatedness be- of variable saliency. Not all nodes of a tree are
tween the unknown and the seed words. An ex- equally informative, so the proposed model se-
ample of sentence-level analysis was proposed in lectively weights the contribution of each node
(Malandrakisetal.,2013). Otherapplicationareas regarding the sentence level representation using
includethedetectionofpublicopinionandpredic- structuralattentionmodel.
tion of election results (Singhal et al., 2015), cor-
relation of mood states and stock market indices
(Bollenetal.,2011). Weevaluateourapproachonthesentence-level
Recently, Recurrent Neural Network (RNN) sentiment classification task using one standard
withLong-ShortTermMemory(LSTM)(Hochre- movie review dataset (Socher et al., 2013). Ex-
iter and Schmidhuber, 1997) or Gated Recur- perimental results show that the proposed model
rent Units (GRU) (Chung et al., 2014) have been outperformsthestate-of-theartmethods.
2 Tree-StructuredGRUs 2.1 BidirectionalTreeGRU
A natural extension of Tree-Structure GRU is the
Recursive GRUs (Tree-Gru) upon tree structures
addition of a bidirectional approach. TreeGRUs
areanextensionofthesequentialGRUsthatallow
calculate an activation for node j with the use of
informationtopropagatethroughnetworktopolo-
previously computed activations lying lower in
gies. Similar toRecursive LSTM networkon tree
the tree structure. The bidirectional approach for
structures (Tai et al., 2015), for every node of a
a tree structure uses information both from under
tree, the Tree-GRU has gating mechanisms that
and lower nodes of the tree for a particular node
modulate the flow of information inside the unit
j. In this manner, a newly calculated activation
without the need of a separate memory cell. The
incorporates content from both the children and
activation h of Tree-GRU for node j is the inter-
j
theparentofaparticularnode.
polation of the previous calculated activation h
jk
Thebidirectionalneuralnetworkcanbetrainedin
ofitskthchildoutofN totalchildrenandthecan-
two seperate phases: i) the Upward phase and ii)
didateactivation(cid:101)hj .
the Downward phase. During the Upward phase,
the network topology is similar to the topology
N
(cid:88) of a TreeGru, every activation is calculated based
hj = zj ∗ hjk +(1−zj)∗(cid:101)hj (1)
onthepreviouslycalculatedactivationswhichare
k=1
found lower on the structure in a bottom up fash-
where z is the update function which decide the ion. When every activation has been computed,
j
degree of update that will occur on the activation fromleavestoroot,thentherootactivationisused
basedontheinputvectorx andpreviouslycalcu- as input of the Downward phase. The Downward
j
latedrepresentationh : phase calculates the activations for every child
jk
of a node using content from the parent in a top
N down fashion. The process of computing the
(cid:88)
z = σ(U ∗x + Wi∗h ) (2) internalrepresentationsbetweenthetwophasesis
j z j z jk
k=1 separated, so in a first pass the network compute
the upward activation and after this is completed,
Thecandidateactivation(cid:101)hj foranodej iscom- thenthedownwardrepresentationsarecomputed.
puted similarly to that of a Recursive Neural Net- The upward activation h↑j similarly to TreeGRU
workasin(Socheretal.,2011): for node j is the interpolation of the previ-
ous calculated activation h↑ of its kth child out
jk
(cid:101)hj = f(Uh∗xj +(cid:88)N Whk ∗(hjk ∗rj)) (3) ofNtotalchildrenandthecandidateactivation(cid:101)h↑j.
k=1
N
where rj is the reset gate which allows the net- h↑ = z↑∗(cid:88)h↑ +(1−z↑)∗(cid:101)h↑ (5)
j j jk j j
worktoforgeteffectivelypreviouscomputedrep-
k=1
resentations when the value is close to 0 and it is
Theupdategate,restgateandcandidateactiva-
computedasfollows:
tionarecomputedasfollows:
N
(cid:88) N
rj = σ(Ur ∗xj + Wrk ∗hjk) (4) zj↑ = σ(Uz ∗x↑j +(cid:88)Wzk ∗h↑jk) (6)
k=1
k=1
Every part of a gated recurrent unit N
xj,hj,rj,zj,(cid:101)hj ∈ Rd where d is the input rj↑ = σ(Ur ∗x↑j +(cid:88)Wrk ∗h↑jk) (7)
vector dimensionality. σ is the sigmoid function k=1
and f is the non-linear tanh function.The set of
N
matrices Wk,U ∈ Rdxd used in 2 - 4 are the (cid:101)h↑j = f(Uh∗x↑j +(cid:88)Wrk ∗(h↑jk ∗rj↑)) (8)
trainable weight parameters which connect the
k=1
kth children node representation with the jth
noderepresentationandtheinputvectorx .
j
are used to form the final sentence representation
s ∈ Rd which is a weighted summation of every
noderepresentationh .
j
u(cid:62)∗u
j w
a = (14)
j (cid:80)N u(cid:62)∗u
i=1 i w
N
(cid:88)
s = a h (15)
i i
i=1
Theproposedattentionmodelisappliedonstruc-
Figure 1: A tree-structured bidirectional neural
turalcontentsinceallnoderepresentationscontain
network with Gated Recurrent Units. The input
syntacticstructuralinformationduringtrainingbe-
vectorsxaregiventothemodelinordertogener-
causeoftherecursivenatureofthenetworktopol-
atethephraserepresentationsy↑ andy↓.
ogy.
The downward activation h↑ for node j is the 3 Experiments
j
interpolation of the previous calculated activation We evaluate the performance of the aforemen-
h↑ ,wherethefunctionpcalculatestheindexof tioned models on the task of sentiment classifi-
p(j)
theparentnode,andthecandidateactivation(cid:101)h↓. cation of sentences sampled from movie reviews.
j
We use the Stanford Sentiment Treebank (Socher
h↓ = z↓∗h↓ +(1−z↓)∗(cid:101)h↓ (9) et al., 2013) dataset which contains sentiment la-
j j p(j) j j
belsforeverysyntacticallyplausiblephraseoutof
The update gate, reset gate and candidate activa- the8544/1101/2210train/dev/testsentences. Each
tionforthedownwardphasearecomputedasfol- phrase is labeled with respect to a 5-class sen-
lows: timent value, i.e. very negative, negative, neu-
tral, positive, very positive. The dataset can also
z↓ = σ(Ud∗h↑+Wd∗h↓ ) (10)
j z j z p(j) be used for a binary classification subtask by ex-
cluding any neutral phrases for the original splits.
r↓ = σ(Ud∗h↑+Wd∗h↓ ) (11)
j r j r p(j) The binary classification subtask is evaluated on
(cid:101)h↓ = f(Ud∗h↑+Wd∗(h↓ ∗r↓)) (12) 6920/872/1821train/dev/testsplits.
j h j h p(j) j
3.1 SentimentClassification
During downward phase, matrix Ud ∈ Rdxd
connectstheupwardrepresentationofnodej with Foralloftheaforementionedarchitecturesateach
the respective jth downward node while Wd ∈ node j we use a softmax classifier to predict the
Rdxd connecttheparentrepresentationp(j). sentiment labelyˆj. For example, the predicted la-
bel yˆ corresponds to the sentiment class of the
j
2.2 StructuralAttention spannedphraseproducedfromnodej. Theclassi-
We introduce Structural Attention, a generaliza- fierforunidirectionalTreeGRUarchitecturesuses
tion of sequential attention model (Luong et al., thehiddenstatehj producedfromrecursivecom-
2015) which extracts informative nodes out of a putations till node j using a set xj of input nodes
syntactictreeandaggregatestherepresentationof topredictthelabelasfollows:
those nodes in order to form the sentence vec-
pˆ (y|x ) = softmax(W ∗h ) (16)
θ j s j
tor. We feed representation h of node through a
j
one-layerMultilayerPerceptronwithWw ∈ Rdxd where Ws ∈ Rdxc and c is the number of senti-
weightmatrixtogetthehiddenrepresentationu . mentclasses.
j
The classifier for bidirectional TreeBiGRU ar-
uj = tanh(Ww ∗hj) (13) chitectures uses both the hidden state h↑ and h↓
j j
produced from recursive computations till node j
Using the softmax function, the weights a for
j during Upward and Downward Phase using a set
each node are obtained based on the similarity
x ofinputnodestopredictthelabelasfollows:
j
of the hidden representation u and a global con-
j
text vetor u ∈ Rd. The normalized weights a pˆ (y|x ) = softmax(W↑∗h↑+W↓∗h↓) (17)
w j θ j s j s j
where W↑,W↓ ∈ Rdxc and c is the number of System Binary Fine-grained
s s
RNN 82.4 43.2
sentiment classes. The predicted label yˆ is the
j MV-RNN 82.9 44.4
argumentwiththemaximumconfidence: RNTN 85.4 45.7
PVec 87.8 48.7
yˆj = argmax(pˆθ(y|xj)) (18) TreeLSTM 88.0 51.0
y DRNN 86.6 49.8
DCNN 86.8 48.5
FortheStructuralAttentionmodels,weuseforthe
CNN-multichannel 88.1 47.4
final sentence representation s to predict the sen- DMN 88.6 52.1
timent label yˆ where j is the corresponding root TreeGRU
j
-withoutattention 88.6 50.5
node of a sentence. The cost function used is the
-withattention 89.0 51.0
negative log-likelihood of the ground-truth label TreeBiGRU
yk ateachnode: -withoutattention 88.5 51.3
-withattention 89.5 52.4
m
(cid:88) λ
E(θ) = pˆ (yk|xk)+ ||θ||2 (19) Table2: TestAccuraciesachievedontheStanford
θ
2
k=1 SentimentTreebankdataset. RNN,MV-RNNand
RNTN (Socher et al., 2013). PVec: (Mikolov et
wheremisthenumberoflabelsinatrainingsam-
al., 2013). TreeLSTM (Tai et al., 2015). DRNN
pleandλistheL2regularizationhyperparameter.
(IrsoyandCardie,2013). DCNN(Kalchbrenneret
al., 2014).CNN-multichannel (Kim, 2014). DMN
NetworkVariant d |θ| (Kumaretal.,2015)
TreeGRU
-withoutattention 300 7323005
-withattention 300 7413605
with probability 0.5 on both the input layer and
TreeBiGRU
-withoutattention 300 8135405 thesoftmaxlayer.
-withattention 300 8317810 The word embeddings are initialized using the
public available Glove vectors with a 300 dimen-
Table 1: Memory dimensions d and total network
sionality. TheGlovevectorsprovide95.5%cover-
parameters|θ|foreverynetworkvariantevaluated
age for the SST dataset. All initialized word vec-
torsarefinetunedduringthetrainingprocessalong
3.2 Results with every other parameter. Every matrix is ini-
tialized with the identity matrix multiplied by 0.5
The evaluation results are presented in Table 2
except for the matrices of the softmax layer and
in terms of accuracy, for several state-of-the-art
the attention layer which are randomly initialized
models proposed in the literature as well as for
fromthenormalGaussiandistribution. Everybias
the TreeGRU and TreeBiGRU models proposed
vectorsisinitializedwithzeros.
in this work. Among the approaches reported in
The training process lasts for 40 epochs. During
the literature, the highest accuracy is yielded by
training, we evaluate the network 4 times every
DRNN and DMN for the binary scheme (88.6),
epochandkeeptheparameterswhichgivethebest
and by DMN for the fine-grained scheme (52.1).
rootaccuracyonthedevelopmentdataset.
We observe that the best performance is achieved
by TreeBiGRU with attention, for both binary
5 Conclusion
(89.5) and fine-grained (52.4) evaluation metrics,
exceeding any previously reported results. In ad- In this short paper, we propose an extension of
dition,theattentionalmechanismemployedinthe Recursive Neural Networks that incorporates a
proposed TreeGRU and TreeBiGRU models im- bidirectional approach with gated memory units
provetheperformanceforbothevaluationmetrics. as well as an attention model on structure level.
Theproposedmodelswereevaluatedonbothfine-
4 HyperparametersandTrainingDetails
grained and binary sentiment classification tasks
The evaluated models are trained using the Ada- on a sentence level. Our results indicate that both
Grad (Duchi et al., 2010) algorithm using 0.01 the direction of the computation and the attention
learningrateandaminibatchofsize25sentences. on a structural level can enhance the performance
L2-regularization is performed on the model pa- ofneuralnetworksonasentimentanalysistask.
rameters with a λ value 10−4. We use dropout
References Richard Socher, Jeffrey Pennington, Eric H. Huang,
AndrewY.Ng,andChristopherD.Manning. 2011.
Johan Bollen, Huina Mao, and Xiaojun Zeng. 2011.
Semi-supervisedrecursiveautoencodersforpredict-
Twittermoodpredictsthestockmarket. Journalof
ing sentiment distributions. In Proceedings of the
ComputationalScience,2(1):1–8.
Conference on Empirical Methods in Natural Lan-
guage Processing, EMNLP ’11, pages 151–161,
Junyoung Chung, C¸aglar Gu¨lc¸ehre, KyungHyun Cho,
Stroudsburg, PA, USA. Association for Computa-
andYoshuaBengio. 2014. Empiricalevaluationof
tionalLinguistics.
gatedrecurrentneuralnetworksonsequencemodel-
ing. CoRR,abs/1412.3555.
Richard Socher, Brody Huval, Christopher D. Man-
John Duchi, Elad Hazan, and Yoram Singer. 2010. ning, and Andrew Y. Ng. 2012. Semantic com-
Adaptive subgradient methods for online learning positionalitythroughrecursivematrix-vectorspaces.
andstochasticoptimization. Technicalreport,EECS In Proceedings of the 2012 Joint Conference on
Department,UniversityofCalifornia,Berkeley. Empirical Methods in Natural Language Process-
ing and Computational Natural Language Learn-
SeppHochreiterandJu¨rgenSchmidhuber. 1997. Long ing, EMNLP-CoNLL’12, pages1201–1211.Asso-
short-term memory. Neural Comput., 9(8):1735– ciationforComputationalLinguistics.
1780.
Richard Socher, Alex Perelygin, Jean Y Wu, Jason
OzanIrsoyandClaireCardie. 2013. Bidirectionalre- Chuang, Christopher D Manning, Andrew Y Ng,
cursiveneuralnetworksfortoken-levellabelingwith andChristopherPotts. 2013. Recursivedeepmod-
structure. CoRR,abs/1312.0493. els for semantic compositionality over a sentiment
treebank. In Proceedings of the conference on
NalKalchbrenner,EdwardGrefenstette,andPhilBlun-
empirical methods in natural language processing
som. 2014. A convolutional neural network for
(EMNLP).
modellingsentences. CoRR,abs/1404.2188.
Kai Sheng Tai, Richard Socher, and Christopher D.
Yoon Kim. 2014. Convolutional neural networks for
Manning. 2015. Improved semantic representa-
sentenceclassification. CoRR,abs/1408.5882.
tions from tree-structured long short-term memory
Ankit Kumar, Ozan Irsoy, Jonathan Su, James Brad- networks. CoRR,abs/1503.00075.
bury,RobertEnglish,BrianPierce,PeterOndruska,
Peter Turney and Michael L Littman. 2002. Un-
Ishaan Gulrajani, and Richard Socher. 2015. Ask
supervised learning of semantic orientation from a
me anything: Dynamic memory networks for natu-
hundred-billion-wordcorpus.
rallanguageprocessing. CoRR,abs/1506.07285.
Minh-Thang Luong, Hieu Pham, and Christopher D. ZhuZhang,XinLi,andYuboChen. 2012. Decipher-
Manning. 2015. Effective approaches to ingword-of-mouthinsocialmedia:Text-basedmet-
attention-based neural machine translation. CoRR, ricsofconsumerreviews. ACMTrans.Manage.Inf.
abs/1508.04025. Syst.,3(1):5:1–5:23.
Nikolaos Malandrakis, Abe Kazemzadeh, Alexandros
Potamianos, andShrikanthNarayanan. 2013. Sail:
A hybrid approach to sentiment analysis. In Pro-
ceedingsSemEval,pages438–442.
TomasMikolov,IlyaSutskever,KaiChen,GregSCor-
rado, and Jeff Dean. 2013. Distributed representa-
tionsofwordsandphrasesandtheircompositional-
ity. In Advances in neural information processing
systems,pages3111–3119.
Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio
Sebastiani, and Veselin Stoyanov. 2016. Semeval-
2016 task 4: Sentiment analysis in twitter. In Pro-
ceedings of the 10th international workshop on se-
manticevaluation(SemEval2016).
Bo Pang and Lillian Lee. 2008. Opinion mining and
sentiment analysis. Found. Trends Inf. Retr., 2(1-
2):1–135.
Kartik Singhal, Basant Agrawal, and Namita Mittal.
2015. Modelingindiangeneralelections: sentiment
analysis of political twitter data. In Information
Systems Design and Intelligent Applications, pages
469–477.
