Neural Models for Sequence Chunking

0.32 MB·

by  Feifei Zhai

Neural Models for Sequence Chunking
FeifeiZhai,SaloniPotdar,BingXiang,BowenZhou
IBMWatson
1101KitchawanRoad,YorktownHeights,NY10598
{fzhai,potdars,bingxia,zhou}@us.ibm.com
7 Abstract O B-NP B-VP I-VP B-ADJP I-ADJP
1
0 Many natural language understanding (NLU) tasks, such as
2 shallow parsing (i.e., text chunking) and semantic slot fill- But it could be much worse
ing, require the assignment of representative labels to the
n
meaningful chunks in a sentence. Most of the current deep
a neural network (DNN) based methods consider these tasks Figure1: Anexampleoftextchunkingwhereeachwordis
J
as a sequence labeling problem, in which a word, rather labeled using the IOB scheme. The chunk “could be” is a
5 thanachunk,istreatedasthebasicunitforlabeling.These verb phrase (VP) and “it” is a single-word chunk with NP
1 chunksaretheninferredbythestandardIOB(Inside-Outside- label.
Beginning)labels.Inthispaper,weproposeanalternativeap-

] proachbyinvestigatingtheuseofDNNforsequencechunk-
L
ing,andproposethreeneuralmodelssothateachchunkcan
C be treated as a complete unit for labeling. Experimental re-
. sultsshowthattheproposedneuralsequencechunkingmod-
s Most of the current DNN based approaches use the IOB
elscanachievestart-of-the-artperformanceonboththetext
c scheme to label chunks. However, this approach of these
[ chunkingandslotfillingtasks.
labels has a few drawbacks. First, we don’t have an ex-

1 plicit model to learn and identify the scope of chunks in a
v Introduction sentence, instead we infer them implicitly (by IOB labels).
7 Semantic slot filling and shallow parsing which are stan- Hence the learned model might not be able to fully utilize
2 dardNLUtasksfallundertheumbrellaofnaturallanguage the training data which could result in poor performance.
0 understanding (NLU), which are usually solved by label- Second,someneuralnetworkslikeRNNorLSTMhavethe
4 ing meaningful chunks in a sentence. This kind of task is ability to encode context information but don’t treat each
0
usually treated as a sequence labeling problem, where ev- chunk as a complete unit. If we can eliminate this draw-
.
1 ery word in a sentence is assigned an IOB-based (Inside- back, it could result in more accurate labeling, especially
0 Outside-Beginning)label.Forexample,inFigure 1,inthe formulti-wordchunks.
7 sentence“Butitcouldbemuchworse”welabel“could”as
1 Sequencechunkingisanaturalsolutiontoovercomethe
B-VP,“be”asI-VP,and“it”asB-NP,while“But”belongs
: twodrawbacksmentionedbefore.Insequencechunking,the
v toanartificialclassO.Thislabelingindicatesthata chunk
originalsequencelabelingtaskisdividedintotwosub-tasks:
i “could be” is a verb phrase (VP) where the label prefix B
X (1)Segmentation,toidentifyscopeofthechunksexplicitly;
meansthebeginningwordofthechunk,whileIreferstothe
r other words within the same semantic chunk; and “it” is a (2) Labeling, to label each chunk as a single unit based on
a thesegmentationresults.
single-wordchunkwithNPlabel.
Such sequence labeling forms the basis for many recent
Lample et al. (2016) used a stack-LSTM (Dyer et al.
deep network based approaches, e.g., convolutional neural
2015)andatransition-basedalgorithmforsequencechunk-
networks (CNN), recurrent neural networks (RNN) or its
ing. In their paper, the segmentation step is based on shift-
variation,longshort-termmemorynetworks(LSTM).RNN
reduce parser based actions. In this paper, we propose an
and LSTM are good at capturing sequential information
alternativeapproachbyrelyingonlyontheneuralarchitec-
(Yao et al. 2013; Huang, Xu, and Yu 2015; Mesnil et al.
tures for NLU. We investigate two different ways for seg-
2015; Peng and Yao 2015; Yang, Salakhutdinov, and Co-
mentation: (1) using IOB labels; and (2) using pointer net-
hen 2016; Kurata et al. 2016; Zhu and Yu 2016), whereas
works (Vinyals, Fortunato, and Jaitly 2015) and propose
CNNcanextracteffectivefeaturesforclassification(Xuand
three neural sequence chunking models. Pointer network
Sarikaya2013;Vu2016).
performsbetterthanthemodelusingIOB.Inaddition,italso
Copyright(cid:13)c 2017,AssociationfortheAdvancementofArtificial achievesstate-of-the-artperformanceonbothtextchunking
Intelligence(www.aaai.org).Allrightsreserved. andslotfillingtasks.
BasicNeuralNetworks NP VP ADJP
RecurrentNeuralNetwork � : <8 : <8
O B B I B I
Recurrentneuralnetwork(RNN)isaneuralnetworkthatis
suitableformodelingsequentialinformation.Althoughthe- But it could be much worse
oretically it is able to capture long-distance dependencies,
in practice it suffers from the gradient vanishing/exploding
Figure2:ModelI:SingleBi-LSTMmodelforbothsegmen-
problems(Bengio,Simard,andFrasconi1994).Longshort-
tationandlabelingsubtasks.
term memory networks (LSTM) were introduced to cope
with these gradient problems and model long-range de-
pendencies (Hochreiter and Schmidhuber 1997) by using a
memorycell.Givenaninputsentencex = (x1,x2,...,xT) ModelI
where T is the sentence length, LSTM hidden state at
For segmentation, the most straightforward and intuitive
timesteptiscomputedby:
wayistotransformitintoasequencelabelingproblemwith
i =σ(Wix +Uih +bi)
t t t−1 3classes:I-inside,O-outside,B-beginning;andthenun-
f =σ(Wfx +Ufh +bf) derstandthescopeofthechunksfromtheselabels.Building
t t t−1
o =σ(Wox +Uoh +bo) onthis,weproposeModelI,whichisaBi-LSTMasshown
t t t−1 (1) inFigure2.Inthemodel,wetakethebi-LSTMhiddenstates
g =tanh(Wgx +Ugh +bg)
t t t−1 generatedbyFormula(2)asfeaturesforbothsegmentation
c =f (cid:12)c +i (cid:12)g andlabeling.
t t t−1 t t
h =o (cid:12)tanh(c ) Forexample,wefirstclassifyeachwordintoanIOBlabel
t t t
as shown in (Figure 2). Suppose a chunk begins at word i
whereσ(·)andtanh(·)aretheelement-wisesigmoidandhy-
with length l (with one B label and followed by (l − 1) I
perbolic tangent functions, (cid:12) is the element-wise multipli-
labels),thenwecancomputeafeaturevectorforachunkas
cationoperator,andi ,f ,o aretheinput,forgetandoutput
t t t follows:
gates. h and c are the hidden state and memory cell
t−1 t−1
of previous timestep respectively. To simplify the notation, ←→ ←−→ ←−−→
weusexttodenoteboththewordanditsembedding. Chj =Average(hi,hi+1,...,hi+l−1) (3)
The bi-directional LSTM (Bi-LSTM), a modification of
where j is the chunk index of the sentence, and
theLSTM,consistsofaforwardandabackwardLSTM.The
Average(·)computestheaverageoftheinputvectors.With
forwardLSTMreadstheinputsentenceasitis(fromx to
−→ −→ −→1 Chj,weapplyasoftmaxlayeroverallchunklabelsforla-
xT)andcomputestheforwardhiddenstates(h1,h2,...,hT), beling.ForexampleinFigure2,“muchworse”isidentified
whilethebackwardLSTMreadsthesentenceinthereverse as a chunk with length 2; and we apply Formula (3) on its
o←r−der←−(from←−xT to x1), and creates backward hidden states hiddenstates,tofinallygetthe“ADJP”label.
(h ,h ,...,h ). Then for each timestep t, the hidden state
1 1 T →− ←−
ModelII
oftheBi-LSTMisgeneratedbyconcatenatingh andh ,
t t
←→ →− ←−
A drawback of Model I is that a single Bi-LSTM may not
h =[h ;h ] (2)
t t t
perform well on both segmentation and labeling subtasks.
ConvolutionalNeuralNetwork To overcome this we propose Model II, which follows the
encoder-decoderframework(Figure 3)(Sutskever,Vinyals,
Convolutional Neural Networks (CNN) have been used to
andLe2014;Bahdanau,Cho,andBengio2014).Similarto
extract features for sentence classification (Kim 2014; Ma
ModelI,weemployaBi-LSTMforsegmentationwithIOB
et al. 2015; dos Santos, Xiang, and Zhou 2015). Given a
labels1. This Bi-LSTM will also serve as an encoder and
sentence,aCNNwithmfiltersandafiltersizeofnextracts −→ ←−
create a sentence representation [h ;h ] (by concatenating
a m-dimension feature vector from every n-gram phrase of T 1
the final hidden state of the forward and backward LSTM)
thesentence.Amax-over-timepooling(max-pooling)layer
whichisusedtoinitializethedecoderLSTM.
isappliedoverallextractedfeaturevectorstocreatethefinal
We modify the general encoder-decoder framework and
indicativefeaturevector(m-dimension)forthesentence.
use chunks as the inputs instead of words. For example,
Following this approach, we use CNN and max-pooling
muchworseisachunkinFigure3,andwetakeitasasin-
layer to extract features from chunks. For each identified
gleinputtothedecoder.TheinputchunkrepresentationC
chunk, we first apply CNN to the embedding of its words j
consistsofseveralparts.WefirstusetheCNNMaxlayerto
(irrespectiveofitbeingasingle-wordchunkorchunk),and
extractimportantinformationfromwordsinsidethechunk:
thenusethemax-poolinglayerontoptogetthechunkfea-
turevectorforlabeling.WeuseCNNMaxtodenotethetwo
Cx =g(x ,x ,...,x ) (4)
layershereafter. j i i+1 i+l−1
1NotethatinModelIandII,wecannotguaranteethatlabel“O”
ProposedModels
isnotfollowedby“I”duringsegmentation.Ifso,wejusttakethe
Inthissection,weintroducethedifferentneuralmodelsfor first“I”as“B”.Infutureworkitisadvisabletoaddthatasahard
sequencechunkinganddiscussthefinallearningobjective. constraint.
O B B I B I
But it could be much worse
<s> But it could be much worse
Figure3: ModelII: Encoder-decoderframework.TheencoderBi-LSTMisusedforsegmentationandthedecoderLSTMis
usedforlabeling.
←→
where g(·) is the CNNMax layer. Then we use the context state h , the ending point candidate word embedding x ,
i i
word embeddingsof the chunkto capture context informa- togetherwithcurrentbeginningwordembeddingx andde-
b
tion(Yaoetal.2013;Mesniletal.2015;Kurataetal.2016). coder hidden state d as features. We also use the chunk
j
The context window size is a hyperparameter to tune. Fi- lengthembedding,LE(i−b+1),asthechunklevelfeature.
nally, we average the hidden states from the encoder Bi- W ,W ,W ,W ,v ,v and LE arealllearnable parameters.
1 2 3 4 1 2
LSTMbyFormula(3).Byusingthesethreeparts,weextract Thentheprobabilityofchoosingendingpointcandidateiis:
different useful information for labeling, and import them
allintothedecoderLSTM.Thus,thedecoderLSTMhidden p(i)= exp(uij) (7)
stateisupdatedby: b+lm−1exp(uk)
k=b j
h =LSTM(Cx ,Ch ,Cw ,h ,c ) (5)
j j j j j−1 j−1 Weusethisprobabilityt(cid:80)oidentifythescopeofchunks.For
example, suppose we just identified word it as a one word
hereCw istheconcatenationofcontextwordembeddings.
j
chunk with label NP in Figure 4. Following the line emit-
Notethatthecomputationofhiddenstateshereissimilarto
ted from it, we will need to decide the ending point of the
Formula (1), the only difference is that here we have three
nextchunk(thebeginningpointisobviouslythewordcould
inputs {Cx ,Ch ,Cw }. The generated hidden states are
j j j
after it). With the maximum chunk length 2, we have two
finallyusedforlabelingbyasoftmaxlayer.
choices, one is to stop at word could and gets a one word
ModelIII chunkcould,andtheotheristostopatwordbeandgener-
atesatwowordchunkcouldbe.Fromthefigure,wecansee
There are two drawbacks of using IOB labels for segmen-
thatthemodelselectsthesecondcase(redcirclepart),and
tation. First, it is hard to use chunk-level features for seg-
createsatwowordchunk.Thischunkwillserveastheinput
mentation,likethelengthofchunks.Also,usingIOBlabels
ofthenextdecodertimestep.Thedecoderhiddenstatesare
cannot compare different chunks directly. The shift-reduce
updatedsimilartoModelII(Equation5).
algorithm used in (Lample et al. 2016) has the same issue.
Theybothtransformamulti-classclassificationproblem(we LearningObjective
couldhavealotofchunkcandidates)intoa3-classclassifi-
Aswedescribedabove,alltheaforementionedmodelssolve
cationproblem,inwhichthechunksareinferredimplicitly.
twosubtasks-segmentationandlabeling.Weusethecross-
To resolve this problem, we further propose Model III,
entropylossfunctionforboththetwosubtasks,andsumthe
which is an encoder-decoder-pointer framework (Figure 4)
twolossestoformthethelearningobjective:
(Nallapatietal.2016).ModelIIIissimilartoModelII,the
onlydifferencebeingthemethodofidentifyingchunks.
L(θ)=L (θ)+L (θ) (8)
segmentation labeling
ModelIIIisagreedyprocessofsegmentationandlabel-
ing,wherewefirstidentifyonechunk,andthenlabelit.This whereθdenotesthelearnableparameters.Alternatively,we
processisrepeateduntilallthewordsareprocessed.Asall could also use weighted sum, or do multi-task learning by
chunksareadjacenttoeachother2,afteronechunkisiden- consideringsegmentationandlabelingasthetwotasks.We
tified,thebeginningpointofthenextoneisalsoknown,and leavetheseextensionsasfuturework.
onlyitsendingpointistobedetermined.Weadoptpointer
network(Vinyals,Fortunato,andJaitly2015)todothis.For Experiments
apossiblechunkbeginningattimestepb,wefirstgeneratea
ExperimentalSetup
featurevectorforeachpossibleendingpointcandidatei:
Weconductexperimentsontextchunkingandsemanticslot
←→
ui =vTtanh(W h +W x +W x +W d ) fillingrespectivelytotesttheperformanceoftheneuralse-
j 1 1 i 2 i 3 b 4 j (6) quence chunking models we propose in this paper. Both
+ vTLE(i−b+1) i∈[b,b+l )
2 m these tasks identify the meaningful chunks in the sentence,
where j is the decoder timestep (i.e., chunk index), l is such as the noun phrase (NP), or the verb phrase (VP) for
m
textchunkinginFigure1,andthe“depart city”forslotfill-
the maximum chunk length. We use the encoder hidden
ingtaskinFigure5.
2Hereaswedon’tknowthelabelofeachchunkduringsegmen- We use the CoNLL 2000 shared task (Tjong Kim Sang
tation,weneedtofeedallthechunkstothedecoderforlabeling. and Buchholz 2000) dataset for text chunking. It contains
length: 2
length: 2
length: 1
length: 1
But it could be much worse
<s> But it could be much worse
Figure4: ModelIII:Encoder-decoder-pointerframework:SegmentationisdonebyapointernetworkandadecoderLSTMis
usedforlabeling.
O O B-depart_city I-depart_city O B-arrival_city {I,O,B}. To compute the segmetnation F1-score, we delete
thecontent labelforeach word,forexample, ifaword has
alabel“B-VP”,wewilldelete“VP”andtheleft“B”isused
flights from San Diego to Boston
forsegmentationF1-score.
Forthetwotasks,weusehiddenstatesizeas100forthe
Figure5: AnexampleofsemanticslotfillingusingtheIOB
forwardandbackwardLSTMrespectivelyinBi-LSTM,and
scheme. “San Deigo” is a multi-word chunk with label
size200fortheLSTMdecoder.Weusedropoutwithrate0.5
“depart city”.
onboththeinputandoutputofallLSTMs.Themini-batch
size is set to 1. The number of training epochs are limited
to 200 for text chunking, and 100 for slot filling. 4 For the
8,936trainingand893testsentences.Thereare12different CNNusedinModelIIandIIIonextractingchunkfeatures,
labels(22withIOBprefixincluded).Sinceitdoesn’thavea thefiltersizeisthesameaswordembeddingdimension,and
validationset,weholdout10%ofthetrainingdata(selected thefilterwindowsizeas2.WeadoptSGDtotrainthemodel,
atrandom)asthevalidationset. andbygridsearch,wetunetheinitiallearningratein[0.01,
Toevaluatetheeffectivenessofourmethodontheseman- 0.1],learningratedecayin[1e-6,1e-4],andcontextwindow
tic slot filling task, we use two different datasets. The first size{1,3,5}.
one is the ATIS dataset, which consists of reservation re- For the word embedding, following (Kurata et al. 2016),
questsfromtheairtraveldomain.Itcontains4,978training wedon’tusepre-trainedembeddingfortheslotfillingtask,
and 893 testing sentences in total, with a vocabulary size but use a randomly initialized embedding and tune the di-
of 572. There are 84 different slot labels (127 if with IOB mension in {30,50,75} by grid search. For text chunk-
prefix). We randomly selected 80% of the training data for ing, we concatenate two different embeddings. The first is
modeltrainingandtherest20%asthevalidationset(Mesnil SENNA embedding (Collobert et al. 2011) with dimension
etal.2015).Followingtheworkof(Kurataetal.2016),we 50.5 Theotherisawordrepresentationgeneratedbasedon
alsousealargerdatasetbycombiningtheATIScorpuswith itscomposedcharacters.weadoptaCNNontotherandomly
theMITRestaurantCorpusandMITMovieCorpus(Liuet initialized character embeddings, with 30 filters and filter
al.2013a;Liuetal.2013b).Thisdatasethas30,229training windowsize3.
and6,810testinginstances.Similartothepreviousdataset,
weuse80%ofthetraininginstancesfortrainingthemodel, TextChunkingResults
and treat the rest 20% as a validation set. This dataset has Results on the text chunking task are shown in Table 1. In
a vocabulary size of 16,049 and the number of slot labels this,the“baseline(Bi-LSTM)”referstoaBi-LSTMmodel
is116(191withIOBprefixincluded).Sincethisdatasetis forsequencelabeling(useIOB-basedlabelsonwordsasin
considerablylargerandincludes3differentdomains,weuse Figure1).“F1”isthefinalevaluationmetric,and“segment-
“LARGE”todenoteithereafter. F1” refers to the segmentation F1-score. From the table,
The final performance is measured in terms of F1-score, we can see that Model I and Model II only have compa-
computed by the public available script conlleval.pl 3. We rable results with the baseline on both evaluation metrics -
report the F1-score on the test set with parameters that segment-F1 and final F1 score. Hence, we infer that using
achievesthebestF1-scoreonthevalidationset.Towardsthe IOBlabelstodosegmentationindependentlymightnotbea
neuralsequencechunkingmodels,afterwegetthelabelfor good choice. However, Model III outperforms the baseline
eachchunk,wewillassigneachofitswordanIOB-basedla- onbothsegmentationandlabeling.
belaccordinglysothatthescriptcandoevaluation.Wealso Wefurthercompareourbestresultwiththecurrentpub-
reportthesegmentationF1-scoretoassessthesegmentation lishedresultsinTable2.Inthetable,(Collobertetal.2011)
performance of different models. This is also computed by
the conlleval.pl script, but only considers three labels, i.e. 4We found that while 100 epochs are enough for slot filling
modeltoconverge,weneed200fortextchunking.
3http://www.cnts.ua.ac.be/conll2000/chunking/ 5http://ronan.collobert.com/senna/
F1 Segment-F1 ATIS LARGE
Train Test Train Test
baseline(Bi-LSTM) 94.13 95.28
1 10275(77.7%) 2096(73.9%) 28511(46.8%) 7283(42.8%)
ModelI 94.01 95.09 2 2726(20.6%) 659(23.2%) 20679(34.0%) 6214(36.5%)
ModelII 94.13 95.22 >=3 224(1.7%) 82(2.9%) 11694(19.2%) 3516(20.7%)
ModelIII 94.72 95.75
Table4: Statisticsonthelengthofchunks:Thefirstcolumn
Table1:Textchunkingresultsofourneuralsequencechunk- denoteschunk-lengths.Forexample,firstcellindicatesthat
ingmodels. thereare10275chunksoflength1,andaccountsfor77.7%
ofallATISchunks.
is the first work of using neural networks for text chunk-
Moreover, Model III gets the best segmentation perfor-
ing. Huang, Xu, and Yu used a BiLSTM-CRF framework
mance over all the models (99.01% and 82.44%), confirm-
together with a lot of handcraft features. Yang, Salakhutdi-
ingthatourpointernetworkinmodelIIIisgoodatthistask.
nov,andCohenextendthisframeworkandemployaGRUto
However,ModelIandIIarecomparabletobaselineonthe
incorporate the character information of words, rather than
easy ATIS data, and are about 1% worse on LARGE data.
usinghandcraftedfeatures.Toourbestknowledge,theygot
the current best results 94.66 on the text chunking task. 6 Thisfurtherconfirmsouranalysisontextchunkingexperi-
mentsthatusingIOBlabelsaloneforsegmentation,(likein
Different from previous work, we model the segmentation
ModelIandII)cannotgiveusagoodresult.
partexplicitlyinourneuralmodels,andwithoutusingCRF,
wegetastate-of-the-artperformanceof94.72.
ATIS LARGE
1 2 >=3 1 2 >=3
Methods F1-score Baseline(Bi-LSTM) 98.90 98.70 98.78 86.25 88.48 54.88
SVMClassifier(KudohandMatsumoto2000) 93.48 ModelI 98.95 98.78 99.39 85.91 87.27 53.30
ModelII 98.83 98.78 98.78 86.42 87.29 52.98
SVMClassifier(KudoandMatsumoto2001) 93.91
ModelIII 99.00 98.93 100.0 89.01 88.69 56.59
SecondorderCRF(ShaandPereira2003) 94.30
HMM+votingscheme(ShenandSarkar2005) 94.01
Table5: Segment-F1ondifferentchunk-lengths.
Convnetworktagger(senna)(Collobertetal.2011) 94.32
BiLSTM-CRF(Huang,Xu,andYu2015) 94.46
BiGRU-CRF(Yang,Salakhutdinov,andCohen2016)7 94.66 Wefurtherinvestigatethesegmentationprocessandshow
thesegmentationF1-scoreondifferentchunklengthsinTa-
ModelIII(Ours) 94.72
ble5.Theresultsdemonstratethatthepoorperformanceon
LARGEdataismainlyduetothebadperformanceoniden-
Table2: ComparisonwithpublishedresultsontheCoNLL
tifyinglongchunks(around55%).OurModelIIIimproves
chunkingdataset.
thisscoreby 2%overbaseline(54.88%vs.56.59%).Asthe
absolute performance on this subset is still low, future re-
searcheffortsshouldfocusonimprovingthisperformance.
SlotFillingResults
Inaddition,ModelIandIIgetcomparablesegmentationre-
sults with the baseline model on one-words chunks, while
ATIS LARGE beingworseonlongerchunks,furthersupportingthisanal-
F1 Segment-F1 F1 Segment-F1 ysis.
baseline(Bi-LSTM) 95.23 98.85 75.73 80.79
ModelI 95.25 98.92 76.68 79.93 LabelingResults FromTable3,weobservethatModelIII
ModelII 95.71 98.82 77.26 79.99 hasthebestF1scoreascomparedtothebaselineandother
ModelIII 95.86 99.01 78.49 82.44 neuralchunkingmodels.AnotherobservationisthatModel
IandIIgetbetterimprovementsoverbaselineeventhough
Table3:Mainresultsofourneuralsequencechunkingmod-
theyarepooratsegmentationinslotfillingtask.
elsonslotfillingtask.
ATIS LARGE
SegmentationResults FromtheTable3,wecanseethat 1 2 >=3 1 2 >=3
thesegment-F1 scoreonATIS dataismuch betterthan the baseline(Bi-LSTM) 95.37 96.03 85.19 79.21 83.56 53.36
ModelI 95.23 96.18 88.48 83.10 82.35 51.76
one on LARGE data (∼99% vs. ∼80%). This is because
ModelII 95.87 96.18 87.80 85.01 82.82 51.15
theATISdataismucheasierforsegmentationthanLARGE ModelIII 95.89 96.19 92.68 84.97 83.89 54.38
data.AsshowninTable4,morethan97%ofthechunksin
ATIS data have only one or two words, while the LARGE Table6: F1-scoresfordifferentchunk-lengths
data has much longer chunks. Also, compared to the small
ATIS vocabulary (572 words), it is harder to learn a good Table 6 gives some insights on this by showing the F1-
segmentation model with a more complicated vocabulary score on different chunk-lengths. Comparing Table 5 and
(about16kwords)inLARGEdata. 6, we can see when Model I and II achieve comparable
segment-F1withbaseline,andtheF-1scoresarehigher.For
6Theyalsogetaperformanceof95.41,butthisnumberisfrom slotfillingtask,thejointlearningframework(Formula(8))
jointtraining,whichneedsthetrainingdataofothertasks. helpslabelingwhileharmssegmentationonmodelIandII.
Moreover,theusageofencoderinModelIIcouldalsohelp allyuseaBiLSTMorBiGRUasthemajorlabelingarchitec-
labelinginthistask(Kurataetal.2016).Finally,ourModel ture,andaLSTMorGRUorCNNtocapturethecharacter-
IIIcouldachievebetterF1scoreonallchunklengths. level information, and finally a CRF layer to model the la-
bel dependency (Lample et al. 2016; Ma and Hovy 2016;
Comparison with Published Results We compare the
Yang,Salakhutdinov,andCohen2016).
ATISresultsofourbestmodel(ModelIII)withcurrentpub-
Inaddition,manysimilarworkshavealsobeenexplored
lished results in Table 7. As shown in the table, many re-
for slot filling, like RNN (Yao et al. 2013; Mesnil et al.
searchers have done a lot of work which uses deep neural
2015), LSTM (Yao et al. 2014; Jaech, Heck, and Osten-
networksforslotfilling.Recentworkshowstherankingloss
dorf 2016), adding external memory (Peng and Yao 2015),
ishelpful(Vuetal.2016),andaddingencoderimprovesthe
addingencoder(Kurataetal.2016),usingrankingloss(Vu
score to 95.66%. The best published result in the table is
etal.2016),addingattention(ZhuandYu2016)andsoon.
from(ZhuandYu2016),whichis95.79%.Comparedwith
In the other direction, people also developed neural net-
previousresults,ourModelIIIgetsthestate-of-the-artper-
workstohelptraditionalsequenceprocessingmethods,like
formance95.86%.
CRFparsing(DurrettandKlein2015)andweightedfinite-
statetransducer(Rastogi,Cotterell,andEisner2016).
Methods F1-score
RNN(Yaoetal.2013) 94.11
Conclusion
CNN-CRF(XuandSarikaya2013) 94.35
Bi-RNN(Mesniletal.2015) 94.73 In this paper, we presented three different models for se-
LSTM(Yaoetal.2014) 94.85 quence chunking. Our experiments show that the segmen-
RNN-SOP(LiuandLane2015) 94.89 tation results of Model I and Model II are comparable to
DeepLSTM(Yaoetal.2014) 95.08 baseline on text chunking data and ATIS data, and worse
RNN-EM(PengandYao2015) 95.25 than the baseline on LARGE data, while Model III gains
Bi-RNNwithrankingloss(Vuetal.2016) 95.56
higher segment-F1 score than baseline, demonstrating that
SequentialCNN(Vu2016) 95.61
the use of IOB labels is not suitable for building segmen-
Encoder-labelerDeepLSTM(Kurataetal.2016) 95.66
tation models independently. Moreover, Model I and II do
BiLSTM-LSTM(focus)(ZhuandYu2016) 95.79
notgiveconsistentimprovementsonthefinalF1score-the
ModelIII(Ours) 95.86
segmentation step improves labeling on slot filling, but not
on the text chunking task. Finally, Model III consistently
Table 7: Comparison with published results on the ATIS
performs better than baseline and gets state-of-the-art per-
data
formanceonthetwotasks.Wealsogaininsightsaboutthe
datasetsweusebycomparingthesegment-F1scoresandF1
We compare our approach against the only set of pub-
scores of model III. For the text chunking data (95.75 vs.
lishedresultsontheLARGEdatafrom(Kurataetal.2016),
94.72) and LARGE data (82.44 vs. 78.49), the scores are
against which we compare our approach. The reported F1
close to each other, indicating that segmentation is a major
score on this dataset by their encoder-decoder model is
challenge in these two datasets compared to labeling. But
74.41, and our best model achieves a score of 78.49 which
for ATIS data (99.01 vs. 95.86), the segmentation score is
issignificantlyhigher.
almost 100 percent, so labeling seems like the main chal-
lengeinthisdataset.Wehopethisinsightencouragesmore
RelatedWork
research efforts on the similar tasks. Finally, the proposed
In recent years, many deep learning approaches have been neural sequence chunking models achieves state-of-the-art
explored for resolving the sequence labeling tasks. (Col- performanceonbothtextchunkingandslotfilling.
lobert et al. 2011) proposed an effective window-based ap-
proach, in which they used a feed-forward neural network References
to classify each word and conditional random fields (CRF)
[Bahdanau,Cho,andBengio2014] Bahdanau, D.; Cho, K.;
tocapturethesequentialinformation.CNNsarealsowidely
and Bengio, Y. 2014. Neural machine translation by
used for extracting effective classification features (Xu and
jointly learning to align and translate. arXiv preprint
Sarikaya2013;Vu2016).
arXiv:1409.0473.
RNNs are a straightforward and better suited choice for
[Bengio,Simard,andFrasconi1994] Bengio,Y.;Simard,P.;
these tasks as they model sequential information. (Huang,
and Frasconi, P. 1994. Learning long-term dependencies
Xu, and Yu 2015) presented a BiLSTM-CRF model, and
withgradientdescentisdifficult. IEEEtransactionsonneu-
achieved state-of-the-art performance on several tasks, like
ralnetworks5(2):157–166.
namedentityrecognitionandtextchunkingwiththehelpof
handcrafted features. (Chiu and Nichols 2015) used a BiL- [ChiuandNichols2015] Chiu, J. P., and Nichols, E. 2015.
STM for labeling and a CNN to capture character-level in- Named entity recognition with bidirectional lstm-cnns.
formation,like(dosSantosandGatti2014)andadditionally arXivpreprintarXiv:1511.08308.
usedhandcraftedfeaturestogaingoodperformance. Many [Collobertetal.2011] Collobert, R.; Weston, J.; Bottou, L.;
works have then been investigated to combine the advan- Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. 2011. Natu-
tages of the above two works and achieved state-of-the-art ral language processing (almost) from scratch. Journal of
performancewithouthandcraftedfeatures.Theseworksusu- MachineLearningResearch12(Aug):2493–2537.
[dosSantosandGatti2014] dosSantos,C.N.,andGatti,M. [MaandHovy2016] Ma, X., and Hovy, E. 2016. End-to-
2014. Deep convolutional neural networks for sentiment endsequencelabelingviabi-directionallstm-cnns-crf.arXiv
analysisofshorttexts. InCOLING,69–78. preprintarXiv:1603.01354.
[dosSantos,Xiang,andZhou2015] dos Santos, C.; Xiang, [Maetal.2015] Ma,M.;Huang,L.;Xiang,B.;andZhou,B.
B.; and Zhou, B. 2015. Classifying relations by ranking 2015. Dependency-basedconvolutionalneuralnetworksfor
withconvolutionalneuralnetworks. InACL,626–634. Bei- sentenceembedding. InACL,volume2,174–179.
jing,China:AssociationforComputationalLinguistics.
[Mesniletal.2015] Mesnil, G.; Dauphin, Y.; Yao, K.; Ben-
[DurrettandKlein2015] Durrett, G., and Klein, D. 2015. gio, Y.; Deng, L.; Hakkani-Tur, D.; He, X.; Heck, L.; Tur,
Neuralcrfparsing. InProceedingsofACL2015,302–312. G.;Yu,D.;etal. 2015. Usingrecurrentneuralnetworksfor
slot filling in spoken language understanding. IEEE/ACM
[Dyeretal.2015] Dyer, C.; Ballesteros, M.; Ling, W.;
Transactions on Audio, Speech, and Language Processing
Matthews, A.; and Smith, N. A. 2015. Transition-based
23(3):530–539.
dependencyparsingwithstacklongshort-termmemory. In
ProceedingsofACL2015,334–343. [Nallapatietal.2016] Nallapati, R.; Zhou, B.; dos Santos,
C.;Gulcehre,C.;andXiang,B. 2016. Abstractivetextsum-
[HochreiterandSchmidhuber1997] Hochreiter, S., and
marizationusingsequence-to-sequencernnsandbeyond. In
Schmidhuber, J. 1997. Long short-term memory. Neural
ProceedingsofCoNLL.
computation9(8):1735–1780.
[PengandYao2015] Peng,B.,andYao,K. 2015. Recurrent
[Huang,Xu,andYu2015] Huang, Z.; Xu, W.; and Yu, K.
neuralnetworkswithexternalmemoryforlanguageunder-
2015. Bidirectional lstm-crf models for sequence tagging.
standing. arXivpreprintarXiv:1506.00195.
arXivpreprintarXiv:1508.01991.
[Rastogi,Cotterell,andEisner2016] Rastogi, P.; Cotterell,
[Jaech,Heck,andOstendorf2016] Jaech, A.; Heck, L.; and
R.; and Eisner, J. 2016. Weighting finite-state transduc-
Ostendorf,M. 2016. Domainadaptationofrecurrentneural
tions with neural context. In Proceedings of NAACL 2016,
networksfornaturallanguageunderstanding.arXivpreprint
623–633.
arXiv:1604.00117.
[ShaandPereira2003] Sha,F.,andPereira,F.2003.Shallow
[Kim2014] Kim, Y. 2014. Convolutional neural networks
parsing with conditional random fields. In Proceedings of
forsentenceclassification. arXivpreprintarXiv:1408.5882.
the2003ConferenceoftheNorthAmericanChapterofthe
[KudoandMatsumoto2001] Kudo, T., and Matsumoto, Y. ACL on Human Language Technology-Volume 1, 134–141.
2001. Chunkingwithsupportvectormachines. InProceed- AssociationforComputationalLinguistics.
ings of NAACL 2001, 1–8. Association for Computational
[ShenandSarkar2005] Shen,H.,andSarkar,A. 2005. Vot-
Linguistics.
ingbetweenmultipledatarepresentationsfortextchunking.
[KudohandMatsumoto2000] Kudoh, T., and Matsumoto, In Conference of the Canadian Society for Computational
Y. 2000. Use of support vector learning for chunk iden- StudiesofIntelligence,389–400. Springer.
tification. In Proceedings of the 2nd Workshop on Learn- [Sutskever,Vinyals,andLe2014] Sutskever,I.;Vinyals,O.;
ingLanguageinLogicandthe4thConferenceonComputa- and Le, Q. V. 2014. Sequence to sequence learning with
tionalNaturalLanguageLearning-Volume7,ConLL’00, neuralnetworks. InNIPS,3104–3112.
142–144.
[TjongKimSangandBuchholz2000] Tjong Kim Sang,
[Kurataetal.2016] Kurata, G.; Xiang, B.; Zhou, B.; and E.F.,andBuchholz,S. 2000. Introductiontotheconll-2000
Yu, M. 2016. Leveraging sentence-level information sharedtask:Chunking. InProceedingsofthe2ndworkshop
with encoder lstm for semantic slot filling. arXiv preprint on Learning language in logic and the 4th conference
arXiv:1601.01530. on Computational natural language learning-Volume 7,
[Lampleetal.2016] Lample, G.; Ballesteros, M.; 127–132. AssociationforComputationalLinguistics.
Kawakami, K.; Subramanian, S.; and Dyer, C. 2016. [Vinyals,Fortunato,andJaitly2015] Vinyals,O.;Fortunato,
Neural architectures for named entity recognition. In In M.;andJaitly,N. 2015. Pointernetworks. InNIPS,2692–
proceedingsofNAACL2016. 2700.
[LiuandLane2015] Liu, B., and Lane, I. 2015. Recurrent [Vuetal.2016] Vu,N.T.;Gupta,P.;Adel,H.;Sch,H.;etal.
neuralnetworkstructuredoutputpredictionforspokenlan- 2016. Bi-directionalrecurrentneuralnetworkwithranking
guageunderstanding. InProc.NIPSWorkshoponMachine lossforspokenlanguageunderstanding. InICASSP,6060–
LearningforSpokenLanguageUnderstandingandInterac- 6064. IEEE.
tions.
[Vu2016] Vu, N. T. 2016. Sequential convolutional neural
[Liuetal.2013a] Liu,J.;Pasupat,P.;Cyphers,S.;andGlass, networks for slot filling in spoken language understanding.
J. 2013a. Asgard: A portable architecture for multilingual arXivpreprintarXiv:1606.07783.
dialoguesystems. InICASSP. IEEE.
[XuandSarikaya2013] Xu,P.,andSarikaya,R. 2013. Con-
[Liuetal.2013b] Liu,J.;Pasupat,P.;Wang,Y.;Cyphers,S.; volutionalneuralnetworkbasedtriangularcrfforjointintent
andGlass,J. 2013b. Queryunderstandingenhancedbyhi- detectionandslotfilling.InProceedingsofASRU2013,78–
erarchicalparsingstructures. InASRU,72–77. IEEE. 83. IEEE.
[Yang,Salakhutdinov,andCohen2016] Yang, Z.; Salakhut-
dinov, R.; and Cohen, W. 2016. Multi-task cross-
lingual sequence tagging from scratch. arXiv preprint
arXiv:1603.06270.
[Yaoetal.2013] Yao,K.;Zweig,G.;Hwang,M.-Y.;Shi,Y.;
and Yu, D. 2013. Recurrent neural networks for language
understanding. InINTERSPEECH,2524–2528.
[Yaoetal.2014] Yao, K.; Peng, B.; Zhang, Y.; Yu, D.;
Zweig, G.; and Shi, Y. 2014. Spoken language under-
standingusinglongshort-termmemoryneuralnetworks. In
SpokenLanguageTechnologyWorkshop(SLT),2014IEEE,
189–194. IEEE.
[ZhuandYu2016] Zhu, S., and Yu, K. 2016. Encoder-
decoder with focus-mechanism for sequence labelling
based spoken language understanding. arXiv preprint
arXiv:1608.02097.