Graph-Based Semi-Supervised Conditional Random Fields For Spoken Language Understanding Using Unaligned Data

0.08 MB·

by  Mohammad Aliannejadi

Graph-Based Semi-Supervised Conditional Random Fields For Spoken
Language Understanding Using Unaligned Data
MohammadAliannejadi MasoudKiaeeha Shahram Khadivi&
AmirkabirUniversity Sharif Universityof Saeed Shiry Ghidary
ofTechnology Technology AmirkabirUniversity
(Tehran Polytechnic) [email protected] ofTechnology
[email protected] (Tehran Polytechnic)
{khadivi, shiry}@aut.ac.ir
Abstract modeled the problem using a push-down
automaton. There are other works which
We experiment graph-based Semi- have dealt with SLU as a sequential labeling
7 SupervisedLearning(SSL)ofConditional problem. RaymondandRiccardi(2007) and
1 Random Fields (CRF) for the application WangandAcero(2006) have fully annotated
0
2 of Spoken Language Understanding the data and trained the model in discriminative
(SLU) on unaligned data. The aligned frameworks such as CRF. CRF captures many
n
a labels for examples are obtained using complex dependencies and models the sequential
J
IBM Model. We adapt a baseline semi- relations between the labels; therefore, it is a
0
supervised CRF by defining new feature powerfulframeworkforSLU.
3
set and altering the label propagation al- TheSemi-SupervisedLearning(SSL)approach

] gorithm. Our results demonstrate that our
L has drawn a raft of interest among the machine
proposed approach significantly improves
C learning community basically because ofits prac-
the performance of the supervised model
. tical application (Chapelleetal.,2006). Manual
s
by utilizing the knowledge gained from
c tagging of data can take considerable effort and
[ thegraph.
time; however, in the training phase of SSL, a

1 largeamountofunlabeled dataalongwithasmall
v 1 Introduction
amount of labeled data is provided. This makes it
3
3 The aim of Spoken Language Understanding morepracticable andcosteffectivethanproviding
5
(SLU)istointerpret theintention oftheuser’s ut- a fully labeled set of training data; thus, SSL is
8
0 terance. Morespecifically,aSLUsystemattempts morefavorable.
1. to find a mapping from user’s utterance in natu- Graph-based SSL, the most active area
0 ral language, to the limited set of concepts that is of research in SSL in the recent years, has
7
structuredandmeaningfulforthecomputer. Asan shown to outperform other SSL methods
1
: example,forthesampleutterance: (Chapelleetal.,2006). Graph-based SSL al-
v
IwanttoreturntoDallasonThursday gorithms are generally run in two steps: graph
i
X
It’scorresponding outputwouldbe: construction and label propagation. Graph
r
a GOAL : RETURN construction is the most important step in graph-
TOLOC.CITY = Dallas based SSL; and, the fundamental approach is
RETURN.DATE = Thursday. to assign labeled and unlabeled examples to
SLUcanbewidelyusedinmanyrealworldappli- nodes of the graph. Then, a similarity function
cations; however, data processing costs may im- is applied to compute similarity between pairs
pede practicability of it. Thus, attempting to train of nodes. The computed similarities are then
aSLUmodelusinglesstrainingdataisakeyissue. assigned as the weight of the edges connecting
The first statistical SLU system was based the nodes (Zhuetal.,2003). Label propaga-
on hidden Markov model and modeled using a tion operates on the constructed graph. Based
finite state semantic tagger employed in AT&T’s on the constraints or properties derived from
CHRONUS system (Pieraccini etal.,1992). the graph, labels are propagated from a few
Their semantic representation was flat-concept; labeled nodes to the entire graph. These con-
but, later HeandYoung(2005) extended the straints include smoothness (Zhuetal.,2003;
representation to a hierarchical structure and Subramanyaetal.,2010; Talukdaretal.,2008;
GarretteandBaldridge, 2013), and sparsity Algorithm1Semi-SupervisedTrainingofCRF
(DasandSmith,2012;Zengetal.,2013). 1: Λ(n=0) = TrainCRF(Dl)
Labeling unaligned training data requires 2: G = BuildGraph(Dl ∪Du)
much less effort compared to aligned data 3: {r} = CalcEmpiricalDistribution(Dl)
(HeandYoung,2005). Nevertheless, unaligned 4: whilenotconverged do
data cannot beused totrain aCRFmodeldirectly 5: {m} = CalcMarginals(Du,Λn)
since CRF requires fully-annotated data. On the 6: {q}=AverageMarginals(m)
other hand, robust parameter estimation of aCRF 7: {ˆq}=LabelPropagation(q, r)
model requires a large set of training data which 8: Duv =ViterbiDecode({ˆq},Λn)
is unrealistic in many practical applications. To 9: Λn+1 = RetrainCRF(Dl ∪Duv,Λn);
overcomethisproblem, theworkinthispaper ap- 10: endwhile
pliessemi-supervisedCRFonunlabeleddata. Itis 11: ReturnfinalΛn
motivatedbythehypothesis thatdataisalignedto
labels in a monotone manner, and words appear-
where Λ∗ is the optimal parameter set of the base
ing in similar contexts tend to have same labels.
CRFmodel and kΛk2 isthesquared ℓ -norm reg-
Underthesecircumstances, wewereabletoreach 2
ularizerwhoseimpactisadjustedbyγ. Atthefirst
1.64%improvementontheF-scoreoverthesuper-
line, Λ∗ is assigned to Λ i.e. the initial pa-
visedCRFand1.38%improvementontheF-score (n=0)
rametersetofthemodel.
overtheselftrained CRF.
In the next step, the k-NN similarity graph (G)
In the following section we describe the algo-
is constructed (line 2), which will be discussed in
rithm this work is based on and our proposed al-
more detail in Section 2.3. In the third step, the
gorithm. InSection3weevaluateourworkandin
empirical label distribution (r) on the labeled data
thefinalsectionconclusions aredrawn.
is computed. The main loop of the algorithm is
then started and the execution continues until the
2 Semi-supervised Spoken Language
resultsconverge.
Understanding
Marginalprobabilityoflabels(m)arethencom-
The input data is unaligned and represented puted on the unlabeled data (Du) using Forward-
as a semantic tree, which is described in Backward algorithm with the parameters of the
(HeandYoung,2005). Thetrainingsentencesand previous CRF model (Λn), and in the next step,
their corresponding semantic trees can be aligned allthemarginallabelprobabilities ofeachtrigram
monotonically; hence, we chose IBM Model 5 areaveraged overitsoccurrences (line5and6).
(KhadiviandNey,2005) to find the best align- In label propagation (line 7), trigram marginals
ment between the words and nodes of the seman- (q) are propagated through the similarity graph
tic tree (labels). Thus, we have circumvented the using an iterative algorithm. Thus, they become
problem of unaligned data. More detailed expla- smooth. Empirical label distribution (r) serves as
nation about this process can be found in our pre- the priori label information for labeled data and
vious work (Aliannejadi etal.,2014). This data trigram marginals (q)actastheseed labels. More
is then used to train the supervised and semi- detaileddiscussion isfoundinSection2.4.
supervised CRFs. Afterwards,havingtheresultsoflabelpropaga-
tion (ˆq) and previous CRF model parameters, la-
2.1 Semi-supervisedCRF bels of the unlabeled data are estimated by com-
bining the interpolated label marginals and the
The proposed semi-supervised learning algorithm
CRFtransition potentials (line 8). Forevery word
is based on (Subramanyaetal.,2010). Here, we
position j for i indexing over sentences, interpo-
quicklyreviewthisalgorithm (Algorithm 1).
latedlabelmarginalsarecalculated asfollows:
Inthefirststep,theCRFmodelistrainedonthe
labeleddata(Dl)according to(1): pˆ(y(j) = y|x )= αp(y(j) = y|x ;Λ )
i i i i n
+(1−α)qˆ (y), (2)
l T(i,j)
Λ∗ = arΛg∈RmKinh−Xi=1logp(yi|xi;Λ)+γkΛk2i, whereT(i,j)isatrigramcenteredatpositionj of
(1) theithsentence andαistheinterpolation factor.
Description Feature specific words are marked as classes according to
the corpus’ accompanying database. As an ex-
Context x x x x x
1 2 3 4 5
ample, city names such as Dallas and Baltimore
LeftContext x x
1 2
arerepresented ascity namewhichisaclasstype.
RightContext x x
4 5 Sincetheseclassesplayanimportantroleincalcu-
CenterWordintrigram x
3 lating similarity of the nodes, IsClass feature is
CenterisClass IsClass(x3) used to determine if a given position in a context
CenterisPreposition IsPreposition(x ) isaclasstype.
3
LeftisPreposition IsPreposition(x ) Furthermore, prepositions like from and be-
2
tween are also important, e.g. when two trigrams
Table 1: Context Features used for constructing
like ”from Washington to” and ”between Dallas
thesimilaritygraph
and” are compared. The two trigrams are totally
different while both of them begin with a prepo-
In the final step, the previous CRF model pa- sition and are continued with a class. Therefore,
IsPrepositionfeaturewouldbeparticularlysuit-
rametersareregularizedusingthelabelsestimated
able to increase the similarity score of these two
fortheunlabeled data intheprevious step (line9)
trigrams. In many cases, these features have a
asfollows:
significant effect in assigning a better similarity
l score.
Λn+1 = argminh−Xlogp(yi|xi;Λn) To define a similarity measure, we compute
Λ∈RK i=1 the Pointwise Mutual Information (PMI) between
u
−η X logp(yi|xi;Λn)+γkΛk2i, (3) afellatuorcecsu.rreTnhceesPoMfIamteraigsurarme traanndsfoeramchs tohfe tihne-
i=l+1
dependence assumption into a ratio (Lin,1998;
where η is a trade-off parameter whose setting is Razmaraetal.,2013). Then, the similarity be-
discussed laterinSection3. tween two nodes is measured as the cosine dis-
tancebetweentheirPMIvectors. Wecarefullyex-
2.2 CRFFeatures
amined the similarity graph on the training data
By aligning the training data, many infor- and found out the head and tail trigrams of each
mative labels are saved which are omit- sentence which contain dummy words, make the
ted in other works (WangandAcero,2006; graph sparse. Hence, we have ignored those tri-
RaymondandRiccardi, 2007). By saving these grams.
information,thefirstorderlabeldependencyhelps
2.4 LabelPropagation
the model to predict the labels more precisely.
Therefore the model manages to predict the After statistical alignment, the training data gets
labels using less lexical features and the feature noisy. Hence, use of traditional label propagation
window that was [-4,+2] in previous works is algorithms causes an error propagation over the
reduced to [0,+2]. Using smaller feature win- whole graph and degrades the whole system per-
dow improves the generalization of the model formance. Thus,wemakeuseoftheModifiedAd-
(Aliannejadi etal.,2014). sorption(MAD)algorithm forlabelpropagation.
MAD algorithm controls the label propagation
2.3 SimilarityGraph
more strictly. This is accomplished by limiting
In our work we have considered trigrams as the theamountofinformationthatpassesfromanode
nodes of the graph and extracted features of each toanother(TalukdarandPereira,2010). Softlabel
trigram x x x according to the 5-word con- vectorsYˆ arefoundbysolvingtheunconstrained
2 3 4 v
text x x x x x it appears in. These features optimization problem in(4):
1 2 3 4 5
are carefully selected so that nodes are correctly
placed in neighborhood of the ones having simi- min Xhµ1(Yl−Yˆl)⊤S (Yl −Yˆl)
lar labels. Table 1 presents the feature set that we Yˆ l∈C
haveapplied toconstruct thesimilaritygraph. +µ Yˆ⊤L′Yˆ + µ Yˆ −R 2 , (4)
2 l l 3(cid:13) l l(cid:13) i
IsClass feature impacts the structure of the
(cid:13) (cid:13)
graph significantly. In the pre-processing phase where µ are hyper-parameters and R is the
i l
empirical label distribution over labels i.e. the %ofLabeledData
prior belief about the labeling of a node. The 10 20 30
first term of the summation is related to label Supervised CRF 86.07 87.69 88.64
score injection from the initial score of the node Self-trained CRF 86.34 87.73 88.64
and makes the output match the seed labels Y Semi-supervised CRF 87.72 88.75 89.12
l
(Razmaraetal.,2013). The second term is asso-
Table 2: Comparison of training results.
ciated with label score acquisition from neighbor
Slot/ValueF-scorein%.
nodesi.e. smoothsthelabelsaccordingtothesim-
ilarity graph. In the last term, the labels are regu-
larized tomatchapriorilabelR inorder toavoid chosen these parameters along with graph fea-
l
false labels for high degree unlabeled nodes. A tures and graph-related parameters by evaluating
solution to the optimization problem in (4) can the model on the development set. We employed
be found with an efficient iterative algorithm de- the L-BFGSalgorithm to optimize CRFobjective
scribedin(TalukdarandCrammer,2009). functions; which is designed to be fast and low-
Many errors of the alignment model are cor- memory consumer for the high-dimensional opti-
rected through label propagation using the MAD mizationproblems(Bertsekas, 1999).
algorithm; whereas, those errors are propagated We have post-processed the sequence of labels
intraditional labelpropagation algorithmssuchas to obtain the slots and their values. The slot-
theonementionedin(Subramanyaetal.,2010). valuepairiscomparedtothereferencetestsetand
the result is reported in F-score of slot classifica-
2.5 SystemOverview tion. Table 2 demonstrates results obtained from
our semi-supervised CRF algorithm compared to
We have implemented the Graph Construc-
thesupervised CRFandself-trained CRF.Experi-
tion in Java and the CRF is implemented
mentswerecarriedouthaving10%,20%and30%
by modifying the source code of CRFSuite
of data being labeled. For each of these tests, la-
(Okazaki, 2007). We have also modified Junto
beled set was selected randomly from the train-
toolkit (TalukdarandPereira,2010) and used it
ing set. This procedure was done 10 times and
for graph propagation. The whole source code of
our system is available online1. The input utter- the reported results are the average of the results
thereof. The Supervised CRF model is trained
ances and their corresponding semantic trees are
only on the labeled fraction of the data. How-
aligned using GIZA++ (OchandNey,2000); and
ever, the Self-trained CRF and Semi-supervised
thenusedtotrainthebaseCRFmodel. Thegraph
CRF have access to the rest of the data as well,
isconstructedusingthelabeledandunlabeleddata
whichareunlabeled. OurSupervised CRFgained
andthemainloopofthealgorithm continuesuntil
91.02F-scorewith100%ofthedatalabeledwhich
convergence. Thefinalparameters oftheCRFare
performs better compared to 89.32% F-score of
retained fordecoding inthetestphase.
RaymondandRiccardi(2007)CRFmodel.
3 Experimental Results AsshowninTable2,theproposed method per-
forms better compared to supervised CRF and
In this section we evaluate our results on self-trained CRF. The most significant improve-
Air Travel Information Service (ATIS) data-set ment occurs when only 10% of training set is
(Dahletal.,1994) which consists of 4478 train- labeled; where we gain 1.65% improvement on
ing, 500 development and 896 test utterances. F-score compared to supervised CRF and 1.38%
The development set was chosen randomly. To comparedtoself-trained CRF.
evaluate our work, we have compared our results
withresultsfromSupervisedCRFandSelf-trained 4 Conclusion
CRF(Yarowsky,1995).
Wepresentedasimplealgorithm totrainCRFina
Forourexperimentswesethyper-parametersas
semi-supervised manner using unaligned data for
follows: for graph propagation, µ = 1,µ =
1 2
SLU. By saving many informative labels in the
0.01, µ = 0.01, for Viterbi decoding, α = 0.1,
3
alignment phase, the base model is trained using
for CRF-retraining, η = 0.1, γ = 0.01. We have
fewerfeatures. Theparameters oftheCRFmodel
1https://github.com/maxxkia/g-ssl-crf are estimated using much less labeled data by
regularizing the model using a nearest-neighbor [Lin1998] DekangLin. 1998. Automaticretrievaland
graph. Results demonstrate that our proposed al- clustering of similar words. In Proceedings of the
36thAnnualMeetingoftheAssociationforCompu-
gorithm significantly improves the performance
tational Linguistics and 17th InternationalConfer-
comparedtosupervised andself-trained CRF.
enceonComputationalLinguistics-Volume2,ACL
’98,pages768–774,Stroudsburg,PA,USA.Associ-
ationforComputationalLinguistics.
References
[OchandNey2000] FranzJosefOchandHermannNey.
[Aliannejadietal.2014] Mohammad Aliannejadi, 2000. Giza++: Training of statistical translation
Shahram Khadivi, SaeedShiry Ghidary, and Mo- models.
hammadHadiBokaei. 2014. Discriminativespoken
language understanding using statistical machine [Okazaki2007] NaoakiOkazaki. 2007. Crfsuite: afast
translation alignment models. In Ali Movaghar, implementationof conditionalrandomfields (crfs).
MansourJamzad,andHosseinAsadi,editors,Artifi- URLhttp://www.chokkan.org/software/crfsuite.
cialIntelligenceandSignalProcessing,volume427
of Communications in Computer and Information [Pieraccinietal.1992] R. Pieraccini, E. Tzoukermann,
Science, pages 194–202. Springer International Z.Gorelov,J.Gauvain,E.Levin,Chin-HuiLee,and
Publishing. J.G. Wilpon. 1992. A speech understanding sys-
tembasedonstatisticalrepresentationofsemantics.
[Bertsekas1999] DimitriPBertsekas. 1999. Nonlinear InAcoustics, Speech,andSignalProcessing, 1992.
programming. ICASSP-92., 1992 IEEE International Conference
on,volume1,pages193–196vol.1,Mar.
[Chapelleetal.2006] Olivier Chapelle, Bernhard
Scho¨lkopf, Alexander Zien, et al. 2006. Semi- [RaymondandRiccardi2007] Christian Raymond and
supervised learning, volume 2. MIT press GiuseppeRiccardi. 2007. Generativeand discrim-
Cambridge. inativealgorithmsfor spokenlanguageunderstand-
ing. In International Conference on Speech Com-
[Dahletal.1994] Deborah A. Dahl, Madeleine Bates, munication and Technologies, pages 1605–1608,
Michael Brown, William Fisher, Kate Hunicke- Antwerp,Belgium,August.
Smith,DavidPallett,ChristinePao,AlexanderRud-
nicky,andElizabethShriberg. 1994. Expandingthe [Razmaraetal.2013] Majid Razmara, Maryam Siah-
scopeoftheatistask:Theatis-3corpus. InProceed- bani, Gholamreza Haffari, and Anoop Sarkar.
ingsoftheWorkshoponHumanLanguageTechnol- 2013. Graph propagation for paraphrasing out-of-
ogy,HLT’94,pages43–48,Stroudsburg,PA,USA. vocabularywordsin statistical machinetranslation.
AssociationforComputationalLinguistics. InProceedingsoftheConferenceoftheAssociation
forComputationalLinguistics(ACL).
[DasandSmith2012] Dipanjan Das and Noah A.
Smith. 2012. Graph-basedlexiconexpansionwith [Subramanyaetal.2010] Amarnag Subramanya, Slav
sparsity-inducing penalties. In Proceedings of the Petrov, and Fernando Pereira. 2010. Efficient
2012 Conference of the North American Chapter graph-basedsemi-supervised learning of structured
of the Association for Computational Linguistics: tagging models. In Proceedings of the 2010 Con-
Human LanguageTechnologies, NAACL HLT ’12, ferenceonEmpiricalMethodsinNaturalLanguage
pages677–687,Stroudsburg,PA,USA.Association Processing, EMNLP ’10, pages 167–176, Strouds-
forComputationalLinguistics. burg,PA,USA.AssociationforComputationalLin-
guistics.
[GarretteandBaldridge2013] Dan Garrette and Jason
Baldridge. 2013. Learning a part-of-speech tag- [TalukdarandCrammer2009] ParthaPratim Talukdar
ger from two hours of annotation. In Proceedings and Koby Crammer. 2009. New regularized
ofNAACL-HLT,pages138–147. algorithms for transductive learning. In Wray
Buntine, Marko Grobelnik, Dunja Mladeni, and
[HeandYoung2005] YulanHeandSteveYoung. 2005. John Shawe-Taylor, editors, Machine Learning
Semantic processing using the hidden vector state and Knowledge Discovery in Databases, volume
model. Computer Speech & Language, 19(1):85– 5782 of Lecture Notes in Computer Science, pages
106. 442–457.SpringerBerlinHeidelberg.
[KhadiviandNey2005] Shahram Khadivi and Her- [TalukdarandPereira2010] ParthaPratimTalukdarand
mann Ney. 2005. Automatic filtering of bilin- Fernando Pereira. 2010. Experiments in graph-
gual corpora for statistical machine translation. In based semi-supervised learning methods for class-
Andre´s Montoyo, Rafael Mun´oz, and Elisabeth instanceacquisition. InProceedingsofthe48thAn-
Me´tais, editors, Natural Language Processing and nualMeeting of the Association for Computational
InformationSystems,volume3513ofLectureNotes Linguistics, ACL ’10, pages 1473–1481, Strouds-
in Computer Science, pages 263–274. Springer burg,PA,USA.AssociationforComputationalLin-
BerlinHeidelberg. guistics.
[Talukdaretal.2008] Partha Pratim Talukdar, Joseph
Reisinger, Marius Pas¸ca, Deepak Ravichandran,
Rahul Bhagat, and Fernando Pereira. 2008.
Weakly-supervised acquisition of labeled class in-
stancesusinggraphrandomwalks. In Proceedings
of the Conference on Empirical Methods in Natu-
ralLanguageProcessing,EMNLP’08,pages582–
590, Stroudsburg, PA, USA. Association for Com-
putationalLinguistics.
[WangandAcero2006] Ye-Yi Wang and Alex Acero.
2006. Discriminative models for spoken language
understanding. In International Conference on
SpeechCommunicationandTechnologies.
[Yarowsky1995] David Yarowsky. 1995. Unsuper-
visedwordsensedisambiguationrivalingsupervised
methods. InProceedingsofthe33rdAnnualMeet-
ing on Association for Computational Linguistics,
ACL ’95, pages 189–196, Stroudsburg, PA, USA.
AssociationforComputationalLinguistics.
[Zengetal.2013] Xiaodong Zeng, Derek F Wong,
Lidia S Chao, and Isabel Trancoso. 2013. Graph-
basedsemi-supervisedmodelforjointchineseword
segmentation and part-of-speech tagging. In ACL,
pages770–779.
[Zhuetal.2003] Xiaojin Zhu, Zoubin Ghahramani,
John Lafferty, et al. 2003. Semi-supervisedlearn-
ingusinggaussianfieldsandharmonicfunctions. In
ICML,volume3,pages912–919.
This figure "graph.jpg" is available in "jpg"(cid:10) format from:
http://arxiv.org/ps/1701.08533v1
This figure "mainflowchart-en.jpg" is available in "jpg"(cid:10) format from:
http://arxiv.org/ps/1701.08533v1