MODELS FOR IMPROVED TRACTABILITY AND ACCURACY IN DEPENDENCY PARSING A ...

166 Pages·2013·1.15 MB·English

MODELSFORIMPROVEDTRACTABILITYANDACCURACYINDEPENDENCY
PARSING
EmilyPitler
ADISSERTATION
in
ComputerandInformationScience
PresentedtotheFacultiesoftheUniversityofPennsylvania
in
PartialFulfillmentoftheRequirementsforthe
DegreeofDoctorofPhilosophy
2013
SupervisorofDissertation Co-SupervisorofDissertation
MitchellP.Marcus SampathKannan
Professor,ComputerandInformationScience Professor,ComputerandInformationScience
GraduateGroupChairperson
ValTannen,Professor,ComputerandInformationScience
DissertationCommittee:
MichaelCollins,Professor,ComputerScience,ColumbiaUniversity
ChrisCallison-Burch,AssistantProfessor,ComputerandInformationScience
MarkLiberman,Professor,Linguistics
BenTaskar,AssociateProfessor,ComputerandInformationScience
MODELS FOR IMPROVED TRACTABILITY AND ACCURACY IN DEPENDENCY
PARSING
COPYRIGHT
2013
EmilyPitler
Acknowledgements
Mitch Marcus and Sampath Kannan were wonderful advisors and I am truly grateful to
both of them for taking me on as a student. Mitch’s patience and unwavering support
allowed me the freedom to find this topic for this thesis. His wide-ranging knowledge of
both linguistics and computer science was incredibly valuable. Sampath inspired me to
strive for clean solutions to problems. His clarity helped me to identify the crucial pieces
ofanyresultandgreatlyimprovedthematerialpresentedhere.
Iamgratefultomythesiscommittee: ChrisCallison-Burch,MikeCollins,MarkLiber-
man, and Ben Taskar. I have benefited from Chris’s advice throughout graduate school, at
both Penn and JHU. I have tried to emulate his clear presentation style in both my talks
and my writing. Mike’s work in parsing served as inspiration for me for much of the work
in this thesis. His detailed questions have both improved this document and pointed the
way towards future directions. Mark’s questions at CLunch over the years always brought
up subtle but important details. Discussions of treebank representations in this document
are largely due to Mark’s influence. Ben helped me identify interesting directions at a for-
mative stage in this work. Throughout the process, Ben encouraged me to simultaneously
investigateboththeoreticalandempiricalissues.
Alargenumberofpeoplecontributedtomyundertakingthisdissertationandfollowing
through. Thanks especially to Jerry Berry and Stephen Rose at Thomas Jefferson High
School for Science and Technology for introducing me to programming and computer sci-
ence; Charles Yang, at Yale and Penn, for introducing me to computational linguistics;
DanaAngluinandBrianScassellatiatYaleforencouragingmetopursuegraduateschool;
iii
and Ken Church, Dekang Lin, and Ani Nenkova for guiding me as I started research. I
benefited tremendously from two summers with Ken: he patiently taught me how to write
papers and how to approach problems from multiple perspectives. Dekang has been very
supportive of me and my research ever since working together at a JHU CLSP Summer
Workshop in 2009. Dekang introduced me to research in parsing and applications of pars-
ing. Ani Nenkova and Annie Louis were wonderful collaborators. Annie Louis was my
constant partner throughout graduate school; it was a pleasure to go through courses, re-
search, internships, and other milestones together. Thanks to Katerina Fragkiadaki, Jenny
Gillenwater, Arun Raghavan, and David Weiss for providing feedback on drafts and prac-
ticetalks;ShaneBergsma,JasonEisner,AravindJoshi,JulieLegate,RyanMcDonald,Slav
Petrov,GiorgioSatta,andDavidYarowskyforproductiveconversationsthatimpactedthis
thesis; Terry Koo for his assistance with the dpo3 parser; Daniel Zeman for his assistance
withtheHamleDTtreebankconversionsoftware;MikeFelkerforremovingadministrative
hurdles; Cheryl Hickey for all her support; the students, staff, and volunteers at White-
Williams Scholars/Philadelphia Futures for a rewarding time and memories; Drew Hilton,
Santosh Nagarakatte, and Arun Raghavan for long games of bridge; Katie Gibson for bak-
ingdeliciouscupcakesandfudge;andotherstudentsIhadthepleasureofoverlappingwith
at Penn, including Adam Aviv, Alexis Baird, John Blitzer, Chris Casinghino, Erwin Chan,
SanjianChen,NikhilDinesh,MarkDredze,RyanGabbard,KuzmanGanchev,Joa˜oGrac¸a,
MichaelGreenberg,KaiHong,LiangHuang,MarieJacob,SarahJohnstone,AndrewKing,
Alex Kulesza, Junyi Li, Constantine Lignos, Xi Lin, Ellie Pavlick, Alex Roderer, Sudeepa
Roy,BenSapp,ParthaPratimTalukdar,AndrewWest,andQiuyeZhao.
Iamdeeplygratefultomycollegefriends: Alison,Betsy,Caroline,Charlie,Guy,Illana,
Jackie, Joel, Julia, Lekshmi, Michael, and Samarth for their visits and encouragement.
Samarth and I were classmates from middle school through undergrad and we wrote our
first paper together. I would like to thank my brother Will, my grandparents, and Sudipto
fortheirsteadfastbeliefinme. Mostimportantly,Iamthankfultomyparentsfornurturing
mycuriosityandfortheirloveandsupport.
iv
ABSTRACT
MODELSFORIMPROVEDTRACTABILITYANDACCURACYINDEPENDENCY
PARSING
EmilyPitler
MitchellP.Marcus
SampathKannan
Automaticsyntacticanalysisofnaturallanguageisoneofthefundamentalproblemsinnat-
ural language processing. Dependency parses (directed trees in which edges represent the
syntacticrelationshipsbetweenthewordsinasentence)havebeenfoundtobeparticularly
usefulformachinetranslation,questionanswering,andotherpracticalapplications.
For English dependency parsing, we show that models and features compatible with
how conjunctions are represented in treebanks yield a parser with state-of-the-art overall
accuracyandsubstantialimprovementsintheaccuracyofconjunctions.
ForlanguagesotherthanEnglish,dependencyparsinghasoftenbeenformulatedasei-
thersearchingovertreeswithoutanycrossingdependencies(projectivetrees)orsearching
over all directed spanning trees. The former sacrifices the ability to produce many natu-
ral language structures; the latter is NP-hard in the presence of features with scopes over
siblingsorgrandparentsinthetree.
Thisthesisexploresalternativewaystosimultaneouslyproducecrossingdependencies
intheoutputandusemodelsthatparametrizeovermultipleedges.
Gap inheritance is introduced in this thesis and quantifies the nesting of subtrees over
intervals. The thesis provides O(n6) and O(n5) edge-factored parsing algorithms for two
new classes of trees based on this property, and extends the latter to include grandparent
factors.
Thisthesisthendefines1-Endpoint-Crossingtrees,inwhichforanyedgethatiscrossed,
all other edges that cross that edge share an endpoint. This property covers 95.8% or more
of dependency parses across a variety of languages. A crossing-sensitive factorization in-
troduced in this thesis generalizes a commonly used third-order factorization (capable of
v
scoringtriplesofedgessimultaneously).
This thesis provides exact dynamic programming algorithms that find the optimal 1-
Endpoint-Crossingtreeundereitheranedge-factoredmodelorthiscrossing-sensitivethird-
order model in O(n4) time, orders of magnitude faster than other mildly non-projective
parsing algorithms and identical to the parsing time for projective trees under the third-
order model. The implemented parser is significantly more accurate than the third-order
projectiveparserundermanyexperimentalsettingsandsignificantlylessaccurateonnone.
vi
Contents
Acknowledgements iii
1 Introduction 1
1.1 Background: Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 Factorizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.3 ParsingwithUnlabeledDataandRelevantFactorizations . . . . . . 7
1.3 Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.1 ProjectiveTrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.2 Arborescences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.3 ExistingDefinitionsofMildlyNon-projectiveTrees . . . . . . . . 10
1.3.4 ClassesofTreesProposedinthisThesis . . . . . . . . . . . . . . . 10
1.4 FactorizationsthatFacilitateSearch . . . . . . . . . . . . . . . . . . . . . 12
1.5 ThesisContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2 Attacking Parsing Bottlenecks with Unlabeled Data and Relevant Factoriza-
tions 16
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 ConversiontoDependencyRepresentations . . . . . . . . . . . . . . . . . 18
2.3 ImplicationsofRepresentationsontheScopeofFactorization . . . . . . . 22
2.3.1 Edge-basedScoring . . . . . . . . . . . . . . . . . . . . . . . . . 22
vii
2.3.2 SiblingScoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.3 GrandparentScoring . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.4 Grandparent-SiblingScoring . . . . . . . . . . . . . . . . . . . . . 25
2.4 UsingUnlabeledDataEffectively . . . . . . . . . . . . . . . . . . . . . . 26
2.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.1 UnlabeledDataFeatureSet . . . . . . . . . . . . . . . . . . . . . 27
2.5.2 Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.5.3 ExperimentalSet-up . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.6 ResultsandDiscussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.6.1 ImpactofFactorization . . . . . . . . . . . . . . . . . . . . . . . . 30
2.6.2 ImpactofUnlabeledData . . . . . . . . . . . . . . . . . . . . . . 31
2.6.3 ComparisonwithOtherParsers . . . . . . . . . . . . . . . . . . . 31
2.6.4 ImpactofDataRepresentation . . . . . . . . . . . . . . . . . . . . 32
2.6.5 PrepositionErrorAnalysis . . . . . . . . . . . . . . . . . . . . . . 33
2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3 DynamicProgrammingforHigherOrderParsingofGap-MindingTrees 35
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3 GapInheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 1-InheritTrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.1 ParsingWell-nested,Blockdegree2,1-InheritTrees . . . . . . . . 41
3.5 Gap-mindingTrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.5.1 Runtimeanalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.6 ExtensiontoGrandparentFactorizations . . . . . . . . . . . . . . . . . . . 51
3.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.8 ExtensiontoArbitraryGapDegree . . . . . . . . . . . . . . . . . . . . . . 55
3.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
viii
4 FindingOptimal1-Endpoint-CrossingTrees 59
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 AdditionalDefinitionsofNon-Projectivity . . . . . . . . . . . . . . . . . . 60
4.3 Edges(andtheirCrossingPoint)DefineIsolatedCrossingRegions . . . . . 61
4.4 ParsingAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.4.1 DecomposinganIntsub-problem . . . . . . . . . . . . . . . . . . 66
4.4.2 DecomposinganLR sub-problem . . . . . . . . . . . . . . . . . . 69
4.4.3 DecomposinganN sub-problem . . . . . . . . . . . . . . . . . . . 70
4.4.4 DecomposinganLorR sub-problem . . . . . . . . . . . . . . . . 70
4.5 DynamicProgramtofindthemaximumscoring
1-Endpoint-CrossingTree . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.6 Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.6.1 GraphTheory: All1-Endpoint-CrossingTreesare2-Planar . . . . . 78
4.6.2 Linguistics: Cross-serialVerbConstructionsand
SuccessiveCyclicity . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.7 ASimplifiedForm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5 ACrossing-SensitiveThird-OrderFactorizationforDependencyParsing 84
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.2.1 Grand-SiblingProjectiveParsing . . . . . . . . . . . . . . . . . . . 87
5.2.2 Edge-factored1-Endpoint-CrossingParsing . . . . . . . . . . . . . 88
5.3 Crossing-SensitiveFactorization . . . . . . . . . . . . . . . . . . . . . . . 90
5.4 ParsingOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.4.1 EnforcingCrossingEdges . . . . . . . . . . . . . . . . . . . . . . 94
5.4.2 ReducedContextinPresenceofCrossings . . . . . . . . . . . . . 100
5.4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
ix
5.5.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6 ConclusionsandFutureWork 113
6.1 FutureDirections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.1.1 DevelopingFasterVariantsof1-Endpoint-CrossingParsing . . . . 114
6.1.2 AlternativeDescriptionsof1-Endpoint-CrossingTrees . . . . . . . 116
6.1.3 LinguisticConnections . . . . . . . . . . . . . . . . . . . . . . . . 118
6.1.4 ApplicationsbeyondParsing . . . . . . . . . . . . . . . . . . . . . 123
A AdditionalInformationAbouttheCoverageof1-Endpoint-CrossingTrees 124
A.1 English . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
A.2 Danish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
A.3 Dutch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
B GrandSib-CrossingParserInvariants 129
C FullDynamicProgramfortheGrandSib-CrossingParser 133
x