Long-Answer Question Answering and Rhetorical-Semantic Relations

272 Pages·2012·1.42 MB·English

Long-Answer Question Answering and
Rhetorical-Semantic Relations
Sasha J. Blair-Goldensohn
Submitted in partial ful(cid:12)llment of the
Requirements for the degree
of Doctor of Philosophy
in the Graduate School of Arts and Sciences
COLUMBIA UNIVERSITY
2007
(cid:13)c 2007
Sasha J. Blair-Goldensohn
All Rights Reserved
ABSTRACT
Long-Answer Question Answering and
Rhetorical-Semantic Relations
Sasha J. Blair-Goldensohn
Over the past decade, Question Answering (QA) has generated considerable interest
and participation in the (cid:12)elds of Natural Language Processing and Information Retrieval.
Conferences such as TREC,CLEF andDUC have examined various aspects of the QA task
in the academic community. In the commercial world, major search engines from Google,
Microsoft and Yahoo have integrated basic QA capabilities into their core web search.
These e(cid:11)orts have focused largely on so-called \factoid" questions seeking a single fact,
such as the birthdate of an individual or the capital city of a country. Yet in the past few
years, there has been growing recognition of a broad class of \long-answer" questions which
cannot be satisfactorily answered in this framework, such as those seeking a de(cid:12)nition,
explanation, or other descriptive information in response. In this thesis, we consider the
problem of answering such questions, with particular focus on the contribution to be made
by integrating rhetorical and semantic models.
We present DefScriber, a system for answering de(cid:12)nitional (\What is X?"), biographi-
cal (\Who is X?") and other long-answer questions using a hybrid of goal- and data-driven
methods. Ourgoal-driven, or top-down, approach is motivated by a set of de(cid:12)nitional pred-
icates which capture information types commonly useful in de(cid:12)nitions; our data-driven, or
bottom-up, approach uses dynamic analysis of input data to guide answer content. In sev-
eral evaluations, we demonstrate that DefScriber outperforms competitive summarization
techniques, and ranks among the top long-answer QA systems being developed by others.
Motivated by our experience with de(cid:12)nitional predicates in DefScriber, we pursue a
set of experiments which automatically acquire broad-coverage lexical models of rhetorical-
semantic relations(RSRs)suchasCauseandContrast. BuildingontheframeworkofMarcu
and Echihabi (Marcu and Echihabi, 2002), we implement techniques to improve the quality
of these models using syntactic (cid:12)ltering and topic segmentation, and present evaluation
results showing that these methods can improve the accuracy of relation classi(cid:12)cation.
Lastly, we implement two approaches for applying the knowledge in our RSR models
to enhance the performance and scope of DefScriber. First, we integrate RSR models into
the answer-building process in DefScriber, (cid:12)nding incremental improvements with respect
to the content and ordering of responses. Second, we use our RSR models to help identify
relevant answer material for an exploratory class of \relation-focused" questions which seek
explanatory or comparative responses. We demonstrate that in the case of explanation
questions, using RSRs can lead to signi(cid:12)cantly more relevant responses.
Contents
1 Introduction 1
1.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Chapter Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 DefScriber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 RSR Modeling and Classi(cid:12)cation . . . . . . . . . . . . . . . . . . . . 5
1.2.3 Applying RSRs in DefScriber . . . . . . . . . . . . . . . . . . . . . . 6
1.2.4 Applying RSRs For Causal and Comparison Questions . . . . . . . . 7
2 DefScriber 9
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 A Hybrid Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.1 Goal-Driven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.2 Data-Driven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.3 Putting it Together. . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3.4 Web Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Versions and Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.1 De(cid:12)nitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.2 Biographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4.3 Topic-Focused Questions. . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5 Integrating Speech Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
i
2.5.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.5.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.6 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3 Rhetorical-Semantic Relations 42
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.1 Rhetorical and Discourse Theory . . . . . . . . . . . . . . . . . . . . 44
3.2.2 Lexical Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2.3 Automatic Relation Modeling . . . . . . . . . . . . . . . . . . . . . . 47
3.3 De(cid:12)ning Rhetorical-Semantic Relations . . . . . . . . . . . . . . . . . . . . 49
3.4 The M&E Relation Modeling Framework . . . . . . . . . . . . . . . . . . . 53
3.5 TextRels: Modeling and Classifying RSRs . . . . . . . . . . . . . . . . . . . 55
3.5.1 Parameter Optimization . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.5.1.1 Stoplist Binning and Minimum Frequency Cuto(cid:11)s . . . . . 60
3.5.1.2 Vocabulary Size and Laplace Lambda . . . . . . . . . . . . 61
3.5.1.3 Overall Optimization . . . . . . . . . . . . . . . . . . . . . 64
3.5.2 Topic Segmentation Filtering . . . . . . . . . . . . . . . . . . . . . . 68
3.5.2.1 Segmenting the Corpus . . . . . . . . . . . . . . . . . . . . 69
3.5.2.2 Training with Segmented Data . . . . . . . . . . . . . . . . 71
3.5.2.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.5.3 Syntax-Based Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.5.3.1 Analyzing and Classifying Syntactic Errors . . . . . . . . . 77
3.5.3.2 Formulating Syntactic Filtering Heuristics . . . . . . . . . 77
3.5.3.3 Evaluating the Heuristics . . . . . . . . . . . . . . . . . . . 82
3.5.3.4 Training with Syntactically Filtered Data . . . . . . . . . . 84
3.5.3.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.6 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 92
ii
4 Applying RSRs in DefScriber 94
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.3 Extending DefScriber for RSR Integration . . . . . . . . . . . . . . . . . . . 99
4.3.1 Common Feature Framework . . . . . . . . . . . . . . . . . . . . . . 101
4.3.2 RSR Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.3.3 Other Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.3.4 Revised Sentence Selection Algorithm . . . . . . . . . . . . . . . . . 104
4.4 Content Selection Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.4.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.4.2 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.5 Content Ordering Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.5.1 Reordering Responses . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.5.2 The Reordering Algorithm. . . . . . . . . . . . . . . . . . . . . . . . 115
4.5.3 Experiment and Data . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.5.4 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.5.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
4.6 DUC 2006 Result Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
4.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5 Applying RSRs for Relation-Focused Questions 127
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5.3 Relation-Focused Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.4 Narrowing Our Focus: Two Question Types . . . . . . . . . . . . . . . . . . 139
5.4.1 Explanation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.4.2 Comparison Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.5 Implementation: CCQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
iii
5.5.1 CCQ Pipeline Overview . . . . . . . . . . . . . . . . . . . . . . . . . 141
5.5.2 Web Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.6 CCQ Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
5.6.1 Query Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5.6.2 Document Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.6.3 Related Term Generation . . . . . . . . . . . . . . . . . . . . . . . . 148
5.6.3.1 Keyword and Cue Terms . . . . . . . . . . . . . . . . . . . 148
5.6.3.2 RSR Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
5.6.3.3 LSA and WordNet Terms . . . . . . . . . . . . . . . . . . . 156
5.6.4 Sentence Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
5.6.5 Answer Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
5.6.6 Parameter Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 161
5.7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
5.7.1 Creating Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . 161
5.7.2 Evaluation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
5.7.3 Explanation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 168
5.7.3.1 Interannotator Agreement. . . . . . . . . . . . . . . . . . . 169
5.7.3.2 Comparing Methods . . . . . . . . . . . . . . . . . . . . . . 171
5.7.4 Comparison Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 175
5.7.4.1 Interannotator Agreement. . . . . . . . . . . . . . . . . . . 175
5.7.4.2 Comparing Methods . . . . . . . . . . . . . . . . . . . . . . 176
5.8 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
5.8.1 Term Suggestion Quality . . . . . . . . . . . . . . . . . . . . . . . . 177
5.8.2 Term-System Interaction . . . . . . . . . . . . . . . . . . . . . . . . 182
5.9 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 184
6 Contributions, Limitations and Future Work 189
6.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
6.2 Contributions by Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
iv
6.2.1 DefScriber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
6.2.2 Rhetorical-Semantic Relations . . . . . . . . . . . . . . . . . . . . . 191
6.2.3 Applying RSRs in DefScriber . . . . . . . . . . . . . . . . . . . . . . 192
6.2.4 Applying RSRs in CCQ . . . . . . . . . . . . . . . . . . . . . . . . . 192
6.3 Deliverables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
6.4 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 194
A Cluster Evaluation Metrics 198
B RSR Patterns and Instance Distributions 200
B.1 Cause Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
B.2 Contrast Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
C CCQ Evaluation Materials and Sample Responses 206
C.1 Evaluation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
C.1.1 Explanation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 206
C.1.2 Comparison Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 208
C.2 Sample Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
C.2.1 Explanation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 210
C.2.2 Comparison Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 218
C.3 Evaluation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
C.3.1 Explanation Rating Instructions . . . . . . . . . . . . . . . . . . . . 226
C.3.2 Comparison Rating Instructions . . . . . . . . . . . . . . . . . . . . 234
Bibliography 237
v
List of Figures
2.1 Pattern extraction and matching for a Genus-Species sentence from an ex-
ample sentence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 DefScriber module-level processing of \What is the Hajj?" Individual mod-
ules are described in Section 2.3. (In the input, T, N and L represent query,
maximum source documents, and desired answer length, respectively.) . . . 21
2.3 Input and output screens for the Web-based interface to DefScriber via the
CUAQ system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4 Mean userjudgmentscoresforeach con(cid:12)gurationandfeaturetypeinSection
2.4.1. Higher is better for all features except Redundancy. . . . . . . . . . . 26
2.5 An example of rewriting in our question-focused summary for document set
192 (\Who is Sonia Gandhi?") . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.6 An example DUC 2005 topic (d407b) and DefScriber’s answer. . . . . . . . 31
2.7 Mean systems-above rank across responsiveness, mean of ROUGE-2 and
ROUGE-SU4 recall, and mean of linguistic quality questions 1-5. DefScriber
is the fourth peer from the left, marked as \Columbia." Note that lower
scores are better. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.8 Cluster Homogeneity, Completeness and Validity for speech and text clus-
tering task. The error bars show Completeness and Homogeneity, while the
midpoints show Validity (at lower thresholds Completeness dominates; at
higher thresholds Homogeneity does) . . . . . . . . . . . . . . . . . . . . . . 39
vi