Automatic Factual Question Generation from Text - Noah's ARK

203 Pages·2011·1.73 MB·English

Automatic Factual Question Generation from Text
Michael Heilman
CMU-LTI-11-004
Language Technologies Institute
School of Computer Science
Carnegie Mellon University
5000 Forbes Ave., Pittsburgh, PA 15213
www.lti.cs.cmu.edu
Thesis Committee:
Vincent Aleven, Carnegie Mellon University
William W. Cohen, Carnegie Mellon University
Lori Levin, Carnegie Mellon University
Diane J. Litman, University of Pittsburgh
Noah A. Smith (chair), Carnegie Mellon University
Submitted in partial fulfillment of the requirements
for the degree of Doctor of Philosophy
in Language and Information Technologies
(cid:13)c 2011, Michael Heilman
Abstract
TextswithpotentialeducationalvaluearebecomingavailablethroughtheInternet(e.g.,Wikipedia,
newsservices). However,usingthesenewtextsinclassroomsintroducesmanychallenges,oneof
whichisthattheyusuallylackpracticeexercisesandassessments. Here,weaddresspartofthischal-
lengebyautomatingthecreationofaspecifictypeofassessmentitem.
Specifically,wefocusonautomaticallygeneratingfactualWHquestions. Ourgoalistocreate
anautomatedsystemthatcantakeasinputatextandproduceasoutputquestionsforassessinga
reader’sknowledgeoftheinformationinthetext. Thequestionscouldthenbepresentedtoateacher,
whocouldselectandrevisetheonesthatheorshejudgestobeuseful.
Afterintroducingtheproblem,wedescribesomeofthecomputationalandlinguisticchallenges
presentedbyfactualquestiongeneration. Wethenpresentanimplementedsystemthatleveragesex-
istingnaturallanguageprocessingtechniquestoaddresssomeofthesechallenges. Thesystemusesa
combinationofmanuallyencodedtransformationrulesandastatisticalquestionrankertrainedona
tailoreddatasetoflabeledsystemoutput.
Wepresentexperimentsthatevaluateindividualcomponentsofthesystemaswellasthesystem
asawhole. Wefound,amongotherthings,thatthequestionrankerroughlydoubledtheacceptability
rateoftop-rankedquestions.
Inauserstudy,wetestedwhetherK-12teacherscouldefficientlycreatefactualquestionsbyse-
lectingandrevisingsuggestionsfromthesystem. Offeringautomaticsuggestionsreducedthetime
andeffortspentbyparticipants,thoughitalsoaffectedthetypesofquestionsthatwerecreated.
Thisresearchsupportstheideathatnaturallanguageprocessingcanhelpteachersefficientlycre-
ateinstructionalcontent. Itprovidessolutionstosomeofthemajorchallengesinquestiongeneration
andananalysisandbetterunderstandingofthosethatremain.
Acknowledgements
Inadditiontothedissertationcommittee,theauthorwouldliketoacknowledgethefollowingpeo-
pleforhelpinghimconductthisresearch: thereviewersofthepublicationsrelatedtothiswork,for
theirhelpfulcomments;BrendanO’Connor,fordevelopingthetreevizparsevisualizationtool,for
hisworkontheARKreftool,andforideasaboutinformationextraction;NathanSchneider,Dipan-
janDas,KevinGimpel,ShayCohen,andothermembersoftheARKresearchgroup,forvarious
helpfuldiscussions;JustinBetteridgefordiscussionsaboutconnectionsbetweenquestiongenera-
tionandinformationextraction;NoraPresson,RuthWylie,andLeighAnnSudol,fordiscussions
abouttheuserinterfaceanduserstudy;MaxineEskenaziandMatthiasScheutz,forbeinghisprevi-
ousadvisors;HowardSeltman,NoraPresson,andTracySweet,forstatisticaladvice;theorganiz-
ersofthequestiongenerationworkshops,forenablingmanythoughtfuldiscussionsaboutthetopic;
JackMostow,foraskingmanygoodquestions;andJillBurstein,forprovidinglinkstosomeofthe
textsusedinthiswork.
Theauthorwouldalsoliketoacknowledgepartialsupportfromthefollowingorganizations: the
SiebelScholarsorganization;theNationalScienceFoundation,foraGraduateResearchFellowship
awardedtotheauthorandforgrantIIS-0915187,awardedtoNoahSmith;andtheU.S.Department
ofEducation’sInstituteofEducationSciences,forgrantR305B040063toCarnegieMellonUni-
versityanditsProgramforInterdisciplinaryEducationResearch(PIER)directedbyDavidKlahr
andSharonCarver. Theviewsexpressedinthisworkarethoseoftheauthorandarenotnecessarily
thoseoftheaboveorganizations.
Finally,theauthorwouldliketothankhisparentsandhiswifefortheirunflaggingsupportover
manyyears.
iii
Contents
1 Introduction 1
1.1 IllustrativeExampleofFactualQuestionGeneration . . . . . . . . . . . . . . . . . . 2
1.2 InstructionalContentGeneration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 TypesofQuestions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.2 TypeofInformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.3 SourceofInformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.4 LengthoftheExpectedAnswer . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.5 CognitiveProcesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 EducationalValueofFactualQuestions . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 ComparisontotheClozeProcedure . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6 PriorWorkonIntelligentToolsforEducation . . . . . . . . . . . . . . . . . . . . . 11
1.7 PriorWorkonOvergeneration-and-Ranking . . . . . . . . . . . . . . . . . . . . . . 13
1.8 ConnectionstoOtherProblemsinNaturalLanguageProcessing . . . . . . . . . . . 15
1.9 PriorWorkonQuestionGeneration . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.9.1 ResearchonOtherQuestionGenerationProblems . . . . . . . . . . . . . . 17
1.9.2 BroadSimilaritiesandDifferences . . . . . . . . . . . . . . . . . . . . . . . 18
1.9.3 SpecificApproachestoFactualQuestionGeneration . . . . . . . . . . . . . 19
1.10 PrimaryContributionsandThesisStatement . . . . . . . . . . . . . . . . . . . . . . 21
1.10.1 SummaryofPrimaryContributions . . . . . . . . . . . . . . . . . . . . . . 21
1.10.2 ThesisStatement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
iv
2 ChallengesinQuestionGeneration 24
2.1 LexicalChallenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.1.1 MappingAnswerstoQuestionWordsandPhrases . . . . . . . . . . . . . . 25
2.1.2 Variationandparaphrasing . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.1.3 Non-compositionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.1.4 AnswersandDistractors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2 SyntacticChallenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.2.1 ShortcomingsofNLPToolsforAnalyzingSyntacticStructures . . . . . . . 29
2.2.2 VarietyofSyntacticConstructions . . . . . . . . . . . . . . . . . . . . . . . 32
2.2.3 ConstraintsonWH-Movement . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.3 DiscourseChallenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.3.1 VaguenessofInformationTakenOutofContext . . . . . . . . . . . . . . . . 35
2.3.2 ImplicitDiscourseRelationsandWorldKnowledge . . . . . . . . . . . . . . 39
2.4 ChallengesRelatedtoUseofQuestionGenerationTools . . . . . . . . . . . . . . . 42
2.4.1 ImportanceandRelevanceofInformation . . . . . . . . . . . . . . . . . . . 42
2.4.2 UsabilityandHuman-ComputerInteractionIssues . . . . . . . . . . . . . . 42
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3 Overgenerate-and-RankFrameworkforQuestionGeneration 44
3.1 LeveragingExistingNLPTools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.1.1 StanfordPhraseStructureParser . . . . . . . . . . . . . . . . . . . . . . . . 45
3.1.2 TheTregexTreeSearchingLanguageandTool . . . . . . . . . . . . . . . 47
3.1.3 SupersenseTagger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.1.4 ARKrefNounPhraseCoreferenceTool . . . . . . . . . . . . . . . . . . . . 48
3.2 Stage1: TransformationsofDeclarativeInputSentences . . . . . . . . . . . . . . . 50
3.2.1 ExtractingSimplifiedFactualStatements . . . . . . . . . . . . . . . . . . . 50
3.2.2 PronounResolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.3 Stage2: QuestionCreation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.3.1 MarkingUnmovablePhrases . . . . . . . . . . . . . . . . . . . . . . . . . . 61
v
3.3.2 GeneratingPossibleQuestionPhrases . . . . . . . . . . . . . . . . . . . . . 62
3.3.3 DecompositionoftheMainVerb . . . . . . . . . . . . . . . . . . . . . . . . 65
3.3.4 Subject-AuxiliaryInversion . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.3.5 RemovingAnswersandInsertingQuestionPhrases . . . . . . . . . . . . . . 67
3.3.6 Post-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.4 Stage3: QuestionRanking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4.1 StatisticalModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.4.3 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4 Experiments 74
4.1 TextCorpora . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.2 SimplifiedStatementExtractionExperiments . . . . . . . . . . . . . . . . . . . . . 75
4.2.1 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.2.2 ExperimentalSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3 SemanticTypeLabelingExperiments . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.3.1 AlternativeApproachestoSemanticTypeLabeling . . . . . . . . . . . . . . 81
4.3.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.4 QuestionRatingScheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.4.1 RatingSchemeandProcess . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4.2 MappingRatingstoAcceptabilityLabels . . . . . . . . . . . . . . . . . . . 88
4.4.3 Inter-raterAgreement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.5 SamplesofTextsandQuestions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.6 EvaluationMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
vi
4.7 QuestionRankingEvaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.7.1 PassiveAggressiveLearningAlgorithmsforRanking . . . . . . . . . . . . . 93
4.7.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.8 FurtherAnalysesofQuestionRanking . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.8.1 EffectofTrainingSetSize . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.8.2 DistributionofQuestionRatingswithandwithoutRanking. . . . . . . . . . 96
4.8.3 AnalysisofFeatureWeights . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.8.4 FeatureAblationStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.9 EffectsofWH-movementConstraints . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.10 End-to-endEvaluationExperiments . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.10.1 Document-LevelEvaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.10.2 Sentence-LevelEvaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.10.3 CharacteristicsofTop-RankedQuestions . . . . . . . . . . . . . . . . . . . 108
4.11 ErrorAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.11.1 ErrorTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.11.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5 UserStudy 117
5.1 Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.2 ExperimentalDesign . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.2.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.2.2 Texts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.2.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.2.4 OutcomeMeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.3 LinearMixedEffectsModeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.4 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
5.4.1 TimeonTask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.4.2 MentalEffortReports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
vii
5.4.3 AnalysisofQuestionTypes . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.4.4 ExitSurveyResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.4.5 UseofInterfaceFeatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.4.6 RevisionofSystem-GeneratedQuestions . . . . . . . . . . . . . . . . . . . 139
5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6 FutureWorkandConclusions 143
6.1 SummaryofExperimentalResults . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
6.2 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.2.1 AlternativeRepresentationsforQuestionGeneration . . . . . . . . . . . . . 145
6.2.2 RemainingChallenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
6.3 SummaryofContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
A RulesforIdentifyingPotentialAnswerPhrases 177
B InstructionsforQuestionAnnotation 182
C UserStudyInstructions 185
D AlternativeRankingMethods 188
E SentenceSimplificationExamplefromMobyDick 194
viii
Chapter 1
Introduction
Howwouldsomeonetellwhetheryouhavereadthistext? Theymightaskyoutosummarizeitor
describehowitrelatestoyourownresearch. Theymightaskyoutodiscussitsstrengthsandweak-
nessesortocompareittoanotherpaper.
Or,asafirststep,justtoseeifyoubotheredtogetthroughit,theymightaskyouwhatfeatures
weusedinourrankingmodel,orwhatcorporawetestedon. Thatis,atfirst,theymightjustaskyou
questionstocheckthatyourememberthebasicfacts. Then,ifitisclearthatyoureadmorethanthe
titleandabstract,theymightmoveontomorechallengingquestions.
Ofcourse,youareprobablyahighlyskilledandmotivatedreader,andtherewouldbenoneed
toassesswhetheryoureadandretainedbasicfactualinformation. However,thatisnotthecasewith
allreaders. Forexample,anelementaryschoolteachermightaskhisorherstudentsbasicquestions
sincetheyarestilllearningtoread.
Generatingsuchquestions,andauthoringreadingassessmentsmoregenerally,canbeatime-
consumingandeffortfulprocess. Inthisresearch,weworktowardautomatingthatprocess. Inpar-
ticular,wefocusontheproblemofautomaticallygeneratingfactualquestionsfromindividualtexts.
Weaimtocreateasystemforquestiongeneration(QG)thatcantakeasinputanarticleoftext
(e.g.,awebpageorencyclopediaarticlethatateachermightselecttosupplementthematerialsina
textbook),andcreateasoutputarankedlistoffactualquestions. Ausercouldthenselectandrevise
thesequestionsinordertocreatepracticeexercisesorpartofaquiztoassesswhetherstudentsread
thetextandretainedknowledgeaboutitstopic.
1
WefocusonQGaboutinformationaltexts—thatis,non-fictiontextsthatconveyfactualinforma-
tionratherthanopinions. WhileQGaboutnarrativesandsubjectiveessayswouldalsobeinteresting
andeducationallyrelevant,weleavetheseproblemstofuturework. TomakeourfactualQGsystem
generallyuseful,weavoidtheuseofdomain-specificknowledge(e.g.,abouthistoricaleventsorge-
ographicallocations)andinsteadfocusonmodelingfairlygenerallexicalandsyntacticphenomena
relatedtoquestionsandthepresentationoffactualinformation.
1.1 Illustrative Example of Factual Question Generation
Inthissection,weprovideexamplesthatillustratethatQGaboutexplicitfactualinformationisa
challengingbutstillfeasibletaskgivencurrentnaturallanguageprocessing(NLP)technologies.
Webeginwitharelativelystraightforwardexample,takenfromanEncyclopediaBritannicaEle-
mentaryEditionarticleaboutthecityofMonrovia.1
(1.1) ...MonroviawasnamedafterJamesMonroe,whowaspresidentoftheUnitedStatesin1822.
InthatyearagroupoffreedU.S.slaves,sponsoredbyaU.S.society,startedanewsettlement
onthecontinentoftheirancestors. AsmoresettlersarrivedfromtheUnitedStatesandfrom
theCaribbeanarea,theareatheycontrolledgrewlarger. In1847Monroviabecamethecapital
ofthenewcountryofLiberia. ...
Anumberofacceptablefactualquestionscanbegeneratedfromthissentencebyanalyzingits
grammaticalstructure,labelingitslexicalitemswithhigh-levelsemantictypes(e.g.,person,loca-
tion,time),andthenperformingsyntactictransformationssuchassubject-auxiliaryinversionand
WH-movement. Fromthefirstsentence,wecanextractwell-formedandspecificquestionssuchas
thefollowing:
(1.2) WhowaspresidentoftheUnitedStatesin1822?
(1.3) WhenwasJamesMonroepresidentoftheUnitedStates?
(1.4) WhowasMonrovianamedafter?
1WemakeuseofadatasetofEncyclopediaBritannicatextsfrompreviousNLPresearch(BarzilayandElhadad,2003).
WedescribethedatasetinmoredetailinChapter4.
2