AMUSE: Multilingual Semantic Parsing for Question Answering over Linked Data

16 Pages·2017·0.44 MB·English

AMUSE: Multilingual Semantic Parsing for Question
Answering over Linked Data
SherzodHakimov,SoufianJebbara,PhilippCimiano
{shakimov, sjebbara, cimiano} @ cit-ec.uni-bielefeld.de
SemanticComputingGroup
CognitiveInteractionTechnology–CenterofExcellence(CITEC)
BielefeldUniversity
33615Bielefeld,Germany
Abstract. ThetaskofansweringnaturallanguagequestionsoverRDFdatahas
received wIde interest in recent years, in particular in the context of the series
of QALD benchmarks. The task consists of mapping a natural language ques-
tiontoanexecutableform,e.g.SPARQL,sothatanswersfromagivenKBcan
be extracted. So far, most systems proposed are i) monolingual and ii) rely on
a set of hard-coded rules to interpret questions and map them into a SPARQL
query. We present the first multilingual QALD pipeline that induces a model
fromtrainingdataformappinganaturallanguagequestionintologicalformas
probabilisticinference.Inparticular,ourapproachlearnstomapuniversalsyn-
tacticdependencyrepresentationstoalanguage-independentlogicalformbased
onDUDES(Dependency-basedUnderspecifiedDiscourseRepresentationStruc-
tures)thatarethenmappedtoaSPARQLqueryasadeterministicsecondstep.
Ourmodelbuildsonfactorgraphsthatrelyonfeaturesextractedfromthedepen-
dencygraphandcorrespondingsemanticrepresentations.Werelyonapproximate
inferencetechniques,MarkovChainMonteCarlomethodsinparticular,aswell
asSampleRanktoupdateparametersusingarankingobjective.Ourfocuslieson
developingmethodsthatovercomethelexicalgapandpresentanovelcombina-
tionofmachinetranslationandwordembeddingapproachesforthispurpose.As
aproofofconceptforourapproach,weevaluateourapproachontheQALD-6
datasetsforEnglish,German&Spanish.
Keywords: questionanswering,multilinguality,QALD,probabilisticgraphical
models,factorgraphs
1 Introduction
ThetaskofQuestionAnsweringoverLinkedData(QALD)hasreceivedincreasedat-
tentionoverthelastyears(seethesurveys[14]and[36]).Thetaskconsistsinmapping
natural language questions into an executable form, e.g. a SPARQL query in particu-
lar,thatallowstoretrieveanswerstothequestionfromagivenknowledgebase.Con-
sIderthequestion:WhocreatedWikipedia?,whichcanbeinterpretedasthefollowing
SPARQLquerywithrespecttoDBpedia1:
1The prefixes dbo and dbr stand for the namespaces http://dbpedia.org/ontology and
http://dbpedia.org/resource/,respectively.
2
SELECT DISTINCT ?uri WHERE { dbr:Wikipedia dbo:author ?uri .}
AnimportantchallengeinmappingnaturallanguagequestionstoSPARQLqueries
lies in overcoming the so called ‘lexical gap’ (see [13], [14]). The lexical gap makes
interpretingtheabovementionedquestioncorrectlychallenging,asthereisnosurface
relationbetweenthequerystringcreatedandtheURIlocalnameauthor.TobrIdgethe
lexical gap, systems need to infer that create should be interpreted as author in the
abovecase.
ThelexicalgapisonlyexacerbatedwhenconsIderingmultiplelanguagesasweface
across-lingualgapthatneedstobebrIdged.ConsIderforinstancethequestion:Werhat
Wikipedia gegru¨ndet?, which involves mapping gru¨nden to author to successfully
interpretthequestion.
Addressing the lexical gap in question answering over linked data, we present a
newsystemwecallAMUSEthatreliesonprobabilisticinferencetoperformstructured
predictioninthesearchspaceofpossibleSPARQLqueriestopredictthequerythathas
thehighestprobabilityofbeingthecorrectinterpretationofthegivenquerystring.As
themaincontributionofthepaper,wepresentanovelapproachtoquestionanswering
over linked data that relies on probabilistic inference to determine the most probable
meaningofaquestiongivenamodel.Theparametersofthemodelareoptimizedona
giventrainingdatasetconsistingofnaturallanguagequestionswiththeircorresponding
SPARQLqueriesasprovIdedbytheQALDbenchmark.Theinferenceprocessbuilds
on approximate inference techniques, Markov Chain Monte Carlo in particular, to as-
signknowledgebase(KB)Identifiersaswellasmeaningrepresentationstoeverynode
in a dependency tree representing the syntactic dependency structure of the question.
Onthebasisoftheseassignedmeaningrepresentationstoeverynode,afullsemantic
representationcanbecomputedrelyingonbottom-upsemanticcompositionalongthe
parsetree.Asanovelty,ourmodelcanbetrainedondifferentlanguagesbyrelyingon
universaldependencies.Toourknowledge,thisisthefirstsystemforquestionanswer-
ingoverlinkeddatathatcanbetrainedtoperformondifferentlanguages(threeinour
case)withouttheneedofimplementinganylanguage-specificheuristicsorknowledge.
Toovercomethecross-linguallexicalgap,weexperimentwithautomaticallytranslated
labelsandrelyonanembeddingapproachtoretrievesimilarwordsintheembedding
space.Weshowthatbyusingwordembeddingsonecaneffectivelycontributetoreduc-
ingthelexicalgapcomparedtoabaselinesystemwhereonlyknownlabelsareused.
2 Approach
Our intuition in this paper is that the interpretation of a natural language question in
terms of a SPARQL query is a compositional process in which partial semantic rep-
resentationsarecombinedwitheachotherinabottom-upfashionalongadependency
treerepresentingthesyntacticstructureofagivenquestion.Insteadofrelyingonhand-
crafted rules guIding the composition, we rely on a learning approach that can infer
such ‘rules’ from training data. We employ a factor graph model that is trained using
arankingobjectiveandSampleRankastrainingproceduretolearnamodelthatlearns
to prefer good over bad interpretations of a question. In essence, an interpretation of
3
aquestionrepresentedasadependencytreeconsistsofanassignmentofseveralvari-
ables:i)aKBIdandsemantictypetoeverynodeintheparsetree,andii)anargument
index(1or2)toeveryedgeinthedependencytreespecifyingwhichslotoftheparent
node,subjectorobject,thechildnodeshouldbeappliedto.Theinputtoourapproach
isthusasetofpairs(q,sp)ofquestionq andSPARQLquerysp.Asanexample,con-
sIderthefollowingquestionsinEnglish,German&Spanish:WhocreatedWikipedia?
WerhatWikipediagegru¨ndet?¿Quie´ncreo´ Wikipedia?respectively.Independentlyof
the language they are expressed in, the three question can be interpreted as the same
SPARQLqueryfromtheintroduction.
OurapproachconsistsoftwoinferencelayerswhichwecallL2KBandQC.Each
of these layers consists of a different factor graph optimized for different subtasks of
theoveralltask.Thefirstinferencelayeristrainedusinganentitylinkingobjectivethat
learnstolinkpartsofthequerytoKBIdentifiers.Inparticular,thisinferencestepas-
signs KB Identifiers to open class words such as nouns, proper nouns, adjectives and
verbs etc. In our case, the knowledge base is DBpedia. We use Universal Dependen-
cies2[28]togetdependencyparsetreesfor3languages.Thesecondinferencelayeris
aqueryconstructionlayerthattakesthetopkresultsfromtheL2KBlayerandassigns
semanticrepresentationstoclosedclasswordssuchasquestionpronouns,determiners,
etc.toyieldalogicalrepresentationofthecompletequestion.Theapproachistrained
ontheQALD-6traindatasetforEnglish,German&Spanishquestionstooptimizethe
parameters of the model. The model learns mappings between the dependency parse
tree for a given question text and RDF nodes in the SPARQL query. As output, our
system produces an executable SPARQL query for a given NL question. All data and
source code are freely available3. As semantic representations, we rely on DUDES,
whicharedescribedinthefollowingsection.
2.1 DUDES
DUDES(Dependency-basedUnderspecifiedDiscourseRepresentationStructures)[9]
isaformalismforspecifyingmeaningrepresentationsandtheircomposition.Theyare
based on Underspecified Discourse Representation Theory (UDRT) [33,10], and the
resultingmeaningrepresentations.Formally,aDUDEisdefinedasfollows:
Definition1. ADUDEisa5-tuple(v,vs,l,drs,slots)where
– visthemainvariableoftheDUDES
– vsisa(possiblyempty)setofvariables,theprojectionvariables
– listhelabelofthemainDRS
– drsisaDRS(themainsemanticcontentoftheDUDE)
– slotsisa(possiblyempty)setofsemanticdependencies
The core of a DUDES is thus a Discourse Representation Structure (DRS) [15].
Themainvariablerepresentsthevariabletobeunifiedwithvariablesinslotsofother
DUDESthattheDUDEinquestionisinsertedinto.EachDUDEcapturesinformation
aboutwhichsemanticargumentsarerequiredforaDUDEtobecompleteinthesense
2http://universaldependencies.org/v2,70treebanks,50languages
3https://github.com/ag-sc/AMUSE
4
that all slots have been filled. These required arguments are modeled as set of slots
thatarefilledvia(functional)applicationofotherDUDES.Theprojectionvariablesare
relevantinmeaningrepresentationsofquestions;theyspecifywhichentityisaskedfor.
When converting DUDES into SPARQL queries, they will directly correspond to the
variables in the SELECT clause of the query. Finally, slots capture information about
whichsyntacticelementsmaptowhichsemanticargumentsintheDUDE.
Asbasicunitsofcomposition,weconsider5pre-definedDUDEStypesthatcorre-
spondtodataelementsinRDFdatasets.WeconsiderResourceDUDESthatrepresent
resourcesorindividualsdenotedbypropernounssuchasWikipedia(see1stDUDESin
Figure1).WeconsiderClassDUDESthatcorrespondtosetsofelements,i.e.classes,
forexampletheclassofPersons(see2ndDUDESinFigure1).WealsoconsiderProp-
erty DUDES that correspond to object or datatype properties such as author (see 3rd
DUDESinFigure1).Wefurtherconsiderrestrictionclassesthatrepresentthemeaning
ofintersectiveadjectivessuchasSwedish(see4thDUDESinFigure1).Finally,aspe-
cialtypeofDUDEScanbeusedtocapturethemeaningofquestionpronouns,e.g.Who
orWhat(see5thDUDESinFigure1).
Fig.1:Exampelesforthe5typesofDUDES
When applying a DUDE d to d where d subcategorizes a number of semantic
2 1 1
arguments,weneedtoindicatewhichargumentd fills.Forinstance,applyingthe1st
2
DUDES in Figure 1 to the 3rd DUDES in Figure 1 at argument index 1 yields the
followingDUDE:
v:-vs:{}l:1
1:
dbo:author(dbr:Wikipedia,y)
(y,a2,2)
2.2 ImperativelyDefinedFactorGraphs
Inthissection,weintroducetheconceptoffactorgraphs[19],followingthenotationsin
[41]and[17].AfactorgraphGisabipartitegraphthatdefinesaprobabilitydistribution
π.ThegraphconsistsofvariablesV andfactorsΨ.Variablescanbefurtherdividedinto
sets of observed variables X and hidden variables Y. A factor Ψ connects subsets of
i
observedvariablesx andhiddenvariablesy ,andcomputesascalarscorebasedonthe
i i
exponentialofthescalarproductofafeaturevectorf (x ,y )andasetofparametersθ :
i i i i
5
Ψi = efi(xi,yi)·θi.Theprobabilityofthehiddenvariablesgiventheobservedvariables
istheproductoftheindividualfactors:
1 (cid:89) 1 (cid:89)
π(y|x;θ)= Ψ (x ,y )= efi(xi,yi)·θi (1)
Z(x) i i i Z(x)
Ψi∈G Ψi∈G
where Z(x) is the partition function. For a given input consisting of a dependency
parsed sentence, the factor graph is rolled out by applying template procedures that
matchoverpartsoftheinputandgeneratecorrespondingfactors.Thetemplatesarethus
imperativelyspecifiedproceduresthatrolloutthegraph.AtemplateT ∈T definesthe
j
subsetsofobservedandhiddenvariables(x(cid:48),y(cid:48))withx(cid:48) ∈ X andy(cid:48) ∈ Y forwhich
j j
itcangeneratefactorsandafunctionf (x(cid:48),y(cid:48))togeneratefeaturesforthesevariables.
j
Additionally,allfactorsgeneratedbyagiventemplateT sharethesameparametersθ .
j j
Withthisdefinition,wecanreformulatetheconditionalprobabilityasfollows:
π(y|x;θ)= 1 (cid:89) (cid:89) efj(x(cid:48),y(cid:48))·θj (2)
Z(x)
Tj∈T (x(cid:48),y(cid:48))∈Tj
Input to our approach is a pair (W,E) consisting of a sequence of words W =
{w ,...,w } and a set of dependency edges E ⊆ W ×W forming a tree. A state
1 n
(W,E,α,β,γ)representsapartialinterpretationoftheinputintermsofpartialseman-
ticrepresentations.Thepartialfunctions α : W → KB,β : W → {t ,t ,t ,t ,t }
1 2 3 4 5
andγ :E →{1,2}mapwordstoKBidentifiers,wordstothefivebasicDUDEStypes,
and edges to indices of semantic arguments, with 1 corresponding to the subject of a
property and 2 corresponding to the object, respectively. Figure 2 shows a schematic
visualizationofaquestionalongwithitsfactorgraph.Factorsmeasurethecompatibil-
itybetweendifferentassignmentsofobservedandhiddenvariables.Theinterpretation
of a question is the one that maximizes the posterior of a model with parameters θ:
y∗ =argmax π(y|x;θ).
y
2.3 Inference
Werelyonanapproximateinferenceprocedure,MarkovChainMonteCarloinparticu-
lar[1].Themethodperformsiterativeinferenceforexploringthestatespaceofpossible
questioninterpretationsbyproposingconcretechangestosetsofvariablesthatdefine
aproposaldistribution.Theinferenceprocedureperformsaniterativelocalsearchand
canbedividedinto(i)generatingpossiblesuccessorstatesforagivenstatebyapplying
changes,(ii)scoringthestatesusingthemodelscore,and(iii)decidingwhichproposal
to accept as successor state. A proposal is accepted with a probability that is propor-
tionaltothelikelihoodassignedbythedistributionπ.Tocomputethelogicalformofa
question,weruntwoinferenceproceduresusingtwodifferentmodels.Thefirstmodel
L2KB is trained using a linking objective that learns to map open class words to KB
identifiers.TheMCMCsamplingprocessisrunformstepsfortheL2KBmodel;the
top k states are used as an input for the second inference model called QC that as-
signsmeaningstoclosedclasswordstoyieldafullfledgedsemanticrepresentationof
the question. Both inference strategies generate successor states by exploration based
6
Fig.2:Factorgraphforthequestion:WhocreatedWikipedia?.Observedvariablesare
depictedasbubbleswithstraightlines;hiddenvariablesasbubbleswithdashedlines.
Blackboxesrepresentfactors.
onedgesinthedependencyparsetree.Weexploreonlythefollowingtypesofedges:
Corearguments,Non-coredependents,NominaldependentsdefinedbyUniversalDe-
pendencies4andnodesthathavethefollowingPOStags:NOUN,VERB,ADJ,PRON,
PROPN, DET. In both inference models, we alternate across iterations between using
theprobabilityofthestategiventhemodelandtheobjectivescoretodecidewhichstate
toaccept.Initially,allpartialassignmentsα ,β ,γ .areempty.
0 0 0
We rely on an inverted index to find all KB IDs for a given query term. The in-
verted index maps terms to candidate KB IDs for all 3 languages. It has been created
takingintoaccountanumberofresources:namesofDBpediaresources,Wikipediaan-
chor texts and links, names of DBpedia classes, synonyms for DBpedia classes from
WordNet [26,16], as well as lexicalizations of properties and restriction classes from
DBlexipedia[40].EntriesintheindexaregroupedbyDUDEStype,sothatitsupports
type-specificretrieval.TheindexstoresthefrequencyofthementionspairedwithKB
ID.Duringretrieval,theindexreturnsanormalizedfrequencyscoreforeachcandidate
KBID.
L2KB:LinkingtoKnowledgeBase ProposalGeneration:TheL2KBproposalgen-
erationproposeschangestoagivenstatebyconsideringsingledependencyedgesand
changing:i)theKBIDsofparentandchildnodes,ii)theDUDEStypeofparentand
childnodes,andiii)theargumentindexattachedtotheedge.TheSemanticTypevari-
ablesrangeoverthe5basicDUDEStypesdefined,whiletheargumentindexvariable
ranges in the set {1,2}. The resulting partial semantic representations for the depen-
4http://universaldependencies.org/u/dep/index.html
7
dency edge are checked for satisfiability with respect to the knowledge base, pruning
theproposalifitisnotsatisfiable.Figure3depictsthelocalexplorationofthedobj-edge
betweenWikipediaandcreated.Theleftimageshowsaninitialstatewithemptyassign-
mentsforallhiddenvariables.TherightimageshowsaproposalthatischangedtheKB
IDsandDUDEtypesofthenodesconnectsbythedobjedge.Theinferenceprocesshas
assigned the KB ID dbo:author and the Property DUDES type to the created node.
The Wikipedia nodes gets assigned the type Resource DUDES as well as the KB ID
dbr:Wikipedia.Thedependencyedgegetsassignedtheargumentindex1,representing
thatdbr:Wikipediashouldbeinsertedatthesubjectpositionofthedbo:authorproperty.
Thepartialsemanticrepresentationrepresentedbythisedgeistheonedepictedatthe
endofSection2.2.Asitissatisfiable,itisnotpruned.Incontrast,astateinwhichthe
edge is assigned the argument index 2 would yield the following non-satisfiable rep-
resentation,correspondingtothingsthatwereauthoredbyWikipediainsteadofthings
thatauthoredWikipedia:
v:-vs:{}l:1
1:
dbo:author(y,dbr:Wikipedia)
(y,a2,2)
Fig.3:Left:InitialstatebasedondependencyparsewhereeachnodehasemptyKBID
andSemanticType.Right:ProposalgeneratedbytheLKBproposalgenerationforthe
questionWhocreatedWikipedia?
ObjectiveFunction: AsobjectivefortheL2KBmodelwerelyonalinkingobjective
that calculates the overlap between inferred entity links and entity links in the gold
standardSPARQLquery.
Allgeneratedstatesarerankedbytheobjectivescore.Top-kstatesarepassedtothe
next sampling step. In the next iteration, the inference is performed on these k states.
Followingthisprocedureformiterationsyieldsasequenceofstates(s ,...,s )that
0 m
aresampledfromthedistributiondefinedbytheunderlyingfactorgraphs.
QC: Query Construction Proposal Generation: Proposals in this inference layer
consistofassignmentsofthetypeQueryVarDUDEStonodesforclasswords,inpar-
ticular determiners, that could fill the argument position of a parent with unsatisfied
arguments.
8
ObjectiveFunction: Asobjectiveweuseanobjectivefunctionthatmeasuresthe(graph)
similaritybetweentheinferredSPARQLqueryandthegoldstandardSPARQLquery.
Figure4showsaninputstateandasampledstatefortheQCinferencelayerofour
examplequery:WhocreatedWikipedia?.Theinitialstate(seeLeft)hasSlot1assigned
to the edge dobj. Property DUDES have 2 slots by definition. The right figure shows
aproposedstateinwhichtheargumentslot2hasbeenassignedtothensubjedgeand
the QueryVar DUDES type has been assigned to node Who. This corresponds to the
representationandSPARQLqueriesbelow:
v:-vs:{y}l:1
1:
dbo:author(dbr:Wikipedia,y)
SELECT DISTINCT ?y WHERE { dbr:Wikipedia dbo:author ?y .}
Fig.4:Left:Inputstate;Right:ProposalgeneratedbytheQCproposalgenerationfor
thequestionWhocreatedWikipedia?
2.4 Features
Asfeaturesforthefactors,weuseconjunctionsofthefollowinginformation:i)lemma
ofparentandchildnodes,ii)KBIdsofparentandchildnodes,iii)POStagsofparent
andchildnodes,iv)DUDEtypeofparentandchild,v)indexofargumentatedge,vi)
dependencyrelationofedge,vii)normalizedfrequencyscoreforretrievedKBIds,viii)
string similarity between KB Id and lemma of node, ix) rdfs:domain and rdfs:range
restrictionsfortheparentKBId(incaseofbeingaproperty).
2.5 LearningModelParameters
Inordertooptimizeparametersθ,weuseanimplementationoftheSampleRank[41]
algorithm.TheSampleRankalgorithmobtainsgradientsfortheseparametersfrompairs
ofconsecutivestatesinthechainbasedonapreferencefunctionPdefinedintermsof
theobjectivefunctionOasfollows:
(cid:40)
1, ifO(s(cid:48))>O(s)
P(s(cid:48),s)= (3)
0, otherwise
9
We have observed that accepting proposals only on the basis of the model score
requires a large number of inference steps. This is due to the fact that the exploration
spaceishugeconsideringallthecandidateresources,predicates,classesetc.inDBpe-
dia. To guide the search towards good solutions, we switch between model score and
objectivescoretocomputethelikelihoodofacceptanceofaproposal.Oncethetraining
procedureswitchesthescoringfunctioninthenextsamplingstep,themodelusesthe
parametersfromtheprevioussteptoscorethestates.
2.6 Addressingthelexicalgap
AkeycomponentintheproposedquestionansweringpipelineistheL2KBlayer.This
layerisresponsibleforproposingpossibleKBidentifiersforpartsofthequestion.Con-
siderthequestionWhoisthewriterofTheHungerGames?Itseemstobeatrivialtask
to link the query word writer to the appropriate identifier dbo:author, however it
stillrequirespriorknowledgeaboutthesemanticsofthequerywordandtheKBentry
(e.g.thatthewriterofabookistheauthor).
Toaddressthelexicalgap,werelyontheonehandonlexicalizationsofDBpedia
propertiesasextractedbyM-ATOLL[39,40]formultiplelanguages5.Inparticularfor
Spanish and German, however, M-ATOLL produces very sparse results. We propose
twosolutionstoovercomethelexicalgapbyusingmachinetranslationtotranslateEn-
glishlabelsintootherlanguagesaswellasusingwordembeddingstoretrievecandidate
propertiesforagivenmentiontext.
Machine Translations We rely on the online dictionary Dict.cc6 as our translation
engine.WequerythewebserviceforeachavailableEnglishlabelandtargetlanguage
and store the obtained translation candidates as new labels for the respective entity
andlanguage.Whilethesetranslationsarepronetobenoisywithoutapropercontext,
we receive a reasonable starting point for the generation of candidate lexicalizations,
especiallyincombinationwiththewordembeddingapproach.
Word Embedding Retrieval Many word embedding methods such as the skip-gram
method[25]havebeenshowntoencodeusefulsemanticandsyntacticproperties.The
objective of the skip-gram method is to learn word representations that are useful for
predictingcontextwords.Asaresult,thelearnedembeddingsoftendisplayadesirable
linear structure that can be exploited using simple vector addition. Motivated by the
compositionality of word vectors, we propose a measure of semantic relatedness be-
tween a mention m and a DBpedia entry e using the cosine similarity between their
respectivevectorrepresentationsv andv .Forthiswefollowtheapproachin[5]to
m e
deriveentityembeddingvectorsfromwordvectors:Wedefinethevectorofamention
masthesumofthevectorsofitstokens7v =(cid:80) v ,wherethev arerawvectors
m t∈m t t
fromthesetofpretrainedskip-gramvectors.Similarly,wederivethevectorrepresenta-
tionofaDBpediaentryebyaddingtheindividualwordvectorsfortherespectivelabel
(cid:80)
l ofe,thusv = v .
e e t∈le t
5M-ATOLLcurrentlyprovideslexicalizationsforEnglish,GermanandSpanish
6http://www.dict.cc
7Weomitallstopwordtokens.
10
As an example, the vector for the mention text movie director is composed as
v = v + v . The DBpedia entry dbo:director has the
moviedirector movie director
labelfilmdirectorandisthuscomposedofv =v +v .
dbo:director film director
Togeneratepotentiallinkingcandidatesgivenamentiontext,wecancomputethe
cosinesimilaritybetweenv andeachpossiblev asameasureofsemanticrelatedness
m e
andthusproducearankingofallcandidateentries.Bypruningtherankingatachosen
threshold,wecancontroltheproducedcandidatelistforprecisionandrecall.
For this work, we trained 3 instances of the skip-gram model with each 100 di-
mensionsontheEnglish,GermanandSpanishWikipediarespectively.Followingthis
approach, the top ranking DBpedia entries for the mention text total population are
listedbelow:
Mention DBpediaentry Cos.Similarity
totalpopulationdbo:populationTotal 1.0
dbo:totalPopulation 1.0
dbo:agglomerationPopulationTotal 0.984
dbo:populationTotalRanking 0.983
dbo:PopulatedPlace/areaTotal 0.979
AmoredetailedevaluationisconductedinSection3whereweinvestigatethecan-
didateretrievalincomparisontoanM-ATOLLbaseline.
3 ExperimentsandEvaluation
WepresentexperimentscarriedoutontheQALD-6datasetcomprisingofEnglish,Ger-
man&Spanishquestions.Wetrainandtestonthemultilingualsubtask.Thisyieldsa
trainingdatasetconsistingof350and100testinstances.Wetrainthemodelwith350
training instances for each language from QALD-6 train dataset by performing 10 it-
erationsoverthedatasetwithlearningratesetto0.01tooptimizetheparameters.We
setk to10.Weperformapreprocessingsteponthedependencyparsetreebeforerun-
ningthroughthepipeline.Thisstepconsistsofmergingnodesthatareconnectedwith
compoundedges.Thisresultsinhavingonenodeforcompoundnamesandreducesthe
traversingtimeandcomplexityforthemodel.Theapproachisevaluatedontwotasks:
alinkingtaskandaquestionansweringtask.Thelinkingtaskisevaluatedbycompar-
ing the proposed KB links to the KB elements contained in the SPARQL question in
terms of F-Measure. The question answering task is evaluated by executing the con-
structed SPARQL query over the DBpedia KB, and comparing the retrieved answers
withanswersretrievedforthegoldstandardSPARQLqueryintermsofF-Measure.
Before evaluating the full pipeline on the QA task, we evaluate the impact of us-
ingdifferentlexicalresourcesincludingthewordembeddingtoinferunknownlexical
relations.
3.1 EvaluatingtheLexiconGeneration
We evaluate the proposed lexicon generation methods using machine translation and
embeddingswithrespecttoalexiconofmanualannotationsthatareobtainedfromthe
training set of the QALD-6 dataset. The manual lexicon is a mapping of mention to
expected KB entry derived from the (question-query) pairs in QALD-6 dataset. Since