Scalable and Distributed Methods for Entity Matching - Aidan Hogan

47 Pages·2011·0.75 MB·English

ScalableandDistributedMethodsforEntityMatching,
ConsolidationandDisambiguationoverLinkedDataCorpora
AidanHogana,AntoineZimmermannb,JürgenUmbricha,AxelPolleresc,StefanDeckera,
aDigitalEnterpriseResearchInstitute,NationalUniversityofIreland,Galway
bINSA-Lyon,LIRIS,UMR5205,F-69621,France
cSiemensAGÖsterreich,Siemensstrasse90,1210Vienna,Austria
Abstract
With respect to large-scale, static, Linked Data corpora, in this paper we discuss scalable and distributed methods for entity
consolidation (aka. smushing, entity resolution, object consolidation, etc.) to locate and process names that signify the same
entity.Weinvestigate(i)abaselineapproach,whichusesexplicitowl:sameAsrelationstoperformconsolidation;(ii)extended
entityconsolidationwhichadditionallyusesasubsetofOWL2RL/RDFrulestoderivenovelowl:sameAsrelationsthroughthe
semantics of inverse-functional properties, functional-properties and (max-)cardinality restrictions with value one; (iii) deriving
weighted concurrence measures between entities in the corpus based on shared inlinks/outlinks and attribute values using
statisticalanalyses;(iv)disambiguating(initially)consolidatedentitiesbasedoninconsistencydetectionusingOWL2RL/RDF
rules. Our methods are based upon distributed sorts and scans of the corpus, where we deliberately avoid the requirement for
indexing all data. Throughout, we offer evaluation over a diverse Linked Data corpus consisting of 1.118 billion quadruples
derived from a domain-agnostic, open crawl of 3.985 million RDF/XML Web documents, demonstrating the feasibility of our
methodsatthatscale,andgivinginsightsintothequalityoftheresultsforreal-worlddata.
Keywords: entityconsolidation,webdata,linkeddata,rdf
1. Introduction TheLinkedOpenDataprojecthasadvocatedthegoal
of providing dereferencable machine readable data in
Over a decade since the dawn of the Semantic Web, a common format (RDF), with emphasis on the re-use
RDFpublishinghasbeguntofindsometractionthrough ofURIsandinter-linkagebetweenremotedatasets—in
adoptionofLinkedDatabestpracticesasfollows: so doing, the project has overseen exports from corpo-
(i) useURIsasnamesforthings(andnotjustdocu- rate entities (e.g., the BBC, BestBuy, Freebase), gov-
ments); ernmental bodies (e.g., the UK Government, the US
(ii) makethoseURIsdereferencableviaHTTP; government), existing structured datasets (e.g., DBPe-
(iii) return useful and relevant RDF content upon dia), social networking sites (e.g., flickr, Twitter, live-
lookupofthoseURIs; journal),academiccommunities(e.g.,DBLP,UniProt),
(iv) includelinkstootherdatasets. aswellasesotericexports(e.g.,LinkedOpenNumbers,
Poképédia).Thisburgeoningwebofstructureddatahas
succinctlybeendubbedthe“WebofData”.
Emailaddresses:[email protected](AidanHogan),
Considering the merge of these structured exports,
[email protected](AntoineZimmermann),
[email protected](JürgenUmbrich), at a conservative estimate there now exists somewhere
[email protected](AxelPolleres), in the order of thirty billion RDF triples published on
[email protected](StefanDecker).
PreprintsubmittedtoElsevier 20/10/11
the Web as Linked Data.1 However, in this respect, fyingandcanonicalisingequivalentidentifiers—thein-
size isn’t everything [73]. In particular, although the dexedcorpus:therewasanobservablelackofURIssuch
situation is improving, individual datasets are still not that coreferent blank-nodes were prevalent [30] even
well-interlinked (cf. [72])—without sufficient linkage, within the same dataset, and thus we observed many
the ideal of a “Web of Data” quickly disintegrates into duplicateresultsreferringtothesamething,leadingto
thecurrentrealityof“ArchipelagosofDatasets”. poorintegrationofdatafromoursourcedocuments.
There have been numerous works that have looked To take a brief example, consider a simple exam-
atbridgingthearchipelagos.Someworksaimatalign- plequery:“WHO DOES TIM BERNERS-LEE KNOW?”.
ingasmallnumberofrelateddatasets(e.g.,[48,58,49]), Knowing that Tim uses the URI timblfoaf:i to refer
thus focusing more on theoretical considerations than tohimselfinhispersonalFOAFprofiledocument,and
scalability, usually combining symbolic (e.g., reason- againknowingthatthepropertyfoaf:knowsrelatespeo-
ing with consistency checking) methods and similarity pletotheir(reciprocated)acquaintances,wecanformu-
measures. Some authors have looked at inter-linkage latethisrequestastheSPARQLquery[53]asfollows:
of domain specific RDF datasets at various degrees
of scale (e.g., [61,59,38,54,46]). Further research has SELECT?person
WHERE{
also looked at exploiting shared terminological data— timblfoaf:ifoaf:knows?person.
as well as explicitly asserted links—to better integrate }
Linked Data collected from thousands or millions of
However,otherpublishersusedifferentURIstoiden-
sources(e.g.,[30,50,15,35]);theworkpresentedherein
tify Tim, where to get more complete answers across
falls most closely into this category. One approach has
these naming schemes, the SPARQL query must use
tackledtheproblemfromthepublishersside,detailinga
disjunctiveUNIONclausesforeachknownURI;herewe
system for manually specifying some (possibly heuris-
giveanexampleusingasampleofidentifiersextracted
tic)criteriaforcreatinglinksbetweentwodatasets[72].
fromarealLinkedDatacorpus(introducedlater):
WeleavefurtherdetailedrelatedworktoSection9.
In this paper, we look at methods to provide bet-
SELECT?person
terlinkagebetweenresources,inparticularfocusingon WHERE{
{timblfoaf:ifoaf:knows?person.}
findingequivalententitiesinthedata.Ournotionofan UNION{identicau:45563foaf:knows?person.}
entity is a representation of something being described UNION{dbpedia:Berners-Leefoaf:knows?person.}
UNION{dbpedia:Dr._Tim_Berners-Leefoaf:knows?person.}
bythedata;e.g.,aperson,aplace,amusician,aprotein, UNION{dbpedia:Tim-Berners_Leefoaf:knows?person.}
UNION{dbpedia:TimBLfoaf:knows?person.}
etc. We say that two entities are equivalent if they are UNION{dbpedia:Tim_Berners-Leefoaf:knows?person.}
coreferent; e.g., refer to the same person, place, etc.2 UNION{dbpedia:Tim_berners-leefoaf:knows?person.}
UNION{dbpedia:Timblfoaf:knows?person.}
Givenacollectionofdatasetsthatspeakaboutthesame UNION{dbpedia:Timothy_Berners-Leefoaf:knows?person.}
UNION{yagor:Tim_Berners-Leefoaf:knows?person.}
referentsusingdifferentidentifiers,wewishtoidentify UNION{fb:en.tim_berners-leefoaf:knows?person.}
UNION{swid:Tim_Berners-Leefoaf:knows?person.}
thesecoreferencesandsomehowmergetheknowledge UNION{dblpperson:100007foaf:knows?person.}
contribution provided by the distinct parties. We call UNION{avtimbl:mefoaf:knows?person.}
UNION{bmpersons:Tim+Berners-Leefoaf:knows?person.}
thismergeconsolidation. ...
}
Inparticular,ourworkisinspiredbytherequirements
oftheSemanticWebSearchEngineproject[32],within WeseedisparateURIsnotonlyacrossdatapublishers,
which we aim to offer search and browsing over large, but also within the same namespace. Clearly, the ex-
static, Linked Data corpora crawled from the Web.3 pandedqueryquicklybecomesextremelycumbersome.
The core operation of SWSE is to take user keyword Inthispaper,welookatbespokemethodsforidenti-
queriesasinput,andtogeneratearankedlistofmatch- fyingandprocessingcoreferenceinamannersuchthat
ing entities as results. After the core components of a the resultant corpus can be consumed as if more com-
crawler, index and user-interface, we saw a clear need plete agreement on URIs was present; in other words,
foracomponentthatconsolidates—bymeansofidenti- using standard query-answering techniques, we want
theenhancedcorpustoreturnthesameanswersforthe
1 http://www4.wiwiss.fu-berlin.de/lodcloud/state/ originalsimplequeryasforthelatterexpandedquery.
2 Herein,weavoidphilosophicaldiscussiononthenotionofiden- Our core requirements for the consolidation compo-
tity;forinterestingdiscussionthereon,see[26].
nentareasfollows:
3 Bystatic,wemeanthatthesystemdoesnotcaterforupdates;this
– the component must give high precision of consoli-
omission allows for optimisations throughout the system. Instead,
we aim at a cyclical indexing paradigm, where new indexes are datedresults;
bulk-loadedinthebackgroundonseparatemachines.
2
– theunderlyingalgorithm(s)must bescalable; the data described in RDF(S)/OWL and also, generic
– theapproachmust befullyautomatic; statisticsderivablefromthecorpus.Inordertoachieve
– themethodsmust bedomainagnostic; high recall, we incorporate additional OWL features
whereacomponentwithpoorprecisionwillleadtogar- tofindnovelcoreferencethroughreasoning.Aimingat
bledfinalresultsmergingunrelatedentities,wherescal- highprecision,weintroducemethodsthatagainexploit
abilityisrequiredtoapplytheprocessoverourcorpora the semantics and also the statistics of the data, but to
typicallyintheorderofabillionstatements(andwhich converselydisambiguateentities:todefeatequivalences
we feasibly hope to expand in future), where the scale found in the previous step that are unlikely to be true
ofthecorporaunderanalysisprecludesanymanualin- accordingtosomecriteria.
tervention, and where—for the purposes of research— As such, extending upon various reasoning and
the methods should not give preferential treatment to consolidation techniques described in previous
any domain or vocabulary of data (other than core works [30,31,33,34], we now give a self-contained
RDF(S)/OWLterms).Alongsidetheseprimaryrequire- treatment of our results in this area. In addition, we
ments, we also identify the following secondary crite- distribute the execution of all methods over a cluster
ria: of commodity hardware, we provide novel, detailed
– theanalysisshould demonstratehighrecall; performance and quality evaluation over a large, real-
– theunderlyingalgorithm(s)should beefficient; world Linked Data corpus of one billion statements,
where the consolidation component should identify as and we also examine exploratory techniques for inter-
many (correct) equivalences as possible, and where linking similar entities based on statistical analyses,
the algorithm should be applicable in reasonable time. as well as methods for disambiguating and repairing
Clearlythesecondaryrequirementsarealsoimportant, incorrectcoreferences.
buttheyaresupercededbythosegivenearlier,wherea
Insummary,inthispaperwe:
certaintrade-offexists:wepreferasystemthatgivesa
– provide some necessary preliminaries and describe
high percentage of correct results and leads to a clean
ourdistributedarchitecture(Section2);
consolidatedcorpusoveranapproachthatgivesahigher
– characterisethe1billionquadrupleLinkedDatacor-
percentage of consolidated results but leads to a par-
pusthatwillbeusedforlaterevaluationofourmeth-
tiallygarbledcorpus;similarly,wepreferasystemthat
ods, particular focusing on the (re-)use of data-level
can handle more data (is more scalable), but may pos-
identifiersinthecorpus(Section3);
siblyhavealowerthroughput(islessefficient).4
– describe and evaluate our distributed base-line ap-
Thus, herein we revisit methods for scalable, pre-
proach for consolidation, which leverages explicit
cise, automatic and domain-agnostic entity consolida-
owl:sameAsrelations(Section4);
tion over large, static Linked Data corpora. In order
– describeandevaluateadistributedapproachthatex-
to make our methods scalable, we avoid dynamic on-
tendsconsolidationtoconsideraricherOWLseman-
disk index structures and instead opt for algorithms
ticsforconsolidating(Section5);
that rely on sequential on-disk reads/writes of com-
– present a distributed algorithm for determining
pressed flat files, using operations such as scans, ex-
weightedconcurrencebetweenentitiesusingstatisti-
ternalsorts,merge-joins,andonlylight-weightornon-
calanalysisofpredicatesinthecorpus(Section6);
criticalin-memoryindices.Inordertomakeourmeth-
– present a distributed approach to disambiguate
ods efficient, we demonstrate distributed implementa-
entities—i.e.,detectlikelyerroneousconsolidation—
tions of our methods over a cluster of shared-nothing
combiningthesemanticsandstatisticsderivablefrom
commodity hardware, where our algorithms attempt to
thecorpus(Section7);
maximise the portion of time spent in embarrassingly
– providecriticaldiscussion(Section8),renderrelated
parallel execution—i.e., parallel, independent compu-
work(Section9)andconcludewithdiscussion(Sec-
tation without need for inter-machine coordination. In
tion10).
order to make our methods domain-agnostic and fully-
automatic, we exploit the generic formal semantics of
2. Preliminaries
4 Of course, entity consolidation has many practical applications In this section, we provide some necessary prelimi-
outside of our referential use-case SWSE, and is useful in many
nariesrelatingto(i)RDF:thestructuredformatusedin
generic query-answering secnarios—we see these requirements as
our corpora (Section 2.1); (ii) RDFS/OWL, which pro-
beingsomewhatfundamentaltoaconsolidationcomponent,tovary-
ingdegrees. vides formal semantics to RDF data, including the se-
3
mantics of equality (Section 2.2); (iii) rules, which are classes, relationships (object properties), and attributes
used to interpret the semantics of RDF and apply rea- (datatype properties) and declaratively assigns them a
soning(Sections2.3,2.4);(iv)authoritativereasoning, semantics.5 With well-defined descriptions of classes
which critically examines the source of certain types and properties, reasoning can then allow for deriving
of Linked Data to help ensure robustness of reasoning newknowledge,includingoverassertionaldata.
(Section2.5);(v)andOWL2RL/RDFrules:astandard OntheSemanticWeb,theRDFS[11]andOWL[42]
rule-basedprofileforsupportingOWLsemantics,asub- standards are prominently used for making statements
set of which we use to deduce equality (Section 2.6). withwell-definedmeaningandenablingsuchreasoning.
Throughout,weattempttopreservenotationandtermi- Although primarily concerned with describing termi-
nologyasprevalentintheliterature. nological knowledge—such as subsumption or equiva-
lence between classes and properties, etc.—RDFS and
OWL also contain features that operate on an asser-
2.1. RDF
tional level. The most interesting such feature for our
purposes is owl:sameAs: a core OWL property that al-
We briefly give some necessary notation relating to
lows for defining equivalences between individuals (as
RDFconstantsandRDFtriples;see[28].
wellasclassesandproperties).Twoindividualsrelated
by means of owl:sameAs are interpreted as referring to
RDF Constant Given the set of URI references U, thesameentity;i.e.,theyarecoreferent.
the set of blank nodes B, and the set of literals L, the Interestingly, OWL also contains other features (on
set of RDF constants is denoted by C = U∪B∪L. a terminological level) that allow for inferring new
AsopposedtotheRDF-mandatedexistentialsemantics owl:sameAsrelations.Suchfeaturesinclude:
for blank-nodes, we interpret blank-nodes as ground – inverse-functionalproperties,whichactas“keyprop-
Skolemconstants;notealsothatwerewriteblank-node erties” for uniquely identifying subjects; e.g., an
labels to ensure uniqueness per document as per RDF :isbndatatype-propertywhosevaluesuniquelyiden-
merging[28]. tifies books and other documents, or the object-
property:biologicalMotherOf,whichuniquelyiden-
RDFTriple Atriplet:=(s,p,o)∈G:=(U∪B)× tifiesthebiologicalmotherofaparticularchild;
U×CiscalledanRDFtriple,wheresiscalledsubject, – functional properties, which work in the inverse di-
p predicate, and o object. We call a finite set of triples rection and uniquely identify objects; e.g., the prop-
G ⊂ G a graph. This notion of a triple restricts the erty:hasBiologicalMother;
positions in which certain terms may appear. We use – cardinality and max-cardinality restrictions, which,
thephrasegeneralisedtriplewheresuchrestrictionsare when given a value of 1, uniquely identify subjects
relaxed,andwhereliteralsandblank-nodesareallowed of a given class; e.g., the value for the property
toappearinthesubject/predicatepositions. :spouse uniquely identifies a member of the class
:Monogamist.
Thus, we see that RDFS and OWL contain various
RDF Triple in Context/Quadruple Given c ∈ U, let
features that allow for inferring and supporting coref-
http(c) denote the possibly empty graph G given by
c erence between individuals. In order to support the se-
retrieving the URI c through HTTP (directly returning
mantics of such features, we apply inference rules, in-
200 Okay). An ordered pair (t, c) with an RDF triple t
troducednext.
= (s, p, o), c ∈ U and t ∈ http(c) is called a triple in
context c. We may also refer to (s, p, o, c) as an RDF
quadrupleorquadqwithcontextc. 2.3. InferenceRules
Herein, we formalise the notion of an inference rule
2.2. RDFS,OWLandowl:sameAs
asapplicableoverRDFgraphs[33].Webeginbydefin-
ingthepreliminaryconceptofatriplepattern,ofwhich
Conceptually, RDF data is composed of assertional
rules are composed, and which may contain variables
data (i.e., instance data) and terminological data (i.e.,
inanyposition.
schemadata).Assertionaldatadefinerelationshipsbe-
tween individuals, provide literal-valued attributes of
5 Terminologicalandassertionaldataarenotbyanymeansneces-
thoseindividuals,anddeclareindividualsasmembersof
sarily disjoint. Furthermore, this distinction is not always required,
classes. Thereafter, terminological data describe those butisusefulforourpurposesherein.
4
Triple Pattern, Basic Graph Pattern A triple pattern G wrt. ruleset R, denoted as Cl (G) , or succinctly G
R
is a generalised triple where variables from the set V wheretherulesetisobvious.
are allowed; i.e.: tv := (sv, pv, ov) ∈ GV, GV :=
The above closure takes a graph and a ruleset and re-
(C∪V)×(C∪V)×(C∪V).Wecallaset(toberead
cursivelyappliestherulesovertheunionoftheoriginal
as conjunction) of triple patterns GV ⊂ GV a basic
graphandtheinferencesuntilafixpoint.
graph pattern. We denote the set of variables in graph
patternGV byV(GV).
2.4. T-splitRules
Intuitively,variablesappearingintriplepatternscanbe
bound by any RDF term in C. Such a mapping from
In [31,33], we formalised an optimisation based on
variablestoconstantsiscalledavariablebinding.
a separation of terminological knowledge from asser-
tional data when applying rules. Similar optimisations
VariableBindings LetΩbethesetofvariablebinding have also been used by other authors to enable large-
V∪C → V∪C that map every constant c ∈ C to scale rule-based inferencing [74,71,43]. This optimisa-
itselfandeveryvariablev ∈Vtoanelementoftheset tion is based on the premise that, in Linked Data (and
C∪V. A triple t is a binding of a triple pattern tv := various other scenarios), terminological data speaking
(sv,pv,ov)iffthereexistsµ∈Ω,suchthatt=µ(tv)= about classes and properties is much smaller than as-
(µ(sv),µ(pv),µ(ov)).AgraphGisabindingofagraph sertional data speaking about individuals. Previous ob-
pattern GV iff there exists a mapping µ ∈ Ω such that servationsindicatethatforaLinkedDatacorpusinthe
(cid:83) µ(tv)=G; we use the shorthand µ(GV)=G. orderofabilliontriples,suchterminologicaldatacom-
tv∈GV
WeuseΩ(GV,G):={µ|µ(GV)⊆G,µ(v)=v ifv ∈/ prises of about 0.1% of the total data volume. Addi-
V(GV)}todenotethesetofvariablebindingmappings tionally,terminologicaldataisveryfrequentlyaccessed
for graph pattern GV in graph G that map variables duringreasoning.Hence,seperatingandstoringtheter-
outsideGV tothemselves. minological data in memory allows for more efficient
executionofrules,albeitatthecostofpossibleincom-
Wecannowgivethenotionofaninferencerule,which
pleteness[74,33].
iscomprisedofapairofbasicgraphpatternsthatform
Wenowreintroducesomeoftheconceptsrelatingto
an“IF⇒THEN”logicalstructure.
separating terminological information [33], which we
will use later when applying rule-based reasoning to
support the semantics of OWL equality. We begin by
Inference Rule We define an inference rule (or often
formalising some related concepts that help define our
just rule) as a pair r := (Ante ,Con ), where the an-
r r
tecedent (or body) Ante ⊂ GV and the consequent notionofterminologicalinformation.
r
(or head) Con ⊂ GV are basic graph patterns such
r
that all variables in the consequent appear in the an-
Meta-class Weconsiderameta-classasaclasswhose
tecedent: V(Con ) ⊆ V(Ante ). We write inference
r r membersarethemselves(always)classesorproperties.
rulesasAnte ⇒Con .
r r Herein, we restrict our notion of meta-classes to the
Finally,rulesallowforapplyinginferencethroughrule set defined in RDF(S) and OWL specifications, where
applications, where the rule body is used to generate examples include rdf:Property, rdfs:Class, owl:-
variablebindingsagainstagivenRDFgraph,andwhere DatatypeProperty, owl:FunctionalProperty, etc. Note
thosebindingsareappliedontheheadtogeneratenew that, e.g., owl:Thing and rdfs:Resource are not meta-
triplesthatthatgraphentails. classes since not all of their members are classes or
properties.
Rule Application and Standard Closure A rule ap-
plication is the immediate consequences Tr(G) := Meta-property A meta-property is one that has a
(cid:83)
µ∈Ω(Anter,G)(µ(Conr) \ µ(Anter)) of a rule r on meta-class as its domain; again, we restrict our no-
a graph G; accordingly, for a ruleset R, T (G) := tion of meta-properties to the set defined in RDF(S)
R
(cid:83)
T (G).Now,letG :=G ∪T (G )andG := and OWL specifications, where examples include
r∈R r i+1 i R i 0
G; the exhaustive application of the TR operator on a rdfs:domain,rdfs:range,rdfs:subClassOf,owl:hasKey,
graphG isthentheleastfixpoint(thesmallestvaluefor owl:inverseOf, owl:oneOf, owl:onProperty, owl:union-
n) such that Gn = TR(Gn). We call Gn the closure of Of, etc. Note that rdf:type, owl:sameAs, rdfs:label,
5
e.g.,donothaveameta-classasdomainandsoarenot (:isbn,a,owl:InverseFunctionalProperty)
consideredasmeta-properties. (:mbox,a,owl:InverseFunctionalProperty)
Terminological Triple We define the set of termino- From these two triples, we can generate two T-ground
logicaltriplesT⊂Gastheunionof: rulesasfollows:
(i) tripleswithrdf:typeaspredicateandameta-class
asobject; (?x1,:isbn,?y),
(ii) tripleswithameta-propertyaspredicate; (?x2,:isbn,?y)
(iii) triples forming a valid RDF list whose head is ⇒(?x1,owl:sameAs,?x2)
theobjectofameta-property(e.g.,alistusedfor
owl:unionOf,etc.). (?x1,:mbox,?y),
(?x2,:mbox,?y)
The above concepts cover what we mean by termino-
⇒(?x1,owl:sameAs,?x2)
logical data in the context of RDFS and OWL. Next,
we define the notion of triple patterns that distinguish These T-ground rules encode the terminological OWL
betweenthesedifferentcategoriesofdata. knowledge given by the previous two triples, and can
be applied directly over the assertional data, e.g., de-
Terminological/AssertionalPattern Werefertoater- scribingspecificbookswithISBNnumbers. (cid:51)
minological -triple/-graph pattern as one whose in-
stancecanonlybeaterminologicaltripleor,resp.,aset
thereof.Anassertionalpatternisanypatternthatisnot 2.5. AuthoritativeT-splitRules
terminological.
Caution must be exercised when applying reasoning
Giventheabovenotionsofterminologicaldata/patterns,
overarbitrarydatacollectedfromtheWeb;e.g.,Linked
we can now define the notion of a T-split inference
Data. In previous works, we have encountered various
rule that distinguishes terminological from assertional
problems when naïvely performing rule-based reason-
information.
ing over arbitrary Linked Data [31,33]. In particular,
third-partyclaimsmadeaboutpopularclassesandprop-
T-split inference rule Given a rule r := erties must be critically analysed to ensure their trust-
(Anter,Conr),wedefineaT-splitrulerτ asthetriple worthiness. As a single example of the type of claims
(AnteTrτ,AnteGrτ,Con)whereAnteTrτ isthesetofter- that can cause issues with reasoning, we found one
minologicalpatternsinAnte ,andAnteG :=Ante \ document that defines nine properties as the domain
r rτ r
AnteT . ofrdf:type;6 thereafter,thesemanticsofrdfs:domain
rτ
mandate that everything which is a member of any
The body of T-split rules are divided into a set of pat-
class (i.e., almost every known resource) can be in-
terns that apply over terminological knowledge, and a
ferredtobea“member”ofeachofthenineproperties.
set of patterns that apply over assertional (i.e., any)
Wefoundthatnaïvereasoningleadto∼200×morein-
knowledge.Suchrulesenableanoptimisationwhereby
ferences than would be expected when considering the
terminologicalpatternsarepre-boundtogenerateanew,
definitions of classes and properties as defined in their
larger set of purely assertional rules called T-ground
“namespacedocuments”.Thus,inthegeneralcase,per-
rules.Weillustratethiswithanexample.
formingrule-basedmaterialisationwithrespecttoOWL
semanticsoverarbitraryLinkedDatarequiressomecrit-
Example: LetR denotethefollowingrule
prp-ifp
ical analysis of the source of data, as per our authori-
(?p,a,owl:InverseFunctionalProperty), tativereasoningalgorithm.Pleasesee[9]foradetailed
(?x1,?p,?y), analysisoftheexplosionininferencesthatoccurswhen
(?x2,?p,?y) authoritativeanalysisisnotapplied.
⇒(?x1,owl:sameAs,?x2) Such observations prompted us to investigate more
robustformsofreasoning.Alongtheselines,whenrea-
When writing T-split rules, we denote terminological soning over Linked Data we apply an algorithm called
patterns in the body by underlining. Also, we use ‘a’ authoritativereasoning[14,31,33],whichcriticallyex-
as a convenient shortcut for rdf:type, which indicates
class-membership.Nowtaketheterminologicaltriples: 6 viz.http://www.eiao.net/rdf/1.0
6
aminesthesourceofterminologicaltriplesandconser- Example: LetR denotethesameruleasusedin
prp-ifp
vativelyrejectsthosethatcannotbedefinitivelytrusted: the previous example, and let the following two triples
i.e.,thosethatredefineclassesand/orpropertiesoutside begivenbythedereferenceddocumentofv1:isbn,but
ofthenamespacedocument. notv2:mbox—i.e.,adocumentauthoritativeforthefor-
mertermbutnotthelatter.
Dereferencing and Authority We first give the func- (v1:isbn,a,owl:InverseFunctionalProperty)
tionhttp:U→2G thatmapsaURItoanRDFgraph (v2:mbox,a,owl:InverseFunctionalProperty)
itreturnsuponaHTTPlookupthatreturnstheresponse
The only authoritative variable appearing in both the
code200Okay;inthecaseoffailure,thefunctionmaps
terminological and assertional patterns of the body of
to an empty RDF graph. Next, we define the function
redir:U→Uthatfollowsa(finite)sequenceofHTTP Rprp-ifp is ?p, bound by v1:isbn and v2:mbox above.
Sincethedocumentservingthetwotriplesisauthorita-
redirects given upon lookup of a URI, or that returns
theURIitselfintheabsenceofaredirectorinthecase tive for v1:isbn, the first triple is considered authorita-
tive,andthefollowingT-groundrulewillbegenerated:
of failure; this function first strips the fragment iden-
tifier (given after the ‘#’ character) from the original
URIifpresent.Thenwegivethedereferencingfunction (?x1,v1:isbn,?y),
deref :http◦redir asthemappingfromaURI(aWeb (?x2,v1:isbn,?y)
location)toanRDFgraphitmayprovidebymeansof ⇒(?x1,owl:sameAs,?x2)
agivenHTTPlookupthatfollowsredirects.Finally,we
can define the authority function—that maps the URI However, since the document is not authoritative for
ofaWebdocumenttothesetofRDFtermsitisauthor- v2:mbox, the second T-ground rule will be considered
non-authoritativeanddiscarded:
itativefor—asfollows:
(?x1,v2:mbox,?y),
auth: U→2C
(?x2,v2:mbox,?y)
u(cid:55)→{c∈U|redir(c)=u}∪B(http(s)) ⇒(?x1,owl:sameAs,?x2)
where B(G) denotes the set of blank-nodes appearing
Wherethird-partydocumentsre-defineremotetermsin
in a graph G. In other words, a Web document is au-
a way that affects inferencing over instance data, we
thoritativeforURIsthatdereferencetoitandtheblank
filtersuchnon-authoritativedefinitions. (cid:51)
nodesitcontains.
We note that making RDF vocabulary URIs derefer-
enceableisencouragedinvariousbest-practices[45,4].
2.6. OWL2RL/RDFRules
Toenforceauthoritativereasoning,whenapplyingrules
withterminologicalandassertionalpatternsinthebody
Inferencerulescanbeusedto(partially)supportthe
we require that terminological triples are served by a
semantics of RDFS and OWL. Along these lines, the
documentauthoritativefortermsthatareboundbyspe-
OWL2RL/RDF[24]rulesetisapartial-axiomatisation
cificvariablepositionsintherule.
of the OWL 2 RDF-Based Semantics, which is appli-
cable for arbitrary RDF graphs and constitutes an ex-
Authoritative T-split Rule Application Given a T- tension of the RDF Semantics [28]; in other words,
split rule and a rule application involving a map- the ruleset supports a standardised profile of reasoning
ping µ as before, for the rule application to be au- that partially covers the complete OWL semantics. In-
thoritative there must additionally exist a µ(v) such terestingly for our scenario, this profile includes infer-
that v ∈ V(AnteT) ∩ V(AnteG), µ(v) ∈ auth(u), ence rules that support the semantics and inference of
µ(AnteT)⊆http(u). We call the set of variables that owl:sameAsasdiscussed.
appear in both the terminological and assertional seg- First, in Table 1, we provide the set of OWL 2
mentoftherulebody(i.e.,V(AnteT)∩V(AnteG))the RL/RDF rules that support the (positive) semantics of
set of authoritative variables for the rule. Where a rule owl:sameAs, axiomatising the symmetry (rule eq-sym)
doesnothaveterminologicalorassertionalpatterns,we and transitivity (rule eq-trans) of the relation. To take
consideranyrule-applicationasauthoritative.Thenota- anexample,ruleeq-symisasfollows:
tionofanauthoritativeT-groundrulefollowslikewise.
7
(?x,owl:sameAs,?y) Notethatalthoughweonlyperformpartialmaterial-
⇒(?y,owl:sameAs,?x) isation,wedonotchangethesemanticsofequality:our
methods are sound with respect to OWL semantics. In
where if we know that (:a, owl:sameAs, :b), the rule addition, alongside the partially materialised data, we
will infer the symmetric relation (:b, owl:sameAs, :a). provideasetofconsolidatedowl:sameAsrelations(con-
Rule eq-trans operates analogously; both together al- taining all of the identifiers in each equivalence class)
low for computing the transitive, symmetric closure of thatcanbeusedto“backward-chain”thefullinferences
theequivalencerelation.Furthermore,Table1alsocon- possiblethroughreplacement(asrequired).Thus,wedo
tainstheOWL2RL/RDFrulesthatsupporttheseman- notconsiderthecanonicalidentifierassomehow‘defini-
ticsofreplacement(ruleseq-rep-*),wherebydatathat tive’ or superceding the other identifiers, but merely
holdsforoneentitymustalsoholdforequivalententi- consideritasrepresentingtheequivalenceclass.7
ties. Finally, herein we do not consider consolidation of
Notethatwe(optionally,andinthecaseoflatereval- literals; one may consider useful applications, e.g., for
uation)choosenottosupport: canonicalising datatype literals, but such discussion is
(i) the reflexive semantics of owl:sameAs, since re- outofthecurrentscope.
flexiveowl:sameAsstatementswillnotleadtoany Aspreviouslymentioned,OWL2RL/RDFalsocon-
consolidation or other non-reflexive equality re- tainsrulesthatuseterminologicalknowledge(alongside
lationsandwillproducealargebulkofmaterial- assertional knowledge) to directly infer owl:sameAs re-
isations; lations. We enumerate these rules in Table 2; note that
(ii) equality on predicates or values for rdf:type, we italicise the labels of rules supporting features new
where we do not want possibly imprecise to OWL 2, and that we list authoritative variables in
owl:sameAsdatatoaffecttermsinthesepositions. bold.8
ApplyingonlytheseOWL2RL/RDFrulesmaymiss
Antecedent inferenceofsomeowl:sameAsstatements.Forexample,
OWL2RL Consequent
assertional consider the example RDF data given in Turtle syn-
eq-sym ?x owl:sameAs?y. ?yowl:sameAs?x . tax[3]asfollows:
eq-trans ?x owl:sameAs?y.?yowl:sameAs?z.?x owl:sameAs?z.
eq-rep-s ?sowl:sameAs?s(cid:48) .?s?p?o. ?s(cid:48) ?p?o.
eq-rep-o ?oowl:sameAs?o(cid:48) .?s?p?o. ?s?p?o(cid:48) . #FromtheFOAFVocabulary:
foaf:homepagerdfs:subPropertyOffoaf:isPrimaryTopicOf.
Table1 foaf:isPrimaryTopicOfowl:inverseOffoaf:primaryTopic.
Rulesthatsupportthepositivesemanticsofowl:sameAs foaf:isPrimaryTopicOfaowl:InverseFunctionalProperty.
#FromExampleDocumentA:
exA:axelfoaf:homepage<http://polleres.net/>.
Giventhatthesemanticsofequalityisquadraticwith #FromExampleDocumentB:
<http://polleres.net/>foaf:primaryTopicexB:apolleres.
respecttotheA-Box,weapplyapartial-materialisation
#Inferredthroughprp-spo1:
approachthatgivesournotionofconsolidation:instead exA:axelfoaf:isPrimaryTopicOf<http://polleres.net/>.
of materialising (i.e., explicitly writing down) all pos- #Inferredthroughprp-inv:
exB:apolleresfoaf:isPrimaryTopicOf<http://polleres.net/>.
sibleinferencesgivenbythesemanticsofreplacement,
#Subsequently,inferredthroughprp-ifp:
weinsteadchooseonecanonicalidentifiertorepresent exA:axelowl:sameAsexB:apolleres.
exB:apolleresowl:sameAsexA:axel.
the set of equivalent terms, thus effectively compress-
ing the data. We have used this approach in previous
The example uses properties from the prominent
works[30–32],andithasalsoappearedinrelatedworks
FOAF vocabulary, which is used for publishing per-
intheliterature[70,6],asacommon-senseoptimisation
sonalprofilesasRDF.Here,weadditionallyneedOWL
forhandlingdata-levelequality.Totakeanexample,in
2RL/RDFrulesprp-invandprp-spo1—handlingstan-
later evaluation (cf. Table 10) we find a valid equiva-
dard owl:inverseOf and rdfs:subPropertyOf inferenc-
lenceclass(setofequivalententities)with33,052mem-
ing respectively—to infer the owl:sameAs relation en-
bers;materialisingallpairwisenon-reflexiveowl:sameAs tailedbythedata.
statements would infer more than 1 billion owl:sameAs
relations (33,0522 - 33,052) = 1,092,434,704; further
assumingthateachentityappearedinonaverage,e.g., 7 Wemayoptionallyconsidernon-canonicalblank-nodeidentifiers
two quadruples, we would infer an additional ∼2 bil- asredundantanddiscardthem.
8 As discussed later, our current implementation requires at least
lionofmassivelyduplicateddata.Bychoosingasingle
one variable to appear in all assertional patterns in the body, and
canonical identifier, we would instead only materialise
sowedonotsupportprp-keyandcls-maxqc3;however,thesetwo
∼100thousandstatements. featuresarenotcommonlyusedinLinkedDatavocabularies[29].
8
Antecedent
OWL2RL Consequent
terminological assertional
prp-fp ?paowl:FunctionalProperty. ?x ?p?y1 ,?y2 . ?y1 owl:sameAs?y2 .
prp-ifp ?paowl:InverseFunctionalProperty.?x1 ?p?y.?x2 ?p?y.?x1 owl:sameAs?x2 .
?xowl:maxCardinality1. ?ua?x .
cls-maxc2 ?xowl:onProperty?p. ?u?p?y1 ,?y2 . ?y1 owl:sameAs?y2 .
?xowl:maxQualifiedCardinality1.
?ua?x .
cls-maxqc4??xxoowwll::oonnPCrloapsserotywl?:Tph.ing. ?u?p?y1 ,?y2 . ?y1 owl:sameAs?y2
Table2
OWL 2 RL/RDF rules that directly produce owl:sameAs relations; we denote authoritative variables with bold and italicise
thelabelsofrulesrequiringnewOWL2constructs
Along these lines, we support an extended set of andtheyshouldnotsharemembers.Alongtheselines,
OWL 2 RL/RDF rules that contain precisely one as- OWL 2 RL/RDF contains rules (with the special sym-
sertional pattern and for which we have demonstrated bol false in the head) that allow for detecting incon-
a scalable implementation called SAOR, designed for sistency in an RDF graph. We use a subset of these
Linked Data reasoning [33]. These rules are listed in consistency-checking OWL 2 RL/RDF rules later in
Table3,andasperthepreviousexample,cangenerate Section 7 to try to automatically detect and repair un-
additional inferences that indirectly lead to the deriva- intendedowl:sameAsinferences;thesupportedsubsetis
tionofnewowl:sameAsdata.Wewillusetheserulesfor listedinTable4.
entityconsolidationinSection5.
Antecedent
OWL2RL
Antecedent terminological assertional
OWL2RL Consequent
terminological assertional ?x owl:sameAs?y.
eq-diff1 -
prp-dom ?prdfs:domain?c. ?x ?p?y. ?x a?c. ?x owl:differentFrom?y.
prp-rng ?prdfs:range?c. ?x ?p?y. ?ya?c. prp-irp ?paowl:IrreflexiveProperty. ?x ?p?x .
prp-symp?paowl:SymmetricProperty. ?x ?p?y. ?y?p?x . prp-asyp ?paowl:AsymmetricProperty ?x ?p?y.?y?p?x .
prp-spo1 ?p1 rdfs:subPropertyOf?p2 . ?x ?p1 ?y.?x ?p2 ?y. prp-pdw ?p1 owl:propertyDisjointWith?p2 .?x ?p1 ?y;?p2 ?y.
pprrpp--eeqqpp12 ??pp11 oowwll::eeqquuiivvaalleennttPPrrooppeerrttyy??pp22 .. ??xx ??pp12 ??yy..??xx ??pp21 ??yy.. prp-adp ??xx aowolw:ml:eAmllbDeirssjo(i?nptP1r,o.p..e,rt?iepsn.). ?u?pi ?y;?pj ?y.(i(cid:54)=j)
prp-inv1 ?p1 owl:inverseOf?p2 . ?x ?p1 ?y.?y?p2 ?x . cls-com ?c1 owl:complementOf?c2 . ?x a?c1 ,?c2 .
prp-inv2 ?p1 owl:inverseOf?p2 . ?x ?p2 ?y.?y?p1 ?x . cls-maxc1 ?xowl:maxCardinality0. ?ua?x ;?p?y.
cls-int2 ?cowl:intersectionOf(?c1 ...?cn).?x a?c. ?x a?c1...?cn . ?xowl:onProperty?p.
cls-uni ?cowl:unionOf(?c1...?ci...?cn). ?x a?ci ?x a?c. ?xowl:maxQualifiedCardinality0.
?x owl:someValuesFromowl:Thing; cls-maxqc2?xowl:onProperty?p. ?ua?x ;?p?y.
cls-svf2 owl:onProperty?p. ?u?p?v. ?ua?x . ?xowl:onClassowl:Thing.
?xowl:hasValue?y; cax-dw ?c1 owl:disjointWith?c2 . ?x a?c1 ,?c2 .
cls-hv1 ?ua?x . ?u?p?y.
?xoowwll::ohnasPVroaplueerty?y?p; . cax-adc ??xx aowolw:ml:eAmllbDeirssjo(i?nct1C,la.s.s.,es?c.n). ?za?ci ,?cj .(i(cid:54)=j)
cls-hv2 owl:onProperty?p. ?u?p?y. ?ua?x . Table4
cax-sco ?c1 rdfs:subClassOf?c2 . ?x a?c1 . ?x a?c2 . OWL 2 RL/RDF rules used to detect inconsistency that we
cax-eqc1 ?c1 owl:equivalentClass?c2 . ?x a?c1 . ?x a?c2 . currentlysupport;wedenoteauthoritativevariableswithbold
cax-eqc2 ?c1 owl:equivalentClass?c2 . ?x a?c2 . ?x a?c1 .
Table3
OWL2RL/RDFrulescontainingpreciselyoneassertionalpattern
inthebody,withauthoritativevariablesinbold(cf.[33])
2.7. Distributionarchitecture
Lastly,aswediscusslater(particularlyinSection7)
LinkedDataisinherentlynoisyandpronetomistakes. Tohelpmeetourscalabilityrequirements,ourmeth-
Althoughourmethodsaresound(i.e.,correct)withre- odsareimplementedonashared-nothingdistributedar-
specttoformalOWLsemantics,suchnoiseinourinput chitecture [64] over a cluster of commodity hardware.
datamayleadtounintendedconsequenceswhenapply- Thedistributedframeworkconsistsofamastermachine
ingreasoningwhichwewishtominimiseand/orrepair. that orchestrates the given tasks, and several slave ma-
Relatedly,OWLcontainsfeaturesthatallowfordetect- chinesthatperformpartsofthetaskinparallel.
ing formal contradictions—called inconsistencies—in The master machine can instigate the following dis-
RDF data. An example of an inconsistency would be tributedoperations:
where something is a member of two disjoint classes, – scatter:partitionon-diskdatausingsomelocalsplit
such as foaf:Person and foaf:Organization; formally, function, and send each chunk to individual slave
theintersectionofsuchdisjointclassesshouldbeempty machinesforsubsequentprocessing;
9
– run: request the parallel execution of a task by the edge and to coordinate upon the larger corpus (e.g., a
slavemachines—suchataskeitherinvolvesprocess- coordinate function hashing on the subject and object
ing of some data local to the slave machine, or the ofthedata,orperhapsanadaptationoftheSPEEDDATE
coordinatemethod(describedlater)forreorganising routing strategy [51]). Such decisions are heavily in-
thedataunderanalysis; fluenced by the scale of the task to perform, the per-
– gather:gatherschunksofoutputdatafromtheslave centageofknowledgethatisgloballyrequired,howthe
swarmandperformssomelocalmergefunctionover input data are distributed, how the output data should
thedata; bedistributed,andthenatureoftheclusteroverwhich
– flood: broadcast global knowledge required by all it should be performed and the task-scheduling possi-
slavemachinesforafuturetask. ble.Thedistributedimplementationofourtasksarede-
The master machine provides input data to the slave signedtoexploitarelativelysmallpercentageofglobal
swarm, provides the control logic required by the dis- knowledgewhichischeaptocoordinate,andwechoose
tributed task (commencing tasks, coordinating timing, toavoid—insofarasreasonable—duplicatingcomputa-
ending tasks), gathers and locally perform tasks on tion.
globalknowledgethattheslavemachineswouldother-
wisehavetoreplicateinparallel,andtransmitsglobally
2.8. Experimentalsetup
requiredknowledge.
The slave machines, as well as performing tasks in
Our entire code-base is implemented on top of stan-
parallel,canperformthefollowingdistributedoperation
dard Java libraries. We instantiate the distributed ar-
(atthebehestofthemastermachine):
chitecture using Java RMI libraries, and using the
– coordinate:localdataoneachslavemachineispar-
lightweight open-source Java RMIIO package9 for
titioned according to some split function, with the
streamingdataforthenetwork.
chunks sent to individual machines in parallel; each
All of our evaluation is based on nine machines
slave machine also gathers the incoming chunks in
connected by Gigabit ethernet,10 each with uniform
parallelusingsomemergefunction.
specifications, viz., 2.2GHz Opteron x86-64, 4GB
The above operation allows slave machines to reor-
main memory, 160GB SATA hard-disks, running Java
ganise (split/send/gather) intermediary amongst them-
1.6.0_12onDebian5.0.4.
selves; the coordinate operation could be replaced by
a pair of gather/scatter operations performed by the
master machine, but we wish to avoid the channelling 3. Experimentalcorpus
ofallintermediarydatathroughonemachine.
Note that herein, we assume that the input corpus is Later in this paper, we discuss the performance
evenly distributed and split across the slave machines, and results of applying our methods over a corpus
and that the slave machines have roughly even speci- of 1.118 billion quadruples derived from an open-
fications: that is, we do not consider any special form domain RDF/XML crawl of 3.985 million web doc-
ofloadbalancing,butinsteadaimtohaveuniformma- uments in mid-May 2010 (detailed in [32,29]). The
chinesprocessingcomparabledata-chunks. crawlwasconductedinabreadth-firstmanner,extract-
We note that there is the practical issue of the mas- ing URIs from all positions of the RDF data. Individ-
termachinebeingidlewaitingfortheslaves,and,more ual URI queues were assigned to different pay-level-
critically,thepotentiallylargeclusterofslavemachines domains (aka. PLDs: domains that require payment,
waiting idle for the master machine. One could over- e.g., deri.ie, data.gov.uk), where we enforced a po-
come idle times with mature task-scheduling (e.g., in- liteness policy of accessing a maximum of two URIs
terleaving jobs) and load-balancing. From an algorith- perPLDpersecond.URIswiththehighestinlinkcount
micpointofview,removingthecentralcoordinationon pereachPLDqueuewerepolledfirst.
the master machine may enable better distributability. Withregardstheresultingcorpus,ofthe1.118billion
Onepossibilitywouldbetoallowtheslavemachinesto quads, 1.106 billion are unique, and 947 million are
duplicatetheaggregationofglobalknowledgeinparal- unique triples. The data contain 23 thousand unique
lel:althoughthiswouldfreeupthemastermachineand predicates and 105 thousand unique class terms (terms
wouldprobablytakeroughlythesametime,duplicating
computation wastes resources that could otherwise be
9
http://openhms.sourceforge.net/rmiio/
exploitedby,e.g.,interleavingjobs.Asecondpossibil- 10Weobserve,e.g.,amaxFTPtransferrateof38MB/secbetween
itywouldbetoavoidtherequirementforglobalknowl- machines.
10