DATA FUSION – RESOLVING DATA CONFLICTS IN INTEGRATION

107 Pages·2009·2.82 MB·English

DDAATTAA FFUUSSIIOONN –
RESOLVING DATA
CONFLICTS IN INTEGRATION
Tutorial at Xin Luna Dong – AT&T Labs-Research
VLDB 2009 Felix Naumann – Hasso Plattner Institute (HPI)
Origins of Data Conflicts
2
OOrriiggiinnaall
ACM Computing
Survey [BN08]
Scanned
Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann
Origins of Data Conflicts
3
Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann
Origins of Data Conflicts
4
IInntteeggrraatteedd ddaattaa
Schering CRM Bayer CRM
Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann
Origins of Data Conflicts: German Names
5
Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann
Origins of Data Conflicts: Difficult Names
6
Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann
Origins of Intra-Source Conflicts
7
NNo integrity or consistency chheckks

Redundant schemata

Typos, transmission errors, incorrect calculations

Variants

Kantstr. / Kantstrasse / Kant Str. / Kant Strasse

Kolmogorov / Kolmogoroff / Kolmogorow

Typical confusion (OCR)

U<->V, 0<->o, 1<->l, etc.

Obsolete values

Different update frequencies, forgotten update

Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann
Origins of Inter-Source Conflicts
8
LLocalllly consiisttentt bbutt gllobballlly iinconsiisttentt

Different data types

Local sppellingg variations and conventions

Addresses

St → Street, Ave → Avenue, etc.

RR.-BBrreeiittsscchheeiidd-SSttrr. 7722 aa →→ RRuuddoollff-BBrreeiittsscchheeiidd.-SSttrr. 7722AA

128 spellings for Frankfurt am Main

 Frankfurt a.M., Frankfurt/M, Frankfurt, Frankfurt a. Main, …
NNaammeess

Dr. Ing. h.c. F. Porsche AG

Hewlett-Packard Development Company, L.P.

NNuummeerriiccaall ddaattaa

10.000 € = 10T EURO = 10k EUR = 10.000,00€ = 10,000.- €

Phone numbers, birth dates, etc.

Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann
Resolution of Data Conflicts?
ttaabblleess.ggoooogglleellaabbss.ccoomm
9
““… ffocus is on ffusing data management and

collaboration: merging multiple data sources,
ddiiscussiion off tthhe ddatta, queryiing, viisualliizattiion, andd
Web publishing.”
““TThhe power off ddata iis trully hharnessedd whhen you

combine data from multiple sources. Fusion Tables
eennaabblleess yyoouu ttoo ffuussee mmuullttiippllee sseettss ooff ddaattaa wwhheenn tthheeyy
are about the same entities. In database speak, we
ccaallll tthhiiss aa jjooiinn oonn aa pprriimmaarryy kkeeyy bbuutt tthhee ddaattaa
originates from multiple independent sources.”
Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann
Web Integration—Google Fusion Tables
10
AAlllloowwss ddiissccuussssiioonn ooff vvaalluueess bbeettwweeeenn uusseerrss

Data Fusion | VLDB 2009 Tutorial | Luna Dong & Felix Naumann