Table Of ContentE
n River Publishers Series in Software Engineering
g
Engineering Agile i
n
e
e
r
in Engineering Agile
Big-Data Systems g
A
g
i
le Big-Data Systems
B
i
g
-
Kevin Feeney, Jim Davies, James Welch, Sebastian Hellmann, D
a
Christian Dirschl, Andreas Koller, Pieter Francois and ta
S Kevin Feeney, Jim Davies, James Welch, Sebastian Hellmann,
Arkadiusz Marciniak (Editors) y
s
t Christian Dirschl, Andreas Koller, Pieter Francois and
e
m
Arkadiusz Marciniak (Editors)
To be effective, data-intensive systems require extensive s
ongoing customisation to reflect changing user requirements,
organisational policies, and the structure and interpretation
of the data they hold. Manual customisation is expensive,
time-consuming, and error-prone. In large complex systems,
K
the value of the data can be such that exhaustive testing is ev
Cin
necessary before any new feature can be added to the existing h F
design. In most cases, the precise details of requirements, risee
tiane
policies and data will change during the lifetime of the ny
system, forcing a choice between expensive modification and A Dir, Jim
conEtinnguiende eorpinegra Atigoilne wBiigth-D aant ain Seyfsfitceimens to duetlsiingens. an approach to rkadiuschl, A Davie
dealing with these problems in software and data engineering, sz Mndres, Ja
dthersocurigbhionug t par odmuectth loifdeoclyocgleys . fIot rd isacluigsnseins gt oothlse wseh icphr occaens sbees arcinas Komes W
used to achieve these goals, and, in a number of case studies, iak (Eller, Pelch
simhopwrosv eh oaw v atrhieet yt ooof lasc aadnedm mice athnodd boulosginye shsa svyes tebmeesn. used to ditorieter, Seb
s) Fras
antia
cn
ois an Hellm
d
a
n
n
,
River Publishers River River Publishers
Engineering Agile Big-Data Systems
RIVER PUBLISHERS SERIES IN SOFTWARE
ENGINEERING
Indexing: All books published in this series are submitted to the Web of
ScienceBookCitationIndex(BkCI),toCrossRefandtoGoogleScholar.
The “River Publishers Series in Software Engineering” is a series of
comprehensive academic and professional books which focus on the theory
and applications of Computer Science in general, and more specifically
ProgrammingLanguages,SoftwareDevelopmentandSoftwareEngineering.
Books published in the series include research monographs, edited
volumes, handbooks and textbooks. The books provide professionals,
researchers, educators, and advanced students in the field with an invaluable
insightintothelatestresearchanddevelopments.
Topicscoveredintheseriesinclude,butarebynomeansrestrictedtothe
following:
• SoftwareEngineering
• SoftwareDevelopment
• ProgrammingLanguages
• ComputerScience
• AutomationEngineering
• ResearchInformatics
• InformationModelling
• SoftwareMaintenance
Foralistofotherbooksinthisseries,visitwww.riverpublishers.com
Engineering Agile Big-Data Systems
Editors
Kevin Feeney
TrinityCollegeDublin,Ireland
Jim Davies
OxfordUniversity,UnitedKingdom
James Welch
OxfordUniversity,UnitedKingdom
Sebastian Hellmann
UniversityofLeipzig,Germany
Christian Dirschl
WoltersKluwer,Germany
Andreas Koller
SemanticWebCompany,Austria
Pieter Francois
OxfordUniversity,UnitedKingdom
Arkadiusz Marciniak
AdamMickiewiczUniversity,Poland
River Publishers
Published,soldanddistributedby:
RiverPublishers
Alsbjergvej10
9260Gistrup
Denmark
RiverPublishers
LangeGeer44
2611PWDelft
TheNetherlands
Tel.:+45369953197
www.riverpublishers.com
ISBN:978-87-7022-016-3(Hardback)
978-87-7022-015-6(Ebook)
(cid:13)cThe Editor(s) (if applicable) and The Author(s) 2018. This book is
publishedopenaccess.
OpenAccess
This book is distributed under the terms of the Creative Commons Attribution-Non-
Commercial 4.0 International License, CC-BY-NC 4.0) (http://creativecommons.org/
licenses/by/4.0/), which permits use, duplication, adaptation, distribution and
reproduction in any medium or format, as long as you give appropriate credit to the
originalauthor(s)andthesource,alinkisprovidedtotheCreativeCommonslicenseand
any changes made are indicated. The images or other third party material in this book
areincludedinthework’sCreativeCommonslicense,unlessindicatedotherwiseinthe
creditline;ifsuchmaterialisnotincludedinthework’sCreativeCommonslicenseand
the respective action is not permitted by statutory regulation, users will need to obtain
permissionfromthelicenseholdertoduplicate,adapt,orreproducethematerial.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,
etc.inthispublicationdoesnotimply,evenintheabsenceofaspecificstatement,that
such names are exempt from the relevant protective laws and regulations and therefore
freeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and
informationinthisbookarebelievedtobetrueandaccurateatthedateofpublication.
Neitherthepublishernortheauthorsortheeditorsgiveawarranty,expressorimplied,
withrespecttothematerialcontainedhereinorforanyerrorsoromissionsthatmayhave
beenmade.
Printedonacid-freepaper.
Contents
Preface xv
Acknowledgements xvii
ListofContributors xix
ListofFigures xxi
ListofTables xxix
ListofAbbreviations xxxi
1 Introduction 1
1.1 StateoftheArtinEngineeringData-IntensiveSystems . . . 2
1.1.1 TheChallenge . . . . . . . . . . . . . . . . . . . . 4
1.2 StateoftheArtinSemantics-DrivenSoftwareEngineering . 5
1.2.1 TheChallenge . . . . . . . . . . . . . . . . . . . . 8
1.3 StateoftheArtinDataQualityEngineering . . . . . . . . . 8
1.3.1 TheChallenge . . . . . . . . . . . . . . . . . . . . 11
1.4 AboutALIGNED . . . . . . . . . . . . . . . . . . . . . . . 12
1.5 ALIGNEDPartners . . . . . . . . . . . . . . . . . . . . . . 15
1.5.1 TrinityCollegeDublin . . . . . . . . . . . . . . . . 15
1.5.2 OxfordUniversity–DepartmentofComputer
Science . . . . . . . . . . . . . . . . . . . . . . . . 15
1.5.3 Oxford University – School of Anthropology and
MuseumEthnography . . . . . . . . . . . . . . . . 15
1.5.4 UniversityofLeipzig–AgileKnowledgeEngineering
andSemanticWeb(AKSW) . . . . . . . . . . . . . 15
1.5.5 SemanticWebCompany . . . . . . . . . . . . . . . 16
1.5.6 WoltersKluwerGermany . . . . . . . . . . . . . . 16
v
vi Contents
1.5.7 AdamMickiewiczUniversityinPoznan´ . . . . . . . 16
1.5.8 WoltersKluwerPoland . . . . . . . . . . . . . . . . 17
1.6 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2 ALIGNEDUseCases–DataandSoftwareEngineering
Challenges 21
ArkadiuszMarciniakandPatrycjaFilipowicz
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2 TheALIGNEDUseCases . . . . . . . . . . . . . . . . . . 24
2.2.1 Seshat:GlobalHistoryDatabank . . . . . . . . . . . 24
2.2.2 PoolPartyEnterpriseApplicationDemonstrator
System . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.3 DBpedia . . . . . . . . . . . . . . . . . . . . . . . 27
2.2.4 JurionandJurionIPG . . . . . . . . . . . . . . . . 29
2.2.5 HealthDataManagement . . . . . . . . . . . . . . 31
2.3 TheALIGNEDUseCasesandDataLifeCycle.
MajorChallengesandOfferedSolutions . . . . . . . . . . . 33
2.4 TheALIGNEDUseCasesandSoftwareLifeCycle.
MajorChallengesandOfferedSolutions . . . . . . . . . . . 36
2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3 Methodology 41
JamesWelch,JimDavies,KevinFeeney,PieterFrancois,
JeremyGibbonsandSeyyedShah
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 SoftwareandDataEngineeringLifeCycles . . . . . . . . . 43
3.2.1 SoftwareEngineeringLifeCycle. . . . . . . . . . . 43
3.2.2 DataEngineeringLifeCycle . . . . . . . . . . . . . 47
3.3 SoftwareDevelopmentProcesses . . . . . . . . . . . . . . . 49
3.3.1 Model-DrivenApproaches . . . . . . . . . . . . . . 49
3.3.2 FormalTechniques . . . . . . . . . . . . . . . . . . 51
3.3.3 Test-DrivenDevelopment . . . . . . . . . . . . . . 52
3.4 IntegrationPointsandHarmonisation . . . . . . . . . . . . 53
3.4.1 IntegrationPoints . . . . . . . . . . . . . . . . . . . 54
3.4.2 BarrierstoHarmonisation . . . . . . . . . . . . . . 55
3.4.3 MethodologyRequirements . . . . . . . . . . . . . 58
3.5 AnALIGNEDMethodology . . . . . . . . . . . . . . . . . 60
3.5.1 AGeneralFrameworkforProcessManagement . . . 60
Contents vii
3.5.2 AnIterativeMethodologyandIllustration . . . . . . 63
3.6 Recommendations. . . . . . . . . . . . . . . . . . . . . . . 65
3.6.1 SampleMethodology . . . . . . . . . . . . . . . . . 66
3.7 SampleSynchronisationPointActivities . . . . . . . . . . . 69
3.7.1 ModelCatalogue:AnalysisandSearch/Browse/
Explore . . . . . . . . . . . . . . . . . . . . . . . . 70
3.7.2 ModelCatalogue:DesignandClassify/Enrich . . . . 71
3.7.3 SemanticBooster:ImplementationandStore/Query 72
3.7.4 Semantic Booster: Maintenance and Search/Browse/
Explore . . . . . . . . . . . . . . . . . . . . . . . . 72
3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.8.1 RelatedWork . . . . . . . . . . . . . . . . . . . . . 74
3.9 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4 ALIGNEDMetaModelOverview 79
RobBrennan,BojanBozic,OdhranGavinandMonikaSolanki
4.1 GenericMetamodel . . . . . . . . . . . . . . . . . . . . . 80
4.1.1 BasicApproach . . . . . . . . . . . . . . . . . . . . 80
4.1.2 NamespacesandURIs . . . . . . . . . . . . . . . . 81
4.1.3 ExpressivityofVocabularies . . . . . . . . . . . . . 82
4.1.4 ReferenceStyleforExternalTerms . . . . . . . . . 82
4.1.5 LinkswithW3CPROV. . . . . . . . . . . . . . . . 82
4.2 ALIGNEDGenericMetamodel . . . . . . . . . . . . . . . . 83
4.2.1 DesignIntentOntology(DIO) . . . . . . . . . . . . 83
4.3 SoftwareEngineering . . . . . . . . . . . . . . . . . . . . . 83
4.3.1 SoftwareLifeCycleOntology . . . . . . . . . . . . 83
4.3.2 SoftwareImplementationProcessOntology(SIP) . . 85
4.4 DataEngineering . . . . . . . . . . . . . . . . . . . . . . . 86
4.4.1 DataLifeCycleOntology . . . . . . . . . . . . . . 86
4.5 DBpediaDataID(DataID) . . . . . . . . . . . . . . . . . . 87
4.6 UnifiedQualityReports . . . . . . . . . . . . . . . . . . . . 89
4.6.1 ReasoningViolationOntology(RVO)Overview . . 89
4.6.2 W3CSHACLReportingVocabulary . . . . . . . . . 91
4.6.3 DataQualityVocabulary . . . . . . . . . . . . . . . 93
4.6.4 Test-DrivenRDFValidationOntology(RUT) . . . . 96
4.6.5 EnterpriseSoftwareDevelopment(DIOPP) . . . . . 109
4.6.6 UnifiedGovernanceDomainOntologies . . . . . . . 111
viii Contents
4.6.7 Semantic Booster and Model Catalogue Domain
Ontology . . . . . . . . . . . . . . . . . . . . . . . 112
4.6.7.1 Modelcatalogue . . . . . . . . . . . . . . 112
4.6.7.2 Booster . . . . . . . . . . . . . . . . . . . 113
4.6.8 PROV . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.6.9 SKOS . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.6.10 OWL . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.6.11 RDFS . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.6.12 RDF . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5 Tools 125
Kevin Feeney, Christian Dirschl, Katja Eck, Dimitris Kontokostas,
Gavin Mendel-Gleason, Helmut Nagy, Christian Mader and
AndreasKoller
5.1 ModelCatalogue . . . . . . . . . . . . . . . . . . . . . . . 125
5.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . 125
5.1.2 ModelCatalogue . . . . . . . . . . . . . . . . . . . 127
5.1.2.1 Architecture . . . . . . . . . . . . . . . . 127
5.1.2.2 Searchingandbrowsingthecatalogue . . 130
5.1.2.3 Editingthecataloguecontents . . . . . . . 131
5.1.2.4 Administration . . . . . . . . . . . . . . . 134
5.1.2.5 Eclipseintegrationandmodel-driven
development . . . . . . . . . . . . . . . . 134
5.1.2.6 Semanticreasoning . . . . . . . . . . . . 136
5.1.2.7 Automationandsearch . . . . . . . . . . 137
5.1.3 SemanticBooster . . . . . . . . . . . . . . . . . . . 138
5.1.3.1 Introduction . . . . . . . . . . . . . . . . 138
5.1.3.2 SemanticBooster . . . . . . . . . . . . . 139
5.2 RDFUnit . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.2.1 RDFUnitIntegration . . . . . . . . . . . . . . . . . 157
5.2.1.1 JUnitXMLreport-basedintegration . . . 158
5.2.1.2 Customapachemaven-basedintegration . 158
5.2.1.3 Theshapesconstraintlanguage
(SHACL) . . . . . . . . . . . . . . . . . 160
5.2.1.4 ComparisonofSHACLtoschemadefinition
usingRDFUnittestpatterns . . . . . . . . 161
5.2.1.5 Comparison of SHACL to auto-generated
RDFUnittestsfromRDFS/OWLaxioms . 162
Contents ix
5.2.1.6 Progress on the SHACL specification and
standardisationprocess . . . . . . . . . . 163
5.2.1.7 SHACLsupportinRDFUnit . . . . . . . 163
5.3 ExpertCurationToolsandWorkflows . . . . . . . . . . . . 164
5.3.1 Requirements . . . . . . . . . . . . . . . . . . . . . 165
5.3.1.1 Graduatedapplicationofsemantics . . . . 165
5.3.1.2 Graph–objectmapping . . . . . . . . . . 165
5.3.1.3 Object/document level state management
andversioning . . . . . . . . . . . . . . . 166
5.3.1.4 Object-basedworkflowinterfaces . . . . . 166
5.3.1.5 Integrated,automated,constraintvalidation 166
5.3.1.6 Resultinterpretation . . . . . . . . . . . . 167
5.3.1.7 Deferredupdates . . . . . . . . . . . . . . 167
5.3.2 Workflow/ProcessModels . . . . . . . . . . . . . . 167
5.3.2.1 Processmodel1–linkeddataobject
creation . . . . . . . . . . . . . . . . . . 167
5.3.2.2 Processmodel2object–linkeddataobject
updates . . . . . . . . . . . . . . . . . . . 168
5.3.2.3 Process model 3 – updates to deferred
updates . . . . . . . . . . . . . . . . . . . 168
5.3.2.4 Processmodel4–schemaupdates . . . . 169
5.3.2.5 Processmodel5–validatingschema
updates . . . . . . . . . . . . . . . . . . . 170
5.3.2.6 Processmodel6–namedgraphcreation . 170
5.3.2.7 Process model 7 – instance data updates
andnamedgraphs . . . . . . . . . . . . . 171
5.4 DacuraApprovalQueueManager . . . . . . . . . . . . . . 172
5.5 DacuraLinkedDataObjectViewer . . . . . . . . . . . . . . 172
5.5.1 CSPDesignofSeshatWorkflowUseCase . . . . . 173
5.5.2 Specification . . . . . . . . . . . . . . . . . . . . . 174
5.6 DacuraQualityService . . . . . . . . . . . . . . . . . . . . 176
5.6.1 TechnicalOverviewofDacuraQualityService . . . 177
5.6.2 DacuraQualityServiceAPI . . . . . . . . . . . . . 178
5.6.2.1 Resourceandinterchangeformat . . . . . 178
5.6.2.2 URI . . . . . . . . . . . . . . . . . . . . 178
5.6.2.3 Literals . . . . . . . . . . . . . . . . . . . 178
5.6.2.4 Literaltypes . . . . . . . . . . . . . . . . 178
5.6.2.5 Quads . . . . . . . . . . . . . . . . . . . 179
5.6.2.6 POSTvariables . . . . . . . . . . . . . . 180
