Agile Data Warehousing for the Enterprise. A Guide for Solutions Architects and Project Leaders

523 Pages·2016·39.813 MB·English

by  Hughes, Ralph

Agile Data Warehousing for the Enterprise
A Guide for Solution Architects and Project Leaders
Agile Data Warehousing
for the Enterprise
A Guide for Solution Architects
and Project Leaders
Ralph Hughes, MA, PMP, CSM
AMSTERDAM(cid:1)BOSTON(cid:1)HEIDELBERG(cid:1)LONDON(cid:1)NEWYORK(cid:1)OXFORD(cid:1)PARIS
SANDIEGO(cid:1)SANFRANCISCO(cid:1)SINGAPORE(cid:1)SYDNEY(cid:1)TOKYO
MorganKaufmannisanimprintofElsevier
AcquiringEditor:SteveElliot
EditorialProjectManager:LindsayLawrence
ProjectManager:PriyaKumaraguruparan
CoverDesigner:MarkRogers
MorganKaufmannisanimprintofElsevier
225WymanStreet,Waltham,MA02451,USA
Copyrightr2016ElsevierInc.Allrightsreserved.
Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,electronicormechanical,
includingphotocopying,recording,oranyinformationstorageandretrievalsystem,withoutpermissioninwritingfrom
thepublisher.Detailsonhowtoseekpermission,furtherinformationaboutthePublisher’spermissionspoliciesandour
arrangementswithorganizationssuchastheCopyrightClearanceCenterandtheCopyrightLicensingAgency,canbe
foundatourwebsite:www.elsevier.com/permissions.
ThisbookandtheindividualcontributionscontainedinitareprotectedundercopyrightbythePublisher(otherthan
asmaybenotedherein).
Notices
Knowledgeandbestpracticeinthisfieldareconstantlychanging.Asnewresearchandexperiencebroadenourunderstanding,
changesinresearchmethods,professionalpractices,ormedicaltreatmentmaybecomenecessary.
Practitionersandresearchersmustalwaysrelyontheirownexperienceandknowledgeinevaluatingandusinganyinformation,
methods,compounds,orexperimentsdescribedherein.Inusingsuchinformationormethodstheyshouldbemindfuloftheirown
safetyandthesafetyofothers,includingpartiesforwhomtheyhaveaprofessionalresponsibility.
Tothefullestextentofthelaw,neitherthePublishernortheauthors,contributors,oreditors,assumeanyliabilityforanyinjury
and/ordamagetopersonsorpropertyasamatterofproductsliability,negligenceorotherwise,orfromanyuseoroperation
ofanymethods,products,instructions,orideascontainedinthematerialherein.
ISBN:978-0-12-396464-9
BritishLibraryCataloguing-in-PublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
LibraryofCongressCataloging-in-PublicationData
AcatalogrecordforthisbookisavailablefromtheLibraryofCongress
ForinformationonallMorganKaufmannpublications
visitourwebsiteatwww.mkp.com
Advance Praise for
Agile Data Warehousing
for the Enterprise
Agile Data Warehouse for the Enterprise is a must read for any data professional tasked with delivering enter-
prisereporting andanalytics inthenimble,speed-to-value environment thatwe findourselves intoday.It marries
anagilemethodologywithdatawarehousing’sbestpracticestocreateablueprintfordeliveringvaluefast.
(cid:1)NikGreen,Director ofBusinessIntelligence foramulti-billiondollarfoodretailer
This comprehensive guide provides asolidandcomplete foundation foragile EDW developmentincludingrevolu-
tionary new paradigms. I especially like the ‘out of the box’ thinking, practicality of the techniques, and the
research and case studies backing up the validity of the proposed approaches. Recommend reading for all DW/BI
technicalleaders.
(cid:1)LenSilverston,CEOofUniversalDataModelsandauthoroftheDataModelResourceBookseries
Ralph’s book goes way beyond just agile programming—it illustrates an iterative approach to the full develop-
ment life cycle and is particularly relevant to issues of data quality that we focus on at DAMA. The hyper model-
ing techniques in particular will allow teams to avoid the death trap of producing big, risky application designs
up-frontbeforeaproject’srequirementsarefullyknown.
(cid:1)KenDunn,PresidentofDAMA’sHoustonchapter
Ralph’s notion of “80/20 specifications” for data warehousing projects really worked, saving our business part-
ners the pain of doing an exhaustive requirements specification up-front. This approach got the team developing
the most important features first, letting the product owner fill in details on topics as each one came up later dur-
ingdevelopment.
(cid:1)NaveenThalanki,ProjectManagerforaFortune500company
If you’re stillprogramming data warehousesby hand, you’re wasting 90percentofyour time andmoney.Ralph’s
guide for project leaders not only outlines the automated, metadata-driven development we’ve been practicing for
years in the Netherlands, but also links that practice to agile requirements, coding, and quality assurance. It is a
uniquebook,amustread.
(cid:1)RonaldDamhof,DW/BIConsultanttotheDutchCentralBank
The industry has struggled to bring the mechanics and benefits of Agile to the data warehousing and business
intelligence communities. Ralph’s work in this area is timely and important to assist in driving success within
your enterprise and project teams. Following the guidance found in this book will help you deliver value at a
rapidpace.
(cid:1)TomHammergren,CTOofCordataHealthcareInnovationsandauthorof
DataWarehousingforDummies
This book presents a thorough approach to building quality into enterprise data warehouses, mitigating risk by
applying proven models for testing, and detailing alternatives so that you can experiment to find the alternatives
that work best for your team. These aren’t mere off-the-cuff ideas, but proven techniques to overcoming the huge
challengesofEDWbyapplyingagileprinciples,illustratedwithstoriesfromthetrenches.
(cid:1)LisaCrispin,Co-AuthorofMoreAgileTesting:LearningJourneysfortheWholeTeam,and
AgileTesting:APracticalGuideforTestersandAgileTeams
Ralphhascustomizedtheagiledevelopmentmethodologyfortheuniqueneedsofbusinessintelligence.Myproject
teams were able to quickly understand and apply his approach without losing time to adapting generic Scrum
techniquestotherigorsofdataintegrationandanalytics.
(cid:1)RonLewis,programmanagerataFortune1000financialservicescompany
Agile Data Warehousing for the Enterprise is a “how to” book with innovative method and process components
such as hyper data modeling and an iterative sub-release value cycle. Ralph provides a clear outline of the con-
cepts,methods,andframeworksyou’llneedtoassembleaworld-classBI/DWprogramofyourown.
(cid:1)HansHultgren,authorofModelingtheAgileDataWarehousewithDataVaultandBI/DWindustryadvisor
AsafriendofRalphandfellowAgileDW/BIauthor,Iamveryexcitedaboutthisbook.Ralph’sextensiveresearch
into what works, his hands-on application of those techniques, and his well-organized method of presenting infor-
mation,arecombinedinthisgame-changingbookfordatawarehousepractitionersseekinggreateragility.
(cid:1)KenCollierPhD,authorofAgileAnalytics
Agile techniquesandRalph’sadaptations for enterprisedata warehousinghave dramatically improvedour ability
to understand business needs and to then plan, track, and deliver upon those needs for large and small projects.
This book provides a thorough treatment of the vital leadership practices for agile projects including require-
ments,architecture,andqualityassurance.
(cid:1)RichardTench,Manager,InformationDelivery,majorCanadianinsurancecompany
List of Figures
Figure1.1 Thenegativefeedbacklooppresentinmosttraditionallymanagedprojects. 2
Figure1.2 Thefivemajorcomponentstoagileenterprisedatawarehousing. 4
Figure1.3 AgileEDWpracticesswitchprojectstoapositivefeedbackloop. 4
Figure1.4 HowateammightacquireagileEDWtechniquesworkingfromtheinsideout. 8
Figure2.1 MindmapofgenericiterativemethodssummarizedinChapters2and3. 14
Figure2.2 AfamilytreeofmethodsandinfluencesleadingtotheagileEDWmethod. 15
Figure2.3 Thetraditionalwaterfallmethod. 17
Figure2.4 TheAgilemanifestocoverpage. 17
Figure2.5 ValuesandprinciplesoftheagilemanifestoandExtremeProgramming. 18
Figure2.6 TheessenceoftheScrummethod. 21
Figure2.7 Typicaluserstory. 22
Figure2.8 AsampleScrumtaskboardasitwouldappearinmid-iteration. 24
Figure2.9 AScrumburndownchartasitwouldappearinmid-iteration. 25
Figure3.1 Leanvalues,principles,andtools. 33
Figure3.2 Value-streamanalysisofdevelopmentworkforachallengedwaterfallproject. 34
Figure3.3 TypicalKanbanworkboard. 42
Figure3.4 Kanban-stylecumulativeflowdiagram. 43
Figure3.5 SamplecycletimedistributionanalysisforaKanbanteam. 44
Figure3.6 Typicalstagesof“Scrumban”—thetransitionfromScrumtoKanban. 48
Figure3.7 Two-tieredScrumbantaskboard. 48
Figure3.8 ValuesandprinciplesoftheRationalUnifiedProcess. 50
Figure3.9 RUPWhaleChart. 51
Figure3.10 GoogleNgramof“Scrum”and“RUP”through2008. 53
Figure4.1 Businessorganizationaltermsusedinthisbook. 64
Figure4.2 Businessconceptualmodel. 72
Figure4.3 Logicaldatamodel. 72
Figure4.4 Physicaldatamodel. 73
Figure4.5 Sampleenterprisedatawarehouse“referencearchitecture”. 75
Figure4.6 Zachmanframeworkadaptedforanenterprisedatawarehousingprogram. 77
Figure4.7 DAMA’sframeworkfordatamanagementfunctions. 78
Figure4.8 Hammergren’smatrixforsequencingDW/BIdevelopmentwork. 79
Figure5.1 TypicalRUP-stylewhalechartforanagileEDWproject. 91
Figure5.2 AgileEDWuserstoriesresultintoomanydeveloperstoriesforone,shortIteration. 92
Figure5.3 Derivingdeveloperstoriesfromuserstories. 94
Figure5.4 A“currentestimate”foranagiledatawarehousingproject. 96
Figure5.5 Agiledatawarehousingrequirespipelinedworkspecialties. 99
Figure5.6 Workpackagestendtoflowdiagonallyacrosstechnicalspecialtiesanditerations. 101
Figure5.7 Cycletimedistributionanalysisforanagiledatawarehousingproject. 103
Figure5.8 Acurrentestimateadjustedforobserveddeliverycycletimes. 104
Figure5.9 Successratesforagiledatawarehousingteams,bynumberofagileprojectscompleted,compared 105
totraditionalmethods.
Figure5.10 Agile’simpactuponkeyperformanceindicatorsfordatawarehousingdevelopmentprojects. 105
Figure5.11 Agiledatawarehousingsurveysindicatethatpractitionershaveovercomesomechallengeareas. 107
Figure6.1 Relativecostofcorrectingdefectsgrowsby100betweenrequirementsandpromotionintoproduction. 112
Figure6.2 Incrementaldeliverymitigatesriskbyincreasingthenumberofproductcheckpoints. 113
Figure6.3 ThesourcesofEDWprojectriskmitigatedwiththreetypesofiterations. 115
Figure6.4 RelativetimingforthethreetypesofiterationsthatAgileEDWemploys. 118
Figure7.1 MindmapoftopicsaddressedinPartIII. 126
Figure7.2 SampleEDWrequirementsexpressedatthreelevels. 127
Figure7.3 Waterfall-stylerequirementsmanagement. 131
Figure7.4 Typicalrequirementsworkbreakdownforatraditionalproject. 133
xvii
xviii List of Figures
Figure7.5 As-isbusinessprocessdiagramshowingasampleworkflowrequiringre-engineering. 135
Figure7.6 To-bebusinessprocessre-engineeredtouseEDWtocommunicatebetweenagents. 136
Figure7.7 Accuracyvs.precision. 137
Figure7.8 Standardriskanalysis. 140
Figure7.9 Standardanalysisadjustedfordollarvalueofeachtypeofrisk. 141
Figure7.10 AgileEDW’srequirementsmanagementbenefitsgreatlyfromintersectingvaluechains. 146
Figure7.11 OverallagileEDWrequirementsmanagementplan. 148
Figure7.12 Enterpriserequirementsmanagementroles. 150
Figure8.1 Bigpicture(cid:1)decomposingepicsintoabacklogofstories. 152
Figure8.2 Immediatebusinessstakeholderformalizingalllevelsofstoriesbylinkingthemtothehierarchy 153
amongbusinessstakeholders.
Figure8.3 Primarytechniquefordecomposinguserstoriesintodevelopersstories.Notethe25-to-1multiplierforthis 158
project’suserstory.
Figure8.4 INVESTandDILBERT’Stest. 158
Figure8.5 Bigpicture(cid:1)recomplingmodulesforperceivedvalue. 160
Figure8.6 Valuebuild-upchartsdistingishingbetweendeliveryenvironments. 166
Figure9.1 Usermodelingexample. 171
Figure9.2 Businessintelligenceuser’shierarchyofneedsfortheexampleproject. 172
Figure9.3 Mindmap&fishbonediagrams. 175
Figure9.4 Previousmindmapre-drawnasafishbonediagram. 175
Figure9.5 Frontandbackofaprojectvisionbox. 177
Figure9.6 Exampleofaprojectvisionstatement. 177
Figure9.7 Productroadmapformattedfordiscussionswithproductowner. 178
Figure9.8 Productroadmapformattedforpresentationtoconflictingstakeholders. 179
Figure10.1 Threeimportantdimensionstoapplicationrequirements. 182
Figure10.2 StreamlinedtemplateforastreamlinedSponsor’sConceptBriefing(SCB). 187
Figure10.3 StreamlinedtemplateforastreamlinedStakeholderRequest(SHR). 190
Figure10.4 TemplateforastreamlinedVisionDocument(VDoc). 192
Figure10.5 Samplevisiondocumentsolutionstatements. 193
Figure10.6 Samplevisiondocumentsolutionstatements. 193
Figure10.7 Definingabusinesssolution. 194
Figure10.8 Samplecontextdiagramforavisiondocument. 195
Figure10.9 Sampletargetbusinessmodelforavisiondocument. 196
Figure10.10 Samplehighlevelarchitecturediagramforavisiondocument. 198
Figure10.11 Targetbusinessmodelforasubreleaseoverview. 199
Figure10.12 Subreleasescopedrawnonadimensionsofvaluediagram. 201
Figure10.13 Subreleaseplansummaryonafront-enddimensionsofvaluediagram. 201
Figure10.14 TemplateforastreamlinedSubreleaseOverview(SRO) 204
Figure10.15 Usecasemodelforasubreleasedescription. 205
Figure10.16 Venndiagramforasubreleasedescription. 206
Figure10.17 Datavalidationstepsforasubreleasedescription. 206
Figure10.18 TemplateforastreamlinedModuleUseCase(MUC). 210
Figure10.19 Communicatingthemainflowofeventswithalevel2dataflowdiagram. 211
Figure10.20 Exampleofasource-to-targetmap. 213
Figure11.1 Tracingrequirementsbetweenvaluechains. 216
Figure11.2 Corporate-levelplanningfunctionsthatgeneratearchitecturalrequirements. 218
Figure11.3 Typicalprojectreleasecycleusedbylargecompanies. 222
Figure11.4 FittingRMartifactsintothepre-developmentiterations. 223
Figure11.5 Preparingthepre-developmentestimateforanewteam. 224
Figure11.6 Requirementsmanagementeffortcurvesandtimingofartifactsoverlengthofaproject(partI). 233
Figure11.7 Requirementsmanagementeffortcurvesandtimingofartifactsoverlengthofaproject(partII). 234
Figure11.8 TheCRISP-DMprocessfordatamining. 237
Figure11.9 InterfacingagileEDWRMwithprojectgovernance. 240
Figure12.1 BasicEDWreferencearchitecturewithdataparadigmslisted. 250
Figure12.2 Giventheadventofhyper-modeledforms,EDWprojectleadersnowhavefourdatamodelingparadigmsto 251
choosefrom.
Figure12.3 Examplesofstandardnormalformdatamodelsofincreasingcomplexity. 252
Figure12.4 Examplesofconformeddimensionalformmodelsofincreasingcomplexity. 254
Figure12.5 Exampleofahypernormalizeddatamodel. 255
Figure12.6 Themainportionofahypergeneralizeddatamodel. 256
Figure12.7 Dataarchitectures,paradigms,andmodels. 257
Figure12.8 Business,logical,andphysicaldatamodels. 259
Figure12.9 Theupdateanomaliesdatanormalizationisdesignedtoprevent. 261
Figure12.10 Contextdiagramforthenormalizationexample. 263
ListofFigures xix
Figure12.11 Samplecase’sdatainitsstartingarrangement,i.e.,zerothnomalform. 264
Figure12.12 Impactofafirstnormalformcorrectionuponsamplecase’sdatamodel. 265
Figure12.13 Impactofasecondnormalformcorrectionuponsamplecase’sdatamodel. 266
Figure12.14 Impactofathirdnormalformcorrectionuponsamplecase’sdatamodel. 267
Figure12.15 Impactofafourthnormalformcorrectionuponsamplecase’sdatamodel. 269
Figure12.16 Impactofafifthnormalformcorrectionuponsamplecase’sdatamodel. 273
Figure12.17 Levelzerodatageneralization. 275
Figure12.18 Levelonedatageneralizationforthepartymodel. 275
Figure12.19 Leveltwodatageneralizationforthepartymodel. 276
Figure12.20 Datageneralizationroll-uppatterns. 276
Figure12.21 Levelthreedatageneralizationforthepartymodel. 277
Figure12.22 Thehub&spokeconceptionofanenterprisedatawarehouse. 279
Figure12.23 AschematicrepresentationofasimplesubjectareainanEDWpresentationlayer. 280
Figure12.24 Standardnormalformmodelsarebrittleinthefaceofchangingrequirements. 283
Figure12.25 ChangeCases#3and#4forconformeddimensionalformmodel. 290
Figure13.1 Asurfacesolutionwitharchitecturalbackfillingthatleverages“ShadowIT”. 295
Figure13.2 Basicdataintegrationusecase,deliveredwithoutdatavirtualization. 298
Figure13.3 Basicdataintegrationusecase,deliveredusingadatavirtualizationserver. 298
Figure13.4 Datavirtualizationreducesthenumberofinterfacesrequiredforagivensetofsolutions. 301
Figure13.5 Generalsurface-solutionsdeliverypatternwithdatavirtualization. 303
Figure13.6 Dimensions-of-valueanalysisforsurfacesolutionswithdatavirtualization. 304
Figure13.7 Datavirtualization’svaluepropositionforagileEDWteams. 306
Figure13.8 Surfacesolutions“channels”enabledbydatavirtualization. 307
Figure13.9 NotableApacheHadoopsoftwarecomponents. 312
Figure13.10 ProcessingpatternforasimpleMap/Reducejoinoperation. 314
Figure13.11 SampleMap/Reducecodeforasimpletwo-tablejoin. 315
Figure13.12 SolutionarchitecturefortheFacebookHivedatawarehousingexample. 320
Figure13.13 Cycle-timeanalysisforbuildingthecomponentsoftraditionalandbig-dataDW/BIsolutions. 322
Figure13.14 Cycle-timeanalysis(cid:1)cumulativetimeinlearningcycles.Showinghowtimeinvestedaccumulatesaslearning 324
cyclesarerepeated.
Figure13.15 Cycle-timeanalysis(cid:1)cumulativetimeinusagecycles.Showinghowtimeinvestedgrowsasapplicationis 324
adaptedbythousandsofendusersansweringeverydaybusinessquestions.RapidlygrowingcostofHDFSinthis
scenarioshouldgivepausetothoseconsideringroutingallofanorganizationsinformationintoa“datalake.”
Figure13.16 BigdataandtraditionalRDBMSsareconverging. 325
Figure13.17 Asuccessionofsurfacesolutionsleveragingbigdata. 326
Figure13.18 EDWreferencearchitecturewithsurfacesolutionsemployingbigdatatechnology. 326
Figure14.1 Ensembledatamodeling. 330
Figure14.2 Familytreeofhypernormalizedmodelingapproaches. 331
Figure14.3 Hypernormalizinga3NFdatamodel(cid:1)startingpoint. 332
Figure14.4 Structuresforthestartingmodel(in3rdnormalform). 333
Figure14.5 HypernormalizationStep1(cid:1)declarebusinesskeys. 334
Figure14.6 Structuresforthehypernormalizedmodel(cid:1)businesskeysandtheirattributes. 335
Figure14.7 HypernormalizationStep2(cid:1)installmany-to-manylinksbetweenbusinesskeys. 336
Figure14.8 Structuresforthehypernormalizedmodel(cid:1)linksandtheirattributes. 337
Figure14.9 HypernormalizationStep3(cid:1)splitoutallattributestotheirowntables. 338
Figure14.10 Hypernormalizedmodelwithabbreviateddepictionoflinkandattributeentities. 338
Figure14.11 Thirdnormalformdatawarehousesareheavilyimpactedbynewentities. 340
Figure14.12 Linkingtablesinahypernormalizeddatawarehouseinsulateexistingtablesagainstdisruptionwhen 341
newentitiesareadded.
Figure14.13 Datavaultmodelexcerptshowingbusinesskeysandlinkingentitieswithmultipleattributetables. 342
Figure14.14 AnchormodeledequivalentoftheHNFordermodel. 344
Figure14.15 Onlyafew,parm-drivenETLmodulesareneededtoloadthebulkofthedatawarehouse. 346
Figure14.16 Prototypesforreusablehypernormalizedloadmodules. 347
Figure14.17 Driverscriptemployingreusableloadmodules. 348
Figure14.18 Driverscriptemployingreusableloadmodules 350
Figure14.19 Prototypesofreusabletestwidgetsandadriverscriptcallingthem. 351
Figure14.20 SQLquerydemonstratingthecorrelatedsubqueriesneededtoretrieveinformationfromhypernormalized 355
datawarehouses.
Figure14.21 Hypernormalizeddesignscanrequiremanycorrelatedsubqueries. 357
Figure14.22 Byusingpoint-in-timetableswhereneeded,wecansimplifyretrievalqueries. 358
Figure14.23 Dataretrievalqueriescanbe(a)simplifiedthroughtablepruningand(b)generatedfromDBMSconstraints. 359
Figure14.24 Columnsintheresultingbridgetable. 360
Figure14.25 EDWreferencearchitectureadaptedforahypernormalizedintegrationlayer. 362
Figure14.26 FourstylesfordistributinghypernormalizedrepositoriesacrosstheEDWreferencearchitecture. 363
xx List of Figures
Figure14.27 DatamodelsforChangeCase#1underhypernormalization. 366
Figure14.28 HypernormalizedmodelneededtosolveChangeCase2. 368
Figure14.29 Joinsofexistingandnewtablesneededtofeedload_link()fortheLink_Party_Order_Installertable. 369
Figure14.30 Joinsofexistingandnewtablesneededtofeedload_link()fortheLink_Party_Rolluptableofdealership 369
relationships.
Figure14.31 Surfacesolutionpatternsemployingahypernormalizedintegrationlayer. 370
Figure15.1 Decomposingdataintoahypergeneralizeddatastoreandthenprojectingittoastarschema. 376
Figure15.2 HypergeneralizingaHNFdatamodel(cid:1)startingpoint. 377
Figure15.3 HypergeneralizationStep1(cid:1)addmetadatatables. 378
Figure15.4 HypergeneralizationStep2(cid:1)simplifytoonetableperfunction. 378
Figure15.5 HypergeneralizationStep3(cid:1)“shred”attributesintoname-valuepairs. 379
Figure15.6 HypergeneralizationStep4(cid:1)temporalizethingtyperelationships. 379
Figure15.7 HypernormalizationStep5(cid:1)temporalizetheremainingentities. 380
Figure15.8 Businessmodelsaremachinereadable. 382
Figure15.9 Dimensionalobjectsfromthebusinessmodeltranslatedtorecordsintheassociativedatamodel. 382
Figure15.10 Samplerecordsforthingsandlinksintheassociativedatastore. 383
Figure15.11 Shreddingattributesintoname-valuepairs. 385
Figure15.12 Pivotingshreddedattributesbacktotheiroriginalformat. 386
Figure15.13 Datawarehousebusinessmodelusedforthechangecases. 388
Figure15.14 Exampleofhowgraphicalmodelchangesimpacttheassociativedatastore. 389
Figure15.15 ChangeCase1’sdatatransformfordimensionsbeforethebusinessmodelisupdated. 390
Figure15.16 StartingdatatransformforthetransactiondataofChangeCase1. 392
Figure15.17 HypergeneralizeddatawarehouseautomationsystemscanaddressthefullEDWreferencearchitecture. 393
Figure15.18 HelpertablesallowEDWadminstowritequeriesagainstbusinessobjects. 394
Figure15.19 RecordsimpactedbyflatteningthehierarchybetweenOrdersandeSegment 396
Figure15.20 StepstoupdatingahypergeneralizedEDW’sdimensionalentitiesandtheirdata. 397
Figure15.21 Creatingderivedcolumnsandmasterdataelementsusingvalue-addedloops. 398
Figure15.22 Usingthemasterdatamanagementutilityofthedatawarehouseautomationtool. 399
Figure15.23 Sampleworkflowformasterdataprocessing,highlightingtheroleofthedatastewards. 400
Figure15.24 Masterdatamanagementfrontendshowingsingle-recordcorrectionscreen. 401
Figure15.25 EDWreferencearchitectureupdatedtoincludemasterdatamanagementlayers. 402
Figure15.26 Hypergeneralizeddatawarehouseperformancebenchmarks. 403
Figure15.27 BusinessmodelchangesneededtoaccomplishChangeCase1. 404
Figure15.28 DatatransformneededforChangeCase1aftermodificationsaremade. 405
Figure15.29 Hypergeneralizedreportingcansuccessfullyspanachangeinbusinessmodels 407
Figure15.30 BusinessmodelchangesneededtoaccomplishChangeCase2. 407
Figure15.31 Pre-loadingtheCorporatePartyobjectsforcustomersinChangeCase2. 408
Figure15.32 DatatransformneededforChangeCase2aftermodificationsaremade. 409
Figure15.33 HGFquerywriterautomaticallyspansmodelingchanges.Thedatawarehouseautomationsystemprovides 410
defaultsupertypeentityreferencesforcustomerrecordsthatexistedbeforethemodelingchange.
Figure15.34 BusinessmodelupdateandresultingreportingforChangeCase4. 411
Figure15.35 DimensionaldatatransformchangesneededtoaccomplishChangeCase4. 412
Figure15.36 TransactiondatatransformchangesneededtoaccomplishChangeCase4. 413
Figure15.37 Surfacesolutionpatternsemployingahypergeneralizedintegrationlayer. 415
Figure16.1 Thelevel-of-effortneededtodeterminetheroot-causeofadefectincreasesexponentiallywiththe 427
numberofdefectsthatexist.
Figure16.2 Visualizingqualityviathenumberoftestsexecutingorpassingbyenvironment. 429
Figure16.3 Theoptimalleveloftestingisabalancebetweentwotypesofrisks. 430
Figure16.4 Stepsinthetest-leddevelopmentapproach. 432
Figure16.5 Relationshipsbetweentesttermsasusedinthisbook. 435
Figure16.6 ThedifferencebetweenQC,QA,andQM. 437
Figure16.7 Relationshipsbetweentwotestingdimensionsandphysicalobjects. 438
Figure16.8 Usingtheagile232QAplanningmatrixtovisualizeateam’schoiceoftesttypes. 442
Figure16.9 “Datacorners”testtechniqueformodelsinstandardnormalformandconformeddimensionalform. 445
Figure16.10 Typicalsituationrequiringateamtousethe“expectedvalues”testtechnique. 447
Figure16.11 Manyunittestsrollforwardintotheapplicationsintegrationtestsuite. 454
Figure16.12 OverviewofanagileQAplanningapproach. 455
Figure17.1 V-Modelshowingqualityassuranceastheflipsideofrequirementswork. 459
Figure17.2 V-Modeladaptedforagiledatawarehousingandshowingtheauthorsandconsumersofrequirementsand 460
specifications.
Figure17.3 Usingthe232QAplanningmatrixupdatedtocommunicatetestwritingresponsibilities. 461
Figure17.4 Teamscanavoid“oversocializing”decisionsbyemployinga“one-up,one-down”validationpractice. 463
Figure17.5 SequencingQAworkwithinaniteration. 464
Figure17.6 Qualityassuranceworklinkedtothelargerprojectcyclessurroundingdevelopmentiterations. 467
ListofFigures xxi
Figure17.7 The232QAplanningmatrixupdatedtoshowwhentestcaseswillrun. 468
Figure17.8 LocatingQAworkamongadatawarehouse’sexecutionenvironments. 468
Figure17.9 232QAplanningmatrixupdatedtoshowwheretestcasesshouldexecute. 470
Figure18.1 232QAplanningmatrixcommunicatinghowteamwillexecutetestcases. 481
Figure18.2 FullregressiontestingforanEDWrequiresmanydatasets. 484
Figure18.3 Automatedtestingcycleforasingletestingscenario. 489
Figure18.4 OverviewofautomatedtestingbyscenarioforanEDW. 490
Figure18.5 Relationshipbetweentestcase,testassertions,andthedatarequiredforeach. 492
Figure18.6 Testcasebuild-upchartforasingleiteration. 492
Figure18.7 Visualizingqualityviasummarytestresultsbysubjectareaandarchitecturallayer. 493
Figure18.8 Asampleprojectqualitydashboardshowingfourmeasuresofqualityachievedbyadevelopmentteam. 495
Figure18.9 Supportingqualityfulfillmentdocumentationwithanautomatedtestengine. 497
Figure19.1 Enterpriseinformationmanagementincludesabusiness-leddatagovernannceprogramandanIT-ledinformation 506
managementprogram.
Figure19.2 AgileEDWprojectstart-upalignswellwiththedatagovernancecycle. 507
Figure19.3 AgileEDWsubreleasecycle. 510
Figure19.4 AgileEDWsubreleasecycleshowingsupportfordatagovernance. 514
Figure19.5 AgileEDWsubreleasecycleshowingsupportforqualityassurance. 516