Table Of ContentData Quality
for Analytics
Using SAS®
Gerhard Svolba
SAS® Press
Data Quality
for Analytics
Using SAS
®
Gerhard Svolba
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
The correct bibliographic citation for this manual is as follows: Svolba, Gerhard. 2012. Data Quality for Analytics
Using SAS®. Cary, NC: SAS Institute Inc.
Data Quality for Analytics Using SAS®
Copyright © 2012, SAS Institute Inc., Cary, NC, USA
ISBN 978-1-61290-227-2 (electronic book)
ISBN 978-1-60764-620-4
All rights reserved. Produced in the United States of America.
For a hard-copy book: No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by any means, electronic, mechanical, photocopying, or otherwise, without the prior written permission
of the publisher, SAS Institute Inc.
For a Web download or e-book: Your use of this publication shall be governed by the terms established by the
vendor at the time you acquire this publication.
The scanning, uploading, and distribution of this book via the Internet or any other means without the permission of
the publisher is illegal and punishable by law. Please purchase only authorized electronic editions and do not
participate in or encourage electronic piracy of copyrighted materials. Your support of others’ rights is appreciated.
U.S. Government Restricted Rights Notice: Use, duplication, or disclosure of this software and related
documentation by the U.S. government is subject to the Agreement with SAS Institute and the restrictions set forth in
FAR 52.227-19, Commercial Computer Software-Restricted Rights (June 1987).
SAS Institute Inc., SAS Campus Drive, Cary, North Carolina 27513-2414
1st printing, April 2012
SAS® Publishing provides a complete selection of books and electronic products to help customers use SAS software
to its fullest potential. For more information about our e-books, e-learning products, CDs, and hard-copy books, visit
the SAS Publishing Web site at support.sas.com/publishing or call 1-800-727-3228.
SAS® and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS
Institute Inc. in the USA and other countries. ® indicates USA registration.
Other brand and product names are registered trademarks or trademarks of their respective companies.
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
For my three teenage sons and their permanent effort
in letting me share the many wonderful moments of their life,
and without whose help
this book would have probably been completed a year earlier.
You are the quality of my life.
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
iv
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
Acknowledgments
Martina, for supporting me and even the crazy idea to write this book in that period of our family life, which is
probably the busiest one.
My parents, for providing me with so many possibilities.
The following persons, who contributed to the book by spending time to discuss data quality topics. It is a
pleasure to work and to discuss with you: Albert Tösch, Andreas Müllner, Bertram Wassermann, Bernadette
Fabits, Christine Hallwirth, Claus Reisinger, Franz Helmreich, Franz König, Helmut Zehetmayr, Josef Pichler,
Manuela Lenk, Mihai Paunescu, Matthias Svolba, Nicole Schwarz, Peter Bauer, Phil Hermes, Stefan Baumann,
Thomas Schierer, and Walter Herrmann.
The reviewers, who took time to review my manuscript and provided constructive feedback and suggestions. I
highly appreciate your effort: Anne Milley, David Barkaway, Jim Seabolt, Mihai Paunescu, Mike Gilliland,
Sascha Schubert, and Udo Sglavo.
The nice and charming SAS Press team for their support throughout the whole process of the creation of this
book: Julie Platt, Stacey Hamilton, Shelley Sessoms, Kathy Restivo, Shelly Goodin, Aimee Rodriguez, Mary
Beth Steinbach, and Lucie Haskins.
The management of SAS Austria for supporting the idea to write this book: Dietmar Kotras and Robert Stindl.
August Ernest Müller, my great-grandfather, designed one of the very early construction plans of a helicopter in
1916 and was able to file a patent in the Austrian-Hungarian monarchy. However, he found no sponsor to
realize his project during World War I. I accidentally found his construction plans and documents at the time I
started writing this book. His work impressed and motivated me a lot.
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
vi
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
Contents
Introduction ....................................................................................................... ix
Part I Data Quality Defined .............................................................. 1
Chapter 1 Introductory Case Studies ................................................................... 3
Chapter 2 Definition and Scope of Data Quality for Analytics ............................. 21
Chapter 3 Data Availability ................................................................................ 31
Chapter 4 Data Quantity .................................................................................... 47
Chapter 5 Data Completeness ........................................................................... 59
Chapter 6 Data Correctness .............................................................................. 71
Chapter 7 Predictive Modeling .......................................................................... 83
Chapter 8 Analytics for Data Quality .................................................................. 95
Chapter 9 Process Considerations for Data Quality ......................................... 107
Part II Data Quality—Profiling and Improvement.......................... 121
Chapter 10 Profiling and Imputation of Missing Values .................................... 123
Chapter 11 Profiling and Replacement of Missing Data in
a Time Series ................................................................................ 141
Chapter 12 Data Quality Control across Related Tables ................................... 159
Chapter 13 Data Quality with Analytics ........................................................... 167
Chapter 14 Data Quality Profiling and Improvement with
SAS Analytic Tools ........................................................................ 181
Part III Consequences of Poor Data Quality—Simulation Studies . 199
Chapter 15 Introdution to Simulation Studies .................................................. 201
Chapter 16 Simulating the Consequences of Poor Data Quality for
Predictive Modeling ....................................................................... 207
Chapter 17 Influence of Data Quality and Data Availability on
Model Quality in Predictive Modeling ............................................. 219
Chapter 18 Influence of Data Completeness on Model Quality in
Predictive Modeling ....................................................................... 231
Chapter 19 Influence of Data Correctness on Model Quality in
Predictive Modeling ....................................................................... 243
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
viii Contents
Chapter 20 Simulating the Consequences of Poor Data Quality in
Time Series Forecasting ............................................................... 255
Chapter 21 Consequences of Data Quantity and Data Completeness in
Time Series Forecasting ............................................................... 265
Chapter 22 Consequences of Random Disturbances in
Time Series Data .......................................................................... 273
Chapter 23 Consequences of Systematic Disturbances in
Time Series Data .......................................................................... 281
Appendix A: Macro Code ................................................................................. 289
Appendix B: General SAS Content and Programs ............................................. 301
Appendix C: Using SAS Enterprise Miner for Simulation Studies ...................... 305
Appendix D: Macro to Determine the Optimal Length of
the Available Data History ........................................................... 311
Appendix E: A Short Overview on Data Structures and
Analytic Data Preparation ............................................................ 319
References ...................................................................................................... 327
Index ............................................................................................................... 329
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
Introduction
Rationale and Trigger to Write This Book
The first impulse
In November 2005, shortly before I finished the full draft of my first book, Data Preparation for Analytics
Using SAS, I was asked whether I wanted to contribute content and knowledge to the topic of data quality for
analytics. At that time it was too late to include data quality into my first book. It also would not have been
advisable to do so, as this important topic would have gone beyond the scope of the book on data preparation.
When Data Preparation for Analytics Using SAS was published in late 2006 I had already begun thinking about
starting a new book on the topic of data quality. However, it wasn’t until 2008 that I started collecting ideas and
opinions on the book you are reading now. After I received the green light to start writing the book from SAS
Publishing, I started work at the end of 2009.
Focus on analytics
My intention was not to write another book on data quality in general, but to write the first book that deals with
data quality from the viewpoint of a statistician, data miner, engineer, operations researcher, or other
analytically minded problem-solver.
Data quality is getting a lot of attention in the market. However, most of the initiatives, publications, and papers
on data quality focus on classical data quality topics, such as elimination of duplicates, standardization of data,
lists of values, value ranges, and plausibility checks. It will not be said here that these topics are not important
for analytics; on the contrary, they build the foundation of data for analysis. However, there are many aspects of
data that are specific to analytics. And these aspects are important to differentiate whether data are suitable for
analysis or not.
For classical data quality, books, best practices, and knowledge material are widely available. For the
implementation of data quality, SAS offers the DataFlux Data Management Platform, a market-leading solution
for typical data quality problems, and many methods in the established SAS modules.
Symbiosis of analytic requirements and analytic capabilities
In many cases, analytics puts higher requirements on data quality but also offers many more capabilities and
options to measure and to improve data quality, like the calculation of representative imputation values for
missing values. Thus there is a symbiosis between the analytical requirements and the analytical capabilities in
the data quality context.
Analytics is also uniquely able to close the loop on data quality since it reveals anomalies in the data that other
applications often miss. SAS is also perfectly suited to analyze and improve data quality.
• In part II, this book shows software capabilities that are important to measure and improve data quality
and, thus, close the loop in the data quality process and show how analytics can improve data quality.
• In part III, this book shows how SAS can be used as a simulation environment for the evaluation of data
quality status and the consequences of inferior data quality. This part also shows new and unique
simulations results on the consequences of data quality.
Svolba, Gerhard. Data Quality for Analytics Using SAS®. Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA.
ALL RIGHTS RESERVED. For additional SAS resources, visit support.sas.com/publishing.
