Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data

176 Pages·2011·2.08 MB·English

by  IBM

Understanding Big Data
About the Authors
Paul C. Zikopoulos, B.A., M.B.A., is the Director of Technical Professionals
for IBM Software Group’s Information Management division and addition-
ally leads the World Wide Database Competitive and Big Data SWAT teams.
Paul is an internationally recognized award-winning writer and speaker
with more than 18 years of experience in Information Management. Paul has
written more than 350 magazine articles and 14 books on database technolo-
gies, including DB2 pureScale: Risk Free Agile Scaling (McGraw-Hill, 2010);
Break Free with DB2 9.7: A Tour of Cost-Slashing New Features (McGraw-Hill,
2010); Information on Demand: Introduction to DB2 9.5 New Features (McGraw-
Hill, 2007); DB2 Fundamentals Certification for Dummies (For Dummies, 2001);
DB2 for Windows for Dummies (For Dummies, 2001), and more. Paul is a DB2
Certified Advanced Technical Expert (DRDA and Clusters) and a DB2 Certi-
fied Solutions Expert (BI and DBA). In his spare time, he enjoys all sorts of
sporting activities, including running with his dog, Chachi; avoiding punches
in his MMA training; trying to figure out why his golf handicap has unex-
plainably decided to go up; and trying to understand the world according to
Chloë, his daughter. You can reach him at [email protected]. Also, keep
up with Paul’s take on Big Data by following him on Twitter @BigData_paulz.
Chris Eaton, B.Sc., is a worldwide technical specialist for IBM’s Information
Management products focused on Database Technology, Big Data, and
Workload Optimization. Chris has been working with DB2 on the Linux,
UNIX, and Windows platform for more than 19 years in numerous roles,
from support, to development, to product management. Chris has spent his
career listening to clients and working to make DB2 a better product. He is
the author of several books in the data management space, including The
High Availability Guide to DB2 (IBM Press, 2004), IBM DB2 9 New Features
(McGraw-Hill, 2007), and Break Free with DB2 9.7: A Tour of Cost-Slashing New
Features (McGraw-Hill, 2010). Chris is also an international award-winning
speaker, having presented at data management conferences across the globe,
and he has one of the most popular DB2 blogs located on IT Toolbox at
http://it.toolbox.com/blogs/db2luw.
Dirk deRoos, B.Sc., B.A., is a member of the IBM World-Wide Technical
Sales Team, specializing in the IBM Big Data Platform. Dirk joined IBM 11
years ago and previously worked in the Toronto DB2 Development lab as its
Information Architect. Dirk has a Bachelor’s degree in Computer Science and
a Bachelor of Arts degree (Honors in English) from the University of New
Brunswick.
Thomas Deutsch, B.A, M.B.A., serves as a Program Director in IBM’s Big
Data business. Tom has spent the last couple of years helping customers with
Apache Hadoop, identifying architecture fit, and managing early stage proj-
ects in multiple customer engagements. He played a formative role in the
transition of Hadoop-based technologies from IBM Research to IBM Software
Group, and he continues to be involved with IBM Research Big Data activities
and the transition of research to commercial products. Prior to this role, Tom
worked in the CTO office’s Information Management division. In that role,
Tom worked with a team focused on emerging technologies and helped cus-
tomers adopt IBM’s innovative Enterprise Mashups and Cloud offerings.
Tom came to IBM through the FileNet acquisition, where he had responsibil-
ity for FileNet’s flagship Content Management product and spearheaded
FileNet product initiatives with other IBM software segments including the
Lotus and InfoSphere segments. With more than 20 years in the industry and
a veteran of two startups, Tom is an expert on the technical, strategic, and
business information management issues facing the enterprise today. Tom
earned a Bachelor’s degree from the Fordham University in New York and an
MBA from the Maryland University College.
George Lapis, MS CS, is a Big Data Solutions Architect at IBM’s Silicon Valley
Research and Development Lab. He has worked in the database software area
for more than 30 years. He was a founding member of R* and Starburst re-
search projects at IBM’s Almaden Research Center in Silicon Valley, as well as
a member of the compiler development team for several releases of DB2. His
expertise lies mostly in compiler technology and implementation. About ten
years ago, George moved from research to development, where he led the
compiler development team in his current lab location, specifically working on
the development of DB2’s SQL/XML and XQuery capabilities. George also
spent several years in a customer enablement role for the Optim Database tool-
set and more recently in IBM’s Big Data business. In his current role, George is
leading the tools development team for IBM’s InfoSphere BigInsights platform.
George has co-authored several database patents and has contributed to nu-
merous papers. He’s a certified DB2 DBA and Hadoop Administrator.
About the Technical Editor
Steven Sit, B.Sc., MS, is a Program Director in IBM’s Silicon Valley Research
and Development Lab where the IBM’s Big Data platform is developed and
engineered. Steven and his team help IBM’s customers and partners evalu-
ate, prototype, and implement Big Data solutions as well as build Big Data
deployment patterns. For the past 17 years, Steven has held key positions
in a number of IBM projects, including business intelligence, database tool-
ing, and text search. Steven holds a Bachelor’s degree in Computer Science
(University of Western Ontario) and a Masters of Computer Science degree
(Syracuse University).
Understanding Big Data
Analytics for Enterprise Class
Hadoop and Streaming Data
Paul C. Zikopoulos
Chris Eaton
Dirk deRoos
Thomas Deutsch
George Lapis
New York Chicago San Francisco
Lisbon London Madrid Mexico City
Milan New Delhi San Juan
Seoul Singapore Sydney Toronto
Copyright © 2012 by The McGraw-Hill Companies. All rights reserved. Except as permitted under the
United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any
form or by any means, or stored in a database or retrieval system, without the prior written permission of
the publisher.
ISBN: 978-0-07-179054-3
MHID: 0-07-179054-3
The material in this eBook also appears in the print version of this title: ISBN: 978-0-07-179053-6,
MHID: 0-07-179053-5.
All trademarks are trademarks of their respective owners. Rather than put a trademark symbol after every
occurrence of a trademarked name, we use names in an editorial fashion only, and to the benefi t of the
trademark owner, with no intention of infringement of the trademark. Where such designations appear in
this book, they have been printed with initial caps.
McGraw-Hill eBooks are available at special quantity discounts to use as premiums and sales
promotions, or for use in corporate training programs. To contact a representative please e-mail us at
[email protected].
Information has been obtained by McGraw-Hill from sources believed to be reliable. However, because of
the possibility of human or mechanical error by our sources, McGraw-Hill, or others, McGraw-Hill does
not guarantee the accuracy, adequacy, or completeness of any information and is not responsible for any
errors or omissions or the results obtained from the use of such information.
TERMS OF USE
This is a copyrighted work and The McGraw-Hill Companies, Inc. (“McGrawHill”) and its licensors
reserve all rights in and to the work. Use of this work is subject to these terms. Except as permitted under
the Copyright Act of 1976 and the right to store and retrieve one copy of the work, you may not decompile,
disassemble, reverse engineer, reproduce, modify, create derivative works based upon, transmit, distribute,
disseminate, sell, publish or sublicense the work or any part of it without McGraw-Hill’s prior consent.
You may use the work for your own noncommercial and personal use; any other use of the work is strictly
prohibited. Your right to use the work may be terminated if you fail to comply with these terms.
THE WORK IS PROVIDED “AS IS.” McGRAW-HILL AND ITS LICENSORS MAKE NO
GUARANTEES OR WARRANTIES AS TO THE ACCURACY, ADEQUACY OR
COMPLETENESS OF OR RESULTS TO BE OBTAINED FROM USING THE WORK, INCLUDING ANY
INFORMATION THAT CAN BE ACCESSED THROUGH THE WORK VIA HYPERLINK OR
OTHERWISE, AND EXPRESSLY DISCLAIM ANY WARRANTY, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE. McGraw-Hill and its licensors do not warrant or
guarantee that the functions contained in the work will meet your requirements or that its operation will be
uninterrupted or error free. Neither McGraw-Hill nor its licensors shall be liable to you or anyone else for
any inaccuracy, error or omission, regardless of cause, in the work or for any damages resulting therefrom.
McGraw-Hill has no responsibility for the content of any information accessed through the work. Under
no circumstances shall McGraw-Hill and/or its licensors be liable for any indirect, incidental, special,
punitive, consequential or similar damages that result from the use of or inability to use the work, even if
any of them has been advised of the possibility of such damages. This limitation of liability shall apply to
any claim or cause whatsoever whether such claim or cause arises in contract, tort or otherwise.
My fifteenth book in my eighteenth year at IBM—it’s hard to believe so
much time has passed and Information Management technology has become
not just my career, but somewhat of a hobby (insert image of Chloe reading
this in a couple of years once she learns the universal “loser” gesture).
I often dedicate my books to people in my life: This book I actually want to
dedicate to the company in my life that turned 100 years old on August 12,
2011: IBM. In this day and age of fluid careers, the U.S. Department of
Labor has remarked that the average learner will have 10 to 14 jobs by the
time they are 38; 1 in 4 workers have been with their employer less than a
year; and 1 in 2 workers have been with their employer less than 5 years.
Sometimes I get asked about my 18-year tenure at IBM in a tone of disbelief
for my generation. In my 18 years at IBM, I’ve had the honor to learn and
participate in the latest technologies, marketing, sales, technical sales,
writing, usability design, development, partner programs, channels,
education, support, services, public speaking, competitive analysis, and
always learning more. IBM has always been a place that nurtures excel-
lence and opportunity for those thirsty to make a difference, and I’ve got a
thirst not yet quenched. IBM deeply encourages learning from others—and
I often wonder if other people feel like they won the lottery with a mentoring
team (Martin Wildberger, Bob Piciano, Dale Rebhorn, and Alyse Passarelli)
like the one I have. Thanks to IBM for providing an endless cup of opportu-
nity and learning experiences.
Finally, to my two gals, whose spirits always run through my soul:
Grace Madeleine Zikopoulos and Chloë Alyse Zikopoulos.
—Paul Zikopoulos
This is the fourth book that I have authored, and every time I dedicate the
book to my wife and family. Well this is no exception, because it’s their
support that makes this all possible, as anyone who has ever spent hours
and hours of their own personal time writing a book can attest to.
To my wife, Teresa, who is always supporting me 100 percent in all that I
do, including crazy ideas like writing a book. She knows full well how much
time it takes to write a book since she is a real author herself and yet she still
supported me when I told her I was going to write this book (you are a
saint). And to Riley and Sophia, who are now old enough to read one of my
books (not that they are really interested in any of this stuff since they
are both under ten). Daddy is finished writing his book so let’s go
outside and play.
—Chris Eaton
I’d like to thank Sandra, Erik, and Anna for supporting me, and giving me
the time to do this. Also, thanks to Paul for making this book happen and
asking me to contribute.
—Dirk deRoos
I would like to thank my ridiculously supportive wife and put in writing for
Lauren and William that yes, I will finally take them to Disneyland again
now that this is published. I’d also like to thank Anant Jhingran for both the
coaching and opportunities he has entrusted in me.
—Thomas Deutsch
“If you love what you do, you will never work a day in your life.” I dedicate
this book to all my colleagues at IBM that I worked with over the years who
helped me learn and grow and have made this saying come true for me.
—George Lapis
Thanks to my IBM colleagues in Big Data Research and Development for
the exciting technologies I get to work on every day. I also want to thank
Paul for the opportunity to contribute to this book. Last but not least, and
most importantly, for my wife, Amy, and my twins, Tiffany and Ronald,
thank you for everything you do, the joy you bring, and for supporting the
time it took to work on this book.
—Steven Sit
CONTENTS AT A GLANCE
PART I
Big Data: From the Business Perspective
1 What Is Big Data? Hint: You’re a Part of It Every Day ... 3
2 Why Is Big Data Important? ......................... 15
3 Why IBM for Big Data? ............................. 35
PART II
Big Data: From the Technology Perspective
4 All About Hadoop: The Big Data Lingo Chapter ....... 51
5 InfoSphere BigInsights: Analytics
for Big Data At Rest .............................. 81
6 IBM InfoSphere Streams: Analytics
for Big Data in Motion ............................ 123
ix