Who's afraid of the big (data) bad wolf?

20 Pages·2013·2.21 MB·English

IBM Software October 2013
Thought Leadership White Paper
Who's afraid of the big (data)
bad wolf?
Survive the big data storm by getting ahead of integration
and governance functional requirements
2 Who's afraid of the big (data) bad wolf?
Once upon a time… Build your house of straw or sticks,
Once upon a time, and not so long ago, data grew slowly and it blows away
over time, and it grew in a linear manner. Today, data volumes Today, performing big data analytics is strategic, and being
are exploding in every facet of our lives. Business leaders are able to supplement (or augment) data warehouses with this
eager to harness the power of big data, but before setting out key information is essential. Apache Hadoop technology is an
into the big data world, it is important to understand that as integral part of this process. Apache Hadoop is an open source
opportunities increase, ensuring that source information is software project that enables the distributed processing of large
trustworthy and protected becomes exponentially more difficult. data sets across clusters of commodity servers. It is designed to
If this trustworthiness issue is not addressed directly, end users scale up from a single server to thousands of machines, with a
may lose confidence in the insights generated from their data, very high degree of fault tolerance. Rather than relying on
which can result in a failure to act on opportunities or against high-e nd hardware, the resiliency of these clusters comes
threats. from the software’s ability to detect and handle failures at the
application layer. Hadoop clearly changes the economics and
To make the most of big data, you have to start with data you the dynamics of large-s cale computing.
trust. But the sheer volume and complexity of big data means
that traditional, manual methods of discovering, governing While Hadoop and Hadoop-b ased solutions have their advan-
and correcting information are no longer feasible. Information tages when it comes to addressing big data volumes, Hadoop
integration and governance must be implemented to support is not designed for data integration. Data integration carries
big data applications, data warehouses and data warehouse its own unique requirements (such as supporting governance,
augmentation initiatives, providing appropriate governance and metadata management, data quality and flexible data delivery
rapid integration from the very start. (Once you get behind, styles) for success. In a recent paper, Gartner states “As use of
it’s like trying to catch a wolf by the tail.) the Hadoop stack continues to grow, organizations are asking
if it is a suitable solution for data integration. Today, the answer
By automating information integration and governance and is no. Not only are many key data integration capabilities
deploying it at the point of data creation, organizations can immature or missing from the stack, but many have not been
boost big data confidence. A solid integration and governance addressed in current projects.”1
program must include automated discovery, profiling and
understanding of diverse data sets to provide context and Using Hadoop to build out a data integration solution is like
enable employees to make informed decisions. It must also be building a house of straw. You might think you have the basic
agile to accommodate a wide variety of data and seamlessly shape right, but once you’ve factored in the costs to make the
integrate with diverse technologies, from data marts to Apache
Hadoop systems.
IBM Software 3
house functional and sound by developing all the necessary
data integration functionality, you might as well let this house
The limits of Hadoop data integration
blow away in the wind.
Hadoop has quickly become the tool of choice for enterprise
Be a smart architect: Insure against big data projects—but succeeding with Hadoop requires an
understanding of what it does well and what it doesn’t.
unpredictable winds
A house built of bricks (or, in this case, a solid and robust
Hadoop is essentially a file system. It is a way to store big
data integration platform) will stand up to all the elements—
data so that it can be analyzed. The performance issues
including the storm of increasing data volumes, velocity and
associated with using Hadoop for data processing are
variety created by the big data phenomenon. well known:
Big data streams in at high velocity—so performance is key. • Hadoop is written in Java, which is slower than frameworks
Data changes rapidly, and it must be fed to various applications in C or C++.
in the system quickly so business leaders can react to changing • MapReduce/ Hadoop File System lands data between
reduce steps, which is a huge performance constraint.
market conditions as soon as possible. To successfully handle
• Hadoop File System centrally manages an index necessary
big data, you need an enterprise-c lass data integration solution
to map tasks to the data distributed throughout the nodes—
that is:
a documented bottleneck.
• Operations requiring data collocation to compute the result
●● Dynamic to meet your current and future performance (joins, aggregations, sorts, deduplications and so on) will
requirements
run inefficiently when the data distribution is different from
●● Extendable and partitioned for fast and easy scalability the index.
●● Integrated with Hadoop (Hadoop itself is not an integration • Hadoop is not a good choice when real-t ime, low- latency
platform, but it can be leveraged as part of an integration processing is required because there is no “real-t ime”
architecture to land and determine the value of data) version of Hadoop available.
●● Part of a rich set of transformation and data quality • Job start-u p is slow, which can be a big performance
components available out of the box to help accelerate penalty, particularly for small, bursty jobs.
your team’s development cycle
Data integration solutions such as IBM InfoSphere Information
Server offer better performance for integration workloads.
A foundation for high-p erforming technology
And InfoSphere Information Server provides full connectivity
The most critical requirement for processing large, enterprise-
to Hadoop, so customers can maintain their data using
class data volumes for big data integration is massive data
Hadoop while taking full advantage of the unlimited data
scalability (MDS). MDS provides the ability to process vast
scalability offered by InfoSphere Information Server. This
quantities of data in parallel, dramatically reducing the amount
offers customers the best of both worlds: inexpensive data
of time it takes to handle various workloads. Unlike other
storage with Hadoop along with unlimited data scalability,
processing models, MDS systems optimize the usage of and therefore, less expensive data processing.
hardware resources, allowing the maximum amount of data
to be processed per node (see Figure 1).
4 Who's afraid of the big (data) bad wolf?
Shared nothing architected software is designed from the
Data scalability across hardware architectures ground up to exploit a shared nothing, massively parallel
processing (MPP) architecture by partitioning data sets across
xy
MPP/Grid computing nodes and executing a single application with the
(Hundreds of TB) 64 way
same application logic executing against each data partition
(see Figure 2). This means there is no single point of contention,
or processing bottleneck, anywhere in the system. Therefore,
16 way
SMP/MPP
there is no upper limitation on data volume, processing
(Hundreds of GB)
throughput, or number of processors and nodes.
8 way
SMP
(GBs)
2 way
MPP
1x grid clusters –
One core Hundreds of cores hundreds of
nodes
• Same functionality
• All architectures
• Linear speedup
• Decision (at runtime) to scale “N way”
Shared disk
Figure 1. The essential characteristics needed to support MDS requirements
Design CPU CPU CPU CPU
enable processing of unlimited data volumes.
once SMP
Shared memory
MDS is important because processing unlimited data volumes
makes it possible to solve many high-v alue business problems
for the first time while ensuring that a hardware platform will
Uniprocessor CPU
yield predictable benefits.
Memory
How do you know if your house is strong enough to hold back
the winds of big data? It’s all in the architecture. MDS systems Figure 2. An example of a shared nothing architecture.
have four key things in common:
Software dataflow allows full exploitation of shared nothing
1. Feature a shared nothing architecture
by making it easy to implement and execute data pipelining and
2. Are implemented using software dataflow
data partitioning within a node and across nodes (see Figure 3).
3. Leverage data partitioning for linear data scalability
Software dataflow also hides the complexities of building and
4. Use a design isolation environment
tuning parallel applications from users.
IBM Software 5
Parallel dataflow with auto repartitioning
Customer Customer Credit card
last name postcode number
U-Z
N-T
G-M
A-F
Repartitioning Repartitioning
Source Target
Figure 3. Software dataflow architecture.
Software dataflow is the best architecture for exploiting Data partitioning means that large data sets are partitioned
multi-c ore processors within an SMP server (application across separate nodes and a single job executes the same
scale-u p) and for scaling out to multiple machines (application application logic against all partitioned data (see Figure 4).
scale-o ut). The architecture: Other approaches, such as task partitioning, cannot deliver
linear data scalability as data volumes grow because the amount
●● Supports pipelined and partitioned parallelism within and of data that can be sorted, merged and aggregated is limited to
across SMP nodes what can be processed on one node.
●● Provides a single mechanism for parallelization across all
hardware architectures, helping to eliminate complexity
●● Reduces the complexities of building, tuning and executing
parallel applications
●● Has no upper limit on data volumes, processing throughput
and numbers of processing nodes
6 Who's afraid of the big (data) bad wolf?
300 million records Dataflow graph Configuration file
One InfoSphere DataStage job processing 300 million customer
records in parallel across six 4-way nodes (24 processors)
Data partition one
Node one
Data partition two
Node two
Data partition three
Node three
Data partition four
Node four
Data partition five
Node five
Data partition six
Node six
Figure 4. Data partitioning architecture.
IBM Software 7
Characteristics of systems with data partitioning include: Finally, design isolation means a developer can design a
data processing job once, and use it in any hardware
●● Distributes data partitions across nodes configuration without needing to redesign and retune
●● Executes one job in parallel across nodes the job (see Figure 5).
●● Enables pipelining and repartitioning between stages and
between nodes without landing to disk
●● Exploits low-c ost grid hardware for big data
Parallel runtime execution
Application assembly: One dataflow graph created with the InfoSphere DataStage GUI
Application execution: Sequential or parallel
Sequential 4-way parallel 128-way parallel
Hardware platform
Uni-processor SMP
128 processor MPP
Figure 5. Design isolation architecture.
8 Who's afraid of the big (data) bad wolf?
Characteristics and benefits include: An integrated, shared metadata environment is another signifi-
cant time-s aver. It needs to be quick and easy for developers to
●● Build once and run without modification anywhere track job progress and to diagnose any problems. Additionally,
●● Reduce complexity with one unified mechanism for visualizing data through a dashboard will save countless hours
parallelizing by giving developers and IT staff a unified picture of what’s
●● Achieve a clean separation between the development of a going on with their data.
job and the expression of parallelization at runtime
●● Eliminate the need for performance tuning every time you As big data stores continue to grow, these high-p erformance
change hardware architecture and time-s aving features become even more important. For
●● Add hardware with no data scalability upper limit IT departments, they can mean the difference between meeting
service-l evel agreements (SLAs) or not, between having time
All of the most scalable platforms (IBM® Netezza®, to work on innovative new projects or being caught in the
IBM PureData™ System, IBM DB2® Database Partitioning drudgery of managing existing systems. And for the business,
Feature, Teradata, Hadoop and IBM InfoSphere® Information it can mean quicker, more-i nformed decision making, which
Server) have been built from the ground up to support these can lead to stronger profits, better service for customers and
four characteristics and can seamlessly exploit MPP and competitive advantage.
commodity grid architectures.
Build your house of bricks to ensure
Enhancing staff efficiency
confidence
Exploiting more efficient big data integration technology helps
For the purpose of big data integration, what does it mean to
employees make better use of their time. The development
“build your house of bricks?” It means ensuring you are being as
environment should be designed to improve efficiency by
efficient and cost-e ffective as possible, while also ensuring that
providing a single design palette in the shared application
you can get the data you need to where you need it, as you need
environment. Developers should never have to flip through a
it (that is, supporting flexible data delivery styles).
lot of different interfaces; everything they need should be easily
accessible.
IBM Software 9
Efficiency and reduced costs underlying information integration infrastructure or real-t ime
Easy self-s ervice data integration
workloads based on data replication capabilities. Ultimately, this
Wherever information resides, you need to integrate it type of solution should support processing optimization, help
efficiently, quickly and flexibly, easily managing information reduce the time and processing required from source to target,
provisioning to or from data warehouses, for augmenting data reduce network traffic and make data available when a full
warehouse projects, or for integrating big data. Self-s ervice extract is not possible. A high degree of automation, an easy-t o-
information provisioning, which may also be thought of as use, intuitive interface, and broad connectivity for data sources
self-s ervice data integration, is often a recurring drain on and targets will help you realize greatest time-t o-v alue when
the enterprise’s IT staff, requiring technically skilled extract, leveraging self-s ervice data integration.
transform and load (ETL) engineers.
Work smarter, not harder—and control costs
In today’s fast- moving business environment, organizations Employee time is a valuable and costly resource. An integration
must empower nontechnical, line-o f-b usiness users so that solution for big data that supports employee productivity and
they may easily retrieve data and populate new systems (such efficiency helps to improve the enterprise’s bottom line, elimi-
as business intelligence and analytics systems), freeing up nate bottlenecks and enhance agility.
expert technical resources to focus on more strategic activities
that deliver higher returns for the business. Additionally, For IT departments, SLAs are often impacted by inefficiencies.
nontechnical users need to be able to do this simply and on As data volume, variety and velocity grow, the time required to
demand across large quantities of data (for example, moving process data integration jobs frequently exceeds the window
thousands of tables, loading a data mart and so on). Design allowed by SLAs. That means IT is no longer meeting the needs
and operational metadata also need to be supported as part of internal customers.
of this process so that the information may be governed and
remains consistent. To improve productivity, it’s important to create design logic for
Hadoop-o riented data integration efforts using the same inter-
Organizations need a solution that specifically supports self- face, concepts and logic constructs as for any other deployment
service data integration needs. Such a solution needs to be able method. This eliminates the investment in extra time and
to intelligently select the right data integration mechanism for resources from learning new coding languages as they evolve, or
moving information—such as batch workloads that leverage the falling back on older methods of performing hand-c oding for
data integration work (see Figure 6).
10 Who's afraid of the big (data) bad wolf?
Working harder Working smarter
Slow analysis speeds require longer Single interface for integration tasks
processing windows and more downtime
Support for multiple data sources
Work in multiple and streaming data
interfaces
Working with different Predetermined, confirmed set of
code languages concepts and logic constructs
Automated processes and elements
limit hand-coding and replication
Start: Start: End:
End: Data gathered and passed directly
Data Data Integrated
Integrated to real-time analytics processes
sources Multiple data sources data
data
handoffs between
Staffing bottlenecks
systems
due to manual processes
Figure 6. By streamlining processes with automation, pre-set templates and other strategies, organizations can significantly enhance their efficiency.
Support a variety of big data sources and types To achieve the greatest efficiency, your information integration
Organizations exploring big data analytics and using technolo- platform needs to effectively handle the wide—and growing—
gies such as Hadoop for data at rest, or streaming technology complexity of heterogeneous enterprise information sources
for data in motion, face many of the same challenges as in other and types with a common, seamless architecture. Supporting
analytical environments, including: new and emerging big data source types is essential and must
include everything from Hadoop Distributed File System
●● Determining the location of the information sources needed (HDFS) for massively scalable and resilient storage, to
for analysis “Not only SQL” (NoSQL) for record storage optimized for
●● Assessing how that information can be moved into the read or write, to InfoSphere Streams for supporting massive-
analytical environment scale, real-t ime analytics. As new applications begin leveraging
●● Deciding how information must be reformatted to make it these technologies, you also need to ensure your information
easier and more efficient to explore integration platform supports those systems and data types as
●● Determining which data should be persisted to quickly get to well (such as Apache HBase, Java Message Service [JMS],
the next level of analysis Mongo DB, Hive, Cassandra, JSON and so on).
You don’t want to get stuck trying to do any of this manually.