Apache Oozie: The Workflow Scheduler for Hadoop

456 Pages·2015·7.25 MB·English

by  Mohammad Kamrul Islam

Apache Oozie
Mohammad Kamrul Islam & Aravind Srinivasan
Apache Oozie
by Mohammad Kamrul Islam and Aravind Srinivasan Copyright © 2015
Mohammad Islam and Aravindakshan Srinivasan. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North,
Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional
use. Online editions are also available for most titles
(http://safaribooksonline.com). For more information, contact our
corporate/institutional sales department: 800-998-9938 or
[email protected].
Editors: Mike Loukides and Marie Beaugureau
Production Editor: Colleen Lobner
Copyeditor: Gillian McGarvey
Proofreader: Jasmine Kwityn
Indexer: Lucie Haskins
Interior Designer: David Futato
Cover Designer: Ellie Volckhausen
Illustrator: Rebecca Demarest
May 2015: First Edition
Revision History for the First Edition
2015-05-08: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781449369927 for release
details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Apache
Oozie, the cover image of a binturong, and related trade dress are trademarks of
O’Reilly Media, Inc.
While the publisher and the authors have used good faith efforts to ensure that
the information and instructions contained in this work are accurate, the
publisher and the authors disclaim all responsibility for errors or omissions,
including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in
this work is at your own risk. If any code samples or other technology this work
contains or describes is subject to open source licenses or the intellectual
property rights of others, it is your responsibility to ensure that your use thereof
complies with such licenses and/or rights.
978-1-44936992-7
[LSI]
Foreword
First developed when I was at Yahoo! in 2008, Apache Oozie remains the most
sophisticated and powerful workflow scheduler for managing Apache Hadoop
jobs. Although simpler open source alternatives have been introduced, Oozie is
still my recommended workflow scheduler due to its ability to handle
complexity, ease of integration with established and emerging Hadoop
components (like Spark), and the growing ecosystem of projects, such as Apache
Falcon, that rely on its workflow engine.
That said, Oozie also remains one of the more challenging schedulers to learn
and master. If ever a system required a comprehensive user’s manual, Oozie is it.
To take advantage of the full power that Oozie has to offer, developers need the
guidance and advice of expert users. That is why I am delighted to see this book
get published.
When Oozie was first developed, I was Chief Architect of Yahoo!’s Search and
Advertising Technology Group. At the time, our group was starting to migrate
the event-processing pipelines of our advertising products from a proprietary
technology stack to Apache Hadoop.
The advertising pipelines at Yahoo! were extremely complex. Data was
processed in batches that ranged from 5 minutes to 30 days in length, with
aggregates “graduating” in complex ways from one time scale to another. In
addition, these pipelines needed to detect and gracefully handle late data,
missing data, software bugs tickled by “black swan” event data, and software
bugs introduced by recent software pushes. On top of all of that, billions of
dollars of revenue — and a good deal of the company’s growth prospects —
depended on these pipelines, raising the stakes for data quality, security, and
compliance. We had about a half-dozen workflow systems in use back then, and
there was a lot of internal competition to be selected as the standard for Hadoop.
Ultimately, the design for Oozie came from ideas from two systems: PacMan, a
system already integrated with Hadoop, and Lexus, a system already in place for
the advertising pipelines.
Oozie’s origins as a second-generation system designed to meet the needs of
extremely complicated applications are both a strength and a weakness. On the
positive side, there is no use case or scenario that Oozie can’t handle — and if
you know what you’re doing, handle well. On the negative side, Oozie suffers
from the over-engineering that you’d expect from second-system effect. It has
complex features that are great for handling complicated applications, but can be
very nonintuitive for inexperienced users. For these newer users, I want to let
you know that Oozie is worth the investment of your time. While the newer,
simpler workflow schedulers are much easier for simple pipelines, it is in the
nature of data pipelines to grow more sophisticated over time. The simpler
solutions will ultimately limit the solutions that you can create. Don’t limit
yourself.
As guides to Oozie, there can be no better experts than Aravind Srinivasan and
Mohammad Kamrul Islam. Aravind represents the “voice of the user,” as he was
one of the engineers who moved Yahoo!’s advertising pipelines over to Oozie,
bringing the lessons of Lexus to the Oozie developers. Subsequently, he has
worked on many other Oozie applications, both inside and outside of Yahoo!.
Mohammad represents the “voice of the developer,” as a core contributor to
Oozie since its 1.x days. Mohammad is currently Vice President of the Oozie
project at the Apache Software Foundation, and he also makes significant
contributions to other Hadoop-related projects such as YARN and Tez.
In this book, the authors have striven for practicality, focusing on the concepts,
principles, tips, and tricks necessary for developers to get the most out of Oozie.
A volume such as this is long overdue. Developers will get a lot more out the
Hadoop ecosystem by reading it.
Raymie Stata, CEO, Altiscale
Preface
Hadoop is fast becoming the de facto big data platform across all industries. An
entire ecosystem of tools, products, and services targeting every functionality
and requirement have sprung up around Hadoop. Apache Oozie occupies an
important space in this ever-expanding ecosystem. Since Hadoop’s early days at
Yahoo!, it has been a natural platform for Extract, Transform, and Load (ETL)
and other forms of data pipelines. Without a mature workflow management and
scheduling system, implementing such pipelines can be a challenge. Oozie
satisfies these requirements and provides a viable tool to implement complex,
real-world data pipelines. In this book, we have tried our best to introduce
readers to all the facets of Oozie and walk them through the intricacies of this
rather powerful and flexible platform.
Software workflow systems are ubiquitous and each system has its own
idiosyncrasies. But Oozie is a lot more than just another workflow system. One
of Oozie’s strengths is that it was custom built from the ground up for Hadoop.
This not only means that Oozie works well on Hadoop, but that the authors of
Oozie had an opportunity to build a new system incorporating much of their
knowledge about other legacy workflow systems. Although some users view
Oozie as just a workflow system, it has evolved into something more than that.
The ability to use data availability and time-based triggers to schedule
workflows via the Oozie coordinator is as important to today’s users as the
workflow. The higher-level concept of bundles, which enable users to package
multiple coordinators into complex data pipelines, is also gaining a lot of
traction as applications and pipelines moving to Hadoop are getting more
complicated.
We are both very lucky to have been involved in Oozie’s journey from its early
days. We have played several roles in its evolution, ranging from developer,
architect, open source committer, Project Management Committee (PMC)
member, product manager, and even demanding customer. We have tried to
leverage all of that perspective to present a comprehensive view of Oozie in this
book. We strongly believe in the vision of Oozie and its potential to make
Hadoop a more powerful platform. Hadoop’s use is expanding and we notice
that users want to use it in smarter and more interesting ways. We have seen
many projects in the past getting bogged down with writing, operating, and
debugging the workflow system meant to manage the business application. By
delegating all of the workflow and scheduling complexities to Oozie, you can
focus on developing your core business application.
This book attempts to explain all the technical details of Oozie and its various
features with specific, real-world examples. The target audience for this book is
Oozie users and administrators at all levels of expertise. Our only requirement
for the reader is a working knowledge of Hadoop and the ecosystem tools. We
are also very aware of the challenges of operating a Hadoop cluster in general
and Oozie in particular, and have tried our best to cover operational issues and
debugging techniques in depth. Last but not the least, Oozie is designed to be
very flexible and extensible and we want to encourage users to get comfortable
with the idea of becoming an Oozie developer if they so desire. We would love
to grow the Oozie community and continue the innovation in this part of the
Hadoop ecosystem. While it would be nice to achieve all of these goals with this
book, the most fundamental hope is that readers find it helpful in using Oozie
and Hadoop more effectively every day in their jobs.
Contents of This Book
We start the book off with a brief introduction to Oozie in Chapter 1 and an
overview of the important concepts in Chapter 2. Chapter 3 gets your hands dirty
right away with detailed instructions on installing and configuring Oozie. We
want this book to be a hands-on experience for our readers, so deployment must
be mastered early.
Oozie is primarily a workflow system in most users’ worlds. Chapters 4 and 5
take you on an in-depth journey through the world of writing and configuring
workflows. These chapters also explain parameterization and variable
substitution in detail. This will establish a very good basis for the rest of the
book, as the other major Oozie features are built on top of the workflow system.
Chapter 6 covers the concepts of the coordinator and helps you to start writing
coordinator apps. We then look at the data dependency mechanism in Chapter 7.
Data triggers are a powerful and distinguishing feature of Oozie and this chapter
explains all the intricacies of managing data dependencies.
Bundles are the higher-level pipeline abstraction and Chapter 8 delves deep into
the world of bundles with specific examples and use cases to clarify some of the
advanced concepts. It also introduces concepts and challenges like reprocessing,
which production pipelines routinely deal with.
In Chapter 9, we cover the powerful security features in Oozie, including
Kerberos support and impersonation. This chapter also explains the management
of shared libraries in Oozie and cron-based scheduling, which comes in handy
for a certain class of use cases.
We cover the developer aspects regarding extending Oozie in Chapter 10.
Readers can learn how to implement custom extensions to their Oozie systems.
It teaches them how to write their own Expression Language (EL) functions and
custom actions.
Last, but not the least, we realize that debugging Oozie workflows and managing
the operational details of Oozie are an important part of mastering Oozie. Thus,
Chapter 11 focuses exclusively on these topics. We start by explaining the
command-line interface (CLI) tool and the REST API and then discuss
monitoring and debugging. We also cover the purge service, reprocessing, and
other operational aspects in this chapter.