Apache Flume: Distributed Log Collection for Hadoop

154 Pages·2013·1.02 MB·English

by  Steve Hoffman

Table of Contents
Apache Flume: Distributed Log Collection for Hadoop
Credits
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers and more
Why Subscribe?
Free Access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Errata
Piracy
Questions
1. Overview and Architecture
Flume 0.9
Flume 1.X (Flume-NG)
The problem with HDFS and streaming data/logs
Sources, channels, and sinks
Flume events
Interceptors, channel selectors, and sink processors
Tiered data collection (multiple flows and/or agents)
Summary
2. Flume Quick Start
Downloading Flume
Flume in Hadoop distributions
Flume configuration file overview
Starting up with "Hello World"
Summary
3. Channels
Memory channel
File channel
Summary
4. Sinks and Sink Processors
HDFS sink
Path and filename
File rotation
Compression codecs
Event serializers
Text output
Text with headers
Apache Avro
File type
Sequence file
Data stream
Compressed stream
Timeouts and workers
Sink groups
Load balancing
Failover
Summary
5. Sources and Channel Selectors
The problem with using tail
The exec source
The spooling directory source
Syslog sources
The syslog UDP source
The syslog TCP source
The multiport syslog TCP source
Channel selectors
Replicating
Multiplexing
Summary
6. Interceptors, ETL, and Routing
Interceptors
Timestamp
Host
Static
Regular expression filtering
Regular expression extractor
Custom interceptors
Tiering data flows
Avro Source/Sink
Command-line Avro
Log4J Appender
The Load Balancing Log4J Appender
Routing
Summary
7. Monitoring Flume
Monitoring the agent process
Monit
Nagios
Monitoring performance metrics
Ganglia
The internal HTTP server
Custom monitoring hooks
Summary
8. There Is No Spoon – The Realities of Real-time Distributed Data
Collection
Transport time versus log time
Time zones are evil
Capacity planning
Considerations for multiple data centers
Compliance and data expiry
Summary
Index
Apache Flume: Distributed Log
Collection for Hadoop
Apache Flume: Distributed Log
Collection for Hadoop
Copyright © 2013 Packt Publishing All rights reserved. No part of this
book may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical
articles or reviews.
Every effort has been made in the preparation of this book to ensure the
accuracy of the information presented. However, the information
contained in this book is sold without warranty, either express or implied.
Neither the author, nor Packt Publishing, and its dealers and distributors
will be held liable for any damages caused or alleged to be caused
directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about
all of the companies and products mentioned in this book by the
appropriate use of capitals. However, Packt Publishing cannot guarantee
the accuracy of this information.
First published: July 2013
Production Reference: 1090713
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78216-791-4
www.packtpub.com
Cover Image by Abhishek Pandey (<[e-mail]>)
Credits
Author
Steve Hoffman
Reviewers
Subash D'Souza
Stefan Will
Acquisition Editor
Kunal Parikh
Commissioning Editor
Sharvari Tawde
Technical Editors
Jalasha D'costa
Mausam Kothari
Project Coordinator
Sherin Padayatty
Proofreader
Aaron Nash
Indexer
Monica Ajmera Mehta
Graphics
Valentina D'silva
Abhinash Sahu
Production Coordinator
Kirtee Shingan
Cover Work
Kirtee Shingan
About the Author
Steve Hoffman has 30 years of software development experience and
holds a B.S. in computer engineering from the University of Illinois
Urbana-Champaign and a M.S. in computer science from the DePaul
University. He is currently a Principal Engineer at Orbitz Worldwide.
More information on Steve can be found at http://bit.ly/bacoboy or on
Twitter @bacoboy.
This is Steve's first book.
I'd like to dedicate this book to my loving wife Tracy. Her dedication to
perusing what you love is unmatched and it inspires me to follow her
excellent lead in all things.
I'd also like to thank Packt Publishing for the opportunity to write this
book and my reviewers and editors for their hard work in making it a
reality.
Finally, I want to wish a fond farewell to my brother Richard who
passed away recently. No book has enough pages to describe in
detail just how much we will all miss him. Good travels brother.
About the Reviewers
Subash D'Souza is a professional software developer with strong
expertise in crunching big data using Hadoop/HBase with Hive/Pig. He
has worked with Perl/PHP/Python, primarily for coding and
MySQL/Oracle as the backend, for several years prior to moving into
Hadoop fulltime. He has worked on scaling for load, code development,
and optimization for speed. He also has experience optimizing SQL
queries for database interactions. His specialties include Hadoop, HBase,
Hive, Pig, Sqoop, Flume, Oozie, Scaling, Web Data Mining, PHP, Perl,
Python, Oracle, SQL Server, and MySQL Replication/Clustering.
I would like to thank my wife, Theresa for her kind words of support
and encouragement.
Stefan Will is a computer scientist with a degree in machine learning and
pattern recognition from the University of Bonn. For over a decade has
worked for several startup companies in Silicon Valley and Raleigh, North
Carolina, in the area of search and analytics. Presently, he leads the
development of the search backend and the Hadoop-based product
analytics platform at Zendesk, the customer service software provider.