Table Of ContentLearning Storm
Copyright©2014PacktPublishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, without the prior written permission of the publisher,
exceptinthecaseofbrief quotationsembeddedincriticalarticlesorreviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers
and distributors will be held liable for any damages caused or alleged to be caused directly or
indirectlybythisbook.
Packt Publishing has endeavored to provide trademark information about all of the companies
and products mentioned in this book by the appropriate use of capitals. However, Packt
Publishing cannotguaranteetheaccuracyofthisinformation.
Firstpublished:August2014
Productionreference:1200814
PublishedbyPacktPublishing Ltd.
LiveryPlace
35LiveryStreet
BirminghamB32PB,UK.
ISBN978-1-78398-132-8
www.packtpub.com
Coverimage byPratyushMohanta(<[email protected]>)
2
Credits
Authors
AnkitJain
AnandNalya
Reviewers
VinothKannan
SonalRaj
DanijelSchiavuzzi
Commissioning Editor
UshaIyer
Acquisition Editor
LlewellynRozario
Content DevelopmentEditor
SankalpPawar
Technical Editors
MenzaMathew
SiddhiRane
CopyEditors
SarangChari
MradulaHegde
Project Coordinator
3
HarshalVed
Proofreaders
SimranBhogal
AmeeshaGreen
PaulHindle
Indexers
HemanginiBari
TejalSoni
PriyaSubramani
Graphics
AbhinashSahu
ProductionCoordinator
SaiprasadKadam
Cover Work
SaiprasadKadam
4
Table of Contents
AbouttheAuthors...............................................................................................................................10
AbouttheReviewers................................................................................................................................10
Preface.....................................................................................................................................................13
Whatthisbookcovers..............................................................................................................................13
Whatyouneedforthisbook...................................................................................................................14
Whothisbookisfor.................................................................................................................................14
Conventions...............................................................................................................................................14
Readerfeedback........................................................................................................................................15
Customersupport.....................................................................................................................................16
Chapter1. Setting Up Stormon a Single Machine.................................................................18
FeaturesofStorm.....................................................................................................................................19
Stormcomponents..................................................................................................................................20
Nimbus................................................................................................................................................20
Supervisornodes...............................................................................................................................20
TheZooKeeper cluster......................................................................................................................20
TheStormdatamodel..............................................................................................................................21
DefinitionofaStormtopology........................................................................................................22
Operation modes................................................................................................................................25
Settingupyourdevelopmentenvironment...................................................................................25
Developingasampletopology.........................................................................................................29
SettingupZooKeeper........................................................................................................................36
SettingupStormonasingledevelopmentmachine....................................................................37
Deployingthesampletopologyonasingle-nodecluster............................................................38
Summary...................................................................................................................................................42
Chapter2. SettingUp a StormCluster.......................................................................................43
SettingupadistributedStormcluster..................................................................................................47
DeployingatopologyonaremoteStormcluster................................................................................49
5
Deployingthesampletopologyontheremotecluster.................................................................50
Configuringtheparallelism ofatopology............................................................................................52
Theworkerprocess............................................................................................................................52
Theexecutor.......................................................................................................................................52
Tasks....................................................................................................................................................52
Configuringparallelism atthecodelevel.......................................................................................53
Distributingworkerprocesses,executors,andtasksin thesampletopology...........................54
Rebalancingtheparallelism ofatopology............................................................................................55
Rebalancingtheparallelism ofthesampletopology....................................................................56
Streamgrouping.......................................................................................................................................58
Shufflegrouping................................................................................................................................58
Fieldsgrouping..................................................................................................................................58
Allgrouping........................................................................................................................................59
Globalgrouping.................................................................................................................................60
Directgrouping...................................................................................................................................61
Localorshufflegrouping..................................................................................................................61
Customgrouping...............................................................................................................................62
Guaranteedmessageprocessing............................................................................................................63
Summary...................................................................................................................................................66
Chapter3. Monitoring theStorm Cluster.................................................................................67
StartingtousetheStormUI...................................................................................................................67
MonitoringatopologyusingtheStormUI..........................................................................................68
Clusterstatisticsusing theNimbusthriftclient...................................................................................75
FetchinginformationwiththeNimbusthriftclient.....................................................................75
Summary...................................................................................................................................................87
Chapter4. Storm andKafka Integration..................................................................................88
TheKafkaarchitecture............................................................................................................................88
Theproducer......................................................................................................................................89
Replication.........................................................................................................................................90
Consumers..........................................................................................................................................90
Brokers.................................................................................................................................................91
6
Dataretention.....................................................................................................................................91
SettingupKafka.......................................................................................................................................92
Settingupasingle-nodeKafkacluster...........................................................................................92
Settingupathree-nodeKafkacluster............................................................................................95
AsampleKafkaproducer........................................................................................................................97
IntegratingKafkawithStorm...............................................................................................................101
Summary.................................................................................................................................................108
Chapter5. Exploring High-level Abstraction in Stormwith Trident..........................109
IntroducingTrident...............................................................................................................................109
UnderstandingTrident'sdatamodel...................................................................................................110
WritingTrident functions,filters, andprojections...........................................................................110
Tridentfunctions..............................................................................................................................110
Tridentfilters.....................................................................................................................................111
Tridentprojections...........................................................................................................................113
Tridentrepartitioning operations........................................................................................................114
Theshuffleoperation.......................................................................................................................114
ThepartitionByoperation...............................................................................................................114
Theglobaloperation........................................................................................................................116
Thebroadcastoperation..................................................................................................................116
ThebatchGlobaloperation..............................................................................................................117
Thepartitionoperation...................................................................................................................118
Tridentaggregators................................................................................................................................119
Thepartitionaggregate...................................................................................................................119
Theaggregate...................................................................................................................................120
Thepersistentaggregate.................................................................................................................123
Aggregatorchaining........................................................................................................................124
Utilizing thegroupByoperation...........................................................................................................125
Anon-transactional topology...............................................................................................................125
AsampleTrident topology....................................................................................................................129
MaintainingthetopologystatewithTrident.....................................................................................133
Atransactional topology.......................................................................................................................134
7
Theopaquetransactionaltopology.....................................................................................................136
DistributedRPC......................................................................................................................................137
WhentouseTrident...............................................................................................................................141
Summary..................................................................................................................................................141
Chapter6. IntegrationofStorm with Batch Processing Tools.......................................142
ExploringApacheHadoop....................................................................................................................142
UnderstandingHDFS......................................................................................................................143
UnderstandingYARN......................................................................................................................145
InstallingApacheHadoop....................................................................................................................146
Settinguppassword-lessSSH........................................................................................................147
GettingtheHadoopbundleandsettingupenvironment variables.........................................148
SettingupHDFS..............................................................................................................................149
SettingupYARN..............................................................................................................................153
IntegrationofStormwithHadoop......................................................................................................156
SettingupStorm-YARN..................................................................................................................156
DeployingStorm-StartertopologiesonStorm-YARN......................................................................161
Summary.................................................................................................................................................163
Chapter7. Integrating Stormwith JMX, Ganglia, HBase, andRedis.........................164
MonitoringtheStormclusterusingJMX...........................................................................................164
MonitoringtheStormclusterusingGanglia......................................................................................167
IntegratingStormwithHBase..............................................................................................................177
IntegratingStormwithRedis...............................................................................................................188
Summary.................................................................................................................................................194
Chapter8. Log Processing with Storm.....................................................................................195
Serverlog-processingelements............................................................................................................195
ProducingtheApachelogin Kafka.....................................................................................................196
Splittingtheserver logline..................................................................................................................200
Identifyingthecountry,theoperating system type,andthebrowsertypefromthelogfile......204
Extracting thesearchedkeyword........................................................................................................207
Persistingtheprocess data....................................................................................................................211
Definingatopologyand theKafkaspout............................................................................................217
8
Deployingatopology............................................................................................................................220
MySQLqueries.......................................................................................................................................221
Calculatingthepage hitsfromeachcountry...............................................................................222
Calculatingthecountforeachbrowser........................................................................................223
Calculatingthecountforeachoperating system........................................................................223
Summary.................................................................................................................................................224
Chapter9. MachineLearning......................................................................................................225
Exploringmachinelearning.................................................................................................................225
UsingTrident-ML..................................................................................................................................226
Theusecase–clusteringsyntheticcontroldata...............................................................................227
Producingatraining datasetintoKafka............................................................................................228
BuildingaTrident topologytobuildtheclusteringmodel..............................................................232
Summary................................................................................................................................................238
Index......................................................................................................................................................240
9
About the Authors
Ankit Jain holds a Bachelor's degree in Computer Science Engineering. He has 4 years of
experience in designing and architecting solutions for the Big Data domain and has been
involved with several complex engagements. His technical strengths include Hadoop, Storm, S4,
HBase, Hive, Sqoop, Flume, ElasticSearch, Machine Learning, Kafka, Spring, Java, and J2EE.
HeiscurrentlyemployedwithImpetusInfotechPvt.Ltd.
He also shares his thoughts on his personal blog at http://ankitasblogger.blogspot.in/. You can
follow him on Twitter at @mynameisanky. He spends most of his time reading books and
playing with different technologies. When not at work, he spends time with his family and
friendswatching moviesandplaying games.
I would like to thank my family and colleagues for always being there for me. Special thanks to
thePacktPublishing team;withoutyouguys,thisworkwouldnothavebeen possible.
Anand Nalya is a full stack engineer with over 8 years of extensive experience in designing,
developing, deploying, and benchmarking Big Data and web-scale applications for both
start-ups and enterprises. He focuses on reducing the complexity in getting things done with
brevityincode.
He blogs about Big Data, web applications, and technology in general at
http://anandnalya.com/. You can also follow him on Twitter at @anandnalya. When not
workingonprojects, hecanbefoundstargazing orreading.
I would like to thank my wife, Nidhi, for putting up with so many of my side projects and my
family members who are always there for me. Special thanks to my colleagues who helped me
validate the writing, and finally, the reviewers and editors at Packt Publishing, without whom
thisworkwouldnothavebeen possible.
About the Reviewers
Vinoth Kannan is a solution architect at WidasConcepts, Germany, that focuses on creating
robust, highly scalable, real-time systems for storage, search, and analytics. He now works in
GermanyafterhisprofessionalstintsinFrance,Italy,andIndia.
10
