Table Of ContentReal-time event systems with Kafka and Kinesis
Alexander Dean
Valentin Crettaz
M A N N I N G
Event Streams in Action
Event Streams
in Action
REAL-TIME EVENT SYSTEMS
WITH KAFKA AND KINESIS
ALEXANDER DEAN
VALENTIN CRETTAZ
MANNING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: [email protected]
©2019 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.
Manning Publications Co. Acquisitions editors: Mike Stephens and
20 Baldwin Road Frank Pohlmann
PO Box 761 Development editors: Jennifer Stout and Cynthia Kane
Shelter Island, NY 11964 Technical development editor: Kostas Passadis
Review editor: Aleks Dragosacljevic´
Production editor: Anthony Calcara
Copy editor: Sharon Wilkey
Proofreader: Melody Dolab
Technical proofreader: Michiel Trimpe
Typesetter: Dennis Dalinnik
Cover designer: Marija Tudor
ISBN: 9781617292347
Printed in the United States of America
brief contents
PART 1 EVENT STREAMS AND UNIFIED LOGS. .............................1
1 ■ Introducing event streams 3
2 ■ The unified log 24
3 ■ Event stream processing with Apache Kafka 38
4 ■ Event stream processing with Amazon Kinesis 60
5 ■ Stateful stream processing 88
PART 2 DATA ENGINEERING WITH STREAMS............................115
6 ■ Schemas 117
7 ■ Archiving events 140
8 ■ Railway-oriented processing 171
9 ■ Commands 208
PART 3 EVENT ANALYTICS .....................................................235
10 ■ Analytics-on-read 237
11 ■ Analytics-on-write 268
v
contents
preface xiii
acknowledgments xiv
about this book xvi
about the authors xix
about the cover illustration xx
PART 1 EVENT STREAMS AND UNIFIED LOGS . ...................1
1 Introducing event streams 3
1.1 Defining our terms 4
Events 5 ■ Continuous event streams 6
1.2 Exploring familiar event streams 7
Application-level logging 7 ■ Web analytics 8
Publish/subscribe messaging 10
1.3 Unifying continuous event streams 12
The classic era 13 ■ The hybrid era 16
The unified era 17
1.4 Introducing use cases for the unified log 19
Customer feedback loops 19 ■ Holistic systems monitoring 21
Hot-swapping data application versions 22
vii
viii CONTENTS
2 The unified log 24
2.1 Understanding the anatomy of a unified log 25
Unified 25 ■ Append-only 26 ■ Distributed 27
Ordered 28
2.2 Introducing our application 29
Identifying our key events 30 ■ Unified log, e-commerce style 31
Modeling our first event 32
2.3 Setting up our unified log 34
Downloading and installing Apache Kafka 34 ■ Creating
our stream 35 ■ Sending and receiving events 36
3 Event stream processing with Apache Kafka 38
3.1 Event stream processing 101 39
Why process event streams? 39 ■ Single-event processing 41
Multiple-event processing 42
3.2 Designing our first stream-processing app 42
Using Kafka as our company’s glue 43 ■ Locking down
our requirements 44
3.3 Writing a simple Kafka worker 46
Setting up our development environment 46 ■ Configuring our
application 47 ■ Reading from Kafka 49 ■ Writing to
Kafka 50 ■ Stitching it all together 51 ■ Testing 52
3.4 Writing a single-event processor 54
Writing our event processor 54 ■ Updating our main
function 56 ■ Testing, redux 57
4 Event stream processing with Amazon Kinesis 60
4.1 Writing events to Kinesis 61
Systems monitoring and the unified log 61 ■ Terminology
differences from Kafka 63 ■ Setting up our stream 64
Modeling our events 65 ■ Writing our agent 66
4.2 Reading from Kinesis 72
Kinesis frameworks and SDKs 72 ■ Reading events with the
AWS CLI 73 ■ Monitoring our stream with boto 79
5 Stateful stream processing 88
5.1 Detecting abandoned shopping carts 89
What management wants 89 ■ Defining our algorithm 90
Introducing our derived events stream 91
CONTENTS ix
5.2 Modeling our new events 92
Shopper adds item to cart 92 ■ Shopper places order 93
Shopper abandons cart 93
5.3 Stateful stream processing 94
Introducing state management 94 ■ Stream windowing 96
Stream processing frameworks and their capabilities 97
Stream processing frameworks 97 ■ Choosing a stream
processing framework for Nile 100
5.4 Detecting abandoned carts 101
Designing our Samza job 101 ■ Preparing our project 102
Configuring our job 103 ■ Writing our job’s Java task 104
5.5 Running our Samza job 110
Introducing YARN 110 ■ Submitting our job 111
Testing our job 112 ■ Improving our job 113
PART 2 DATA ENGINEERING WITH STREAMS..................115
6 Schemas 117
6.1 An introduction to schemas 118
Introducing Plum 118 ■ Event schemas as contracts 120
Capabilities of schema technologies 121 ■ Some schema
technologies 123 ■ Choosing a schema technology for Plum 125
6.2 Modeling our event in Avro 125
Setting up a development harness 126 ■ Writing our health check
event schema 127 ■ From Avro to Java, and back again 129
Testing 131
6.3 Associating events with their schemas 132
Some modest proposals 132 ■ A self-describing event for
Plum 135 ■ Plum’s schema registry 137
7 Archiving events 140
7.1 The archivist’s manifesto 141
Resilience 142 ■ Reprocessing 143 ■ Refinement 144
7.2 A design for archiving 146
What to archive 146 ■ Where to archive 147
How to archive 148
7.3 Archiving Kafka with Secor 149
Warming up Kafka 150 ■ Creating our event archive 152
Setting up Secor 153
