Table Of ContentDylan Scott
Viktor Gamov
Dave Klein
Foreword by Jun Rao
M A N N I N G
Kafka in Action
ii
Kaf ka in Action
DYLAN SCOTT
VIKTOR GAMOV
AND DAVE KLEIN
FOREWORD BY JUN RAO
MAN NING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: [email protected]
©2022 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.
The author and publisher have made every effort to ensure that the information in this book
was correct at press time. The author and publisher do not assume and hereby disclaim any
liability to any party for any loss, damage, or disruption caused by errors or omissions, whether
such errors or omissions result from negligence, accident, or any other cause, or from any usage
of the information herein.
Manning Publications Co. Development editor: Toni Arritola
20 Baldwin Road Technical development editors: Raphael Villela, Nickie Buckner
PO Box 761 Review editor: Aleksandar Dragosavljevic´
Shelter Island, NY 11964 Production editor: Andy Marinkovich
Copy editor: Frances Buran
Proofreader: Katie Tennant
Technical proofreaders: Felipe Esteban Vildoso Castillo,
Mayur Patil, Sumant Tambe,
Valentin Crettaz, and William
Rudenmalm
Typesetter and cover designer: Marija Tudor
ISBN 9781617295232
Printed in the United States of America
Dylan: I dedicate this work to Harper, who makes me so proud every day, and to Noelle,
who brings even more joy to our family every day. I would also like to dedicate this book to
my parents, sister, and wife, who are always my biggest supporters.
Viktor: I dedicate this work to my wife, Maria, for her support during the process of
writing this book. It’s a time-consuming task, time that I needed to carve out here and
there. Without your encouragement, nothing would have ever happened. I love you. Also,
I would like to dedicate this book to (and thank) my children, Andrew and Michael, for
being so naïve and straightforward. When people asked where daddy is working, they
would say, “Daddy is working in Kafka.”
Dave: I dedicate this work to my wife, Debbie, and our children, Zachary, Abigail,
Benjamin, Sarah, Solomon, Hannah, Joanna, Rebekah, Susanna, Noah, Samuel,
Gideon, Joshua, and Daniel. Ultimately, everything I do, I do for the honor of
my Creator and Savior, Jesus Christ.
vi
brief contents
PART 1 GETTING STARTED .......................................................... 1
1 ■ Introduction to Kafka 3
2 ■ Getting to know Kafka 17
PART 2 APPLYING KAFKA .......................................................... 41
3 ■ Designing a Kafka project 43
4 ■ Producers: Sourcing data 66
5 ■ Consumers: Unlocking data 87
6 ■ Brokers 111
7 ■ Topics and partitions 129
8 ■ Kafka storage 144
9 ■ Management: Tools and logging 158
PART 3 GOING FURTHER ........................................................ 179
10 ■ Protecting Kafka 181
11 ■ Schema registry 197
12 ■ Stream processing with Kafka Streams and ksqlDB 209
vii
viii BRIEF CONTENTS
contents
foreword xv
preface xvi
acknowledgments xviii
about this book xx
about the authors xxiii
about the cover illustration xxiv
PART 1 GETTING STARTED ............................................... 1
1 Introduction to Kafka 3
1.1 What is Kafka? 4
1.2 Kafka usage 8
Kafka for the developer 8 ■ Explaining Kafka to your
manager 9
1.3 Kafka myths 10
Kafka only works with Hadoop® 10 ■ Kafka is the same as other
message brokers 11
1.4 Kafka in the real world 11
Early examples 12 ■ Later examples 13 ■ When Kafka might
not be the right fit 14
1.5 Online resources to get started 15
References 15
ix
