Flow Architectures: The Future of Streaming and Event-Driven Integration

254 Pages·2021·11.7 MB·english

by  James Urquhart

Flow
Architectures
The Future of Streaming and
Event-Driven Integration
James Urquhart

Flow Architectures
The Future of Streaming and
Event-Driven Integration
James Urquhart
BBeeiijjiinngg BBoossttoonn FFaarrnnhhaamm SSeebbaassttooppooll TTookkyyoo
Flow Architectures
by James Urquhart
Copyright © 2021 James Urquhart. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional
sales department: 800-998-9938 or [email protected].
Acquisitions Editor: Melissa Duffield Indexer: WordCo Indexing Services, Inc.
Development Editor: Amelia Blevins Interior Designer: David Futato
Production Editor: Kate Galloway Cover Designer: Karen Montgomery
Copyeditor: Piper Editorial, LLC Illustrator: Kate Dullea
Proofreader: Justin Billing
January 2021: First Edition
Revision History for the First Edition
2021-01-06: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781492075899 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Flow Architectures, the cover image,
and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the author, and do not represent the publisher’s views.
While the publisher and the author have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.
978-1-492-07589-9
[LSI]
To my children, Owen and Emery:
You each inspire me every day. I am so very proud of you.
And to my wife, Mia: You are my favorite.

Table of Contents
Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
The 10-Year Impact of the World Wide Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
1. Introduction to Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What Is Flow? 3
Flow and Integration 6
Flow and Event-Driven Architectures 9
The Ancestors of Flow 12
Code and Flow 13
The Chapters Ahead 15
2. The Business Case for Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Drivers for Flow Adoption 18
Improving Customer Experience 18
Improved Organizational Efficiency 20
Innovation and Experimentation 24
Enablers of Flow Adoption 26
Lowering the Cost of Stream Processing 26
Increasing the Flexibility of Data Flow Design 29
Creating the Great Flow Ecosystem 33
What Businesses Will Require from Flow 36
The Effects of Flow Adoption 37
Expanding the Use of Timely Data 37
The Importance (and Peril) of Flow Networks 38
Flow’s Impact on Jobs and Expertise 40
v
Flow and New Business and Institutional Models 41
Flow and Scale 43
Next Steps 44
3. Understanding the Flow Value Chain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Recap: The High-Level Properties for Flow 45
Wardley Mapping and Promise Theory 46
Wardley Mapping 47
Promise Theory 48
Building a Flow Integration Value Chain 51
Establishing a Scope for the Map 51
Establishing Our Users and User Need 52
Flow Integration Components 53
Interaction Components 56
The Final Piece 60
Mapping Our Value Chain 61
Determining a Measure of Technology Evolution 61
Turning Our Value Chain into a Map 65
Our Final Model and Next Steps 70
4. Evaluating the Current Streaming Market. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Service Buses and Message Queues 74
Message Queues 75
Service Buses 76
Mapping Service Buses and Message Queues 77
Internet of Things 78
MQTT 79
HTTP and WebSocket 81
Mapping Internet of Things Architectures 82
Event Processing 83
Functions, Low-Code, and No-Code Processors 84
Log-Based Stream Processing Platforms 85
Stateful Stream Processing 87
Mapping Event Processing Platforms 88
Streaming Architectures and Integration Today 89
Next Steps 91
5. Evaluating the Emergence of Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Mapping the Evolution to Flow 94
Gameplay 97
Market: Standards Game 98
Accelerators: Exploiting Network Effects 99
vi | Table of Contents
Ecosystem: Cocreation 100
The Others 101
Inertia 102
Vendor Inertia 102
Enterprise Inertia 104
Here Be Dragons 105
Flow Requirements, Challenges, and Opportunities 108
Security 108
Agility 113
Timeliness 116
Manageability 120
Memory 125
Control of Intellectual Property 126
Flow Pattern Challenges and Opportunities 128
The Collector Pattern 128
The Distributor Pattern 129
The Signal Pattern 130
The Facilitator Pattern 132
The Unexpected 133
6. Building for a Flow Future. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Identifying Flow in Your Business 136
Flow Use Cases 137
Modeling Flow 148
“Event-First” Use Cases for Flow 151
Messaging Versus Eventing 152
Discrete Events Versus Event Series 155
Single Actions Versus Workflows 159
Driving Flow Forward 162
Driving Technology Development 162
Driving Flow Networks 166
We Will Make Flow Happen 170
Appendix. Evaluating the Current Flow Market. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Table of Contents | vii