RabbitMQ in Depth

266 Pages·2017·9.79 MB·English

by  Gavin M. Roy

IN DEPTH
Gavin M. Roy
M A N N I N G
RabbitMQ in Depth

RabbitMQ in Depth
GAVIN ROY
MANNING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: [email protected]
©2018 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.
Manning Publications Co. Development editor: Karen Miller
20 Baldwin Road Technical editor: James Titcumb
PO Box 761 Technical development editor: Phillip Warner
Shelter Island, NY 11964 Copyeditor: Andy Carroll
Proofreader: Alyson Brener
Technical proofreader: Karsten Strøbæk
Typesetter: Dennis Dalinnik
Cover designer: Marija Tudor
ISBN: 9781617291005
Printed in the United States of America
1 2 3 4 5 6 7 8 9 10 – EBM – 22 21 20 19 18 17
brief contents
PART 1 RABBITMQ AND APPLICATION ARCHITECTURE ...............1
1 ■ Foundational RabbitMQ 3
2 ■ How to speak Rabbit: the AMQ Protocol 18
3 ■ An in-depth tour of message properties 38
4 ■ Performance trade-offs in publishing 58
5 ■ Don’t get messages; consume them 79
6 ■ Message patterns via exchange routing 101
PART 2 MANAGING RABBITMQ IN THE DATA CENTER
OR THE CLOUD..........................................................133
7 ■ Scaling RabbitMQ with clusters 135
8 ■ Cross-cluster message distribution 148
PART 3 INTEGRATIONS AND CUSTOMIZATION.........................175
9 ■ Using alternative protocols 177
10 ■ Database integrations 205
v

contents
preface xiii
acknowledgments xiv
about this book xv
PART 1 RABBITMQ AND APPLICATION ARCHITECTURE......1
1 Foundational RabbitMQ 3
1.1 RabbitMQ’s features and benefits 4
RabbitMQ and Erlang 5 ■ RabbitMQ and AMQP 6
1.2 Who’s using RabbitMQ, and how? 7
1.3 The advantages of loosely coupled architectures 8
Decoupling your application 10 ■ Decoupling database
writes 11 ■ Seamlessly adding new functionality 12
Replication of data and events 12 ■ Multi-master federation
of data and events 13 ■ The Advanced Message
Queuing model 14
1.4 Summary 16
vii
viii CONTENTS
2 How to speak Rabbit: the AMQ Protocol 18
2.1 AMQP as an RPC transport 19
Kicking off the conversation 20 ■ Tuning in to the right channel 20
2.2 AMQP’s RPC frame structure 21
AMQP frame components 21 ■ Types of frames 22
Marshaling messages into frames 23 ■ The anatomy of
a method frame 24 ■ The content header frame 26
The body frame 26
2.3 Putting the protocol to use 27
Declaring an exchange 27 ■ Declaring a queue 28
Binding a queue to an exchange 29 ■ Publishing a message
to RabbitMQ 29 ■ Consuming messages from RabbitMQ 30
2.4 Writing a message publisher in Python 32
2.5 Getting messages from RabbitMQ 36
2.6 Summary 37
3 An in-depth tour of message properties 38
3.1 Using properties properly 39
3.2 Creating an explicit message contract with
content-type 41
3.3 Reducing message size with gzip and
content-encoding 43
3.4 Referencing messages with message-id and
correlation-id 45
Message-id 45 ■ Correlation-id 45
3.5 Born-on dating: the timestamp property 46
3.6 Automatically expiring messages 47
3.7 Balancing speed with safety using delivery-mode 48
3.8 Validating message origin with app-id and user-id 49
app-id 50 ■ user-id 51
3.9 Getting specific with the message type property 51
3.10 Using reply-to for dynamic workflows 52
3.11 Custom properties using the headers property 53
3.12 The priority property 54
3.13 A property you can’t use: cluster-id/reserved 54
3.14 Summary 55
CONTENTS ix
4 Performance trade-offs in publishing 58
4.1 Balancing delivery speed with guaranteed delivery 59
What to expect with no guarantees 60 ■ RabbitMQ won’t accept
non-routable messages with mandatory set 62 ■ Publisher
Confirms as a lightweight alternative to transactions 64
Using alternate exchanges for unroutable messages 66
Batch processing with transactions 68 ■ Surviving node failures
with HA queues 70 ■ HA queues with transactions 72
Persisting messages to disk via delivery-mode 2 72
4.2 When RabbitMQ pushes back 75
Checking the connection status with rabbitpy 77 ■ Using the
management API for connection status 77
4.3 Summary 78
5 Don’t get messages; consume them 79
5.1 Basic.Get vs. Basic.Consume 80
Basic.Get 80 ■ Basic.Consume 82
5.2 Performance-tuning consumers 84
Using no-ack mode for faster throughput 85 ■ Controlling
consumer prefetching via quality of service settings 86
Using transactions with consumers 89
5.3 Rejecting messages 90
Basic.Reject 90 ■ Basic.Nack 91 ■ Dead letter exchanges 92
5.4 Controlling queues 94
Temporary queues 94 ■ Permanent queues 97 ■ Arbitrary
queue settings 99
5.5 Summary 99
6 Message patterns via exchange routing 101
6.1 Simple message routing using the direct exchange 102
Creating the application architecture 103 ■ Creating the
RPC worker 107 ■ Writing a simple RPC publisher 110
6.2 Broadcasting messages via the fanout exchange 115
Modifying the facial detection consumer 116 ■ Creating a simple
image-hashing consumer 117
6.3 Selectively routing messages with the topic exchange 119
6.4 Selective routing with the headers exchange 122