Table Of Content(cid:53)(cid:73)(cid:70)(cid:2)(cid:37)(cid:70)(cid:87)(cid:48)(cid:81)(cid:84)(cid:2)(cid:20)(cid:16)(cid:19)(cid:2)(cid:53)(cid:80)(cid:80)(cid:77)(cid:44)(cid:74)(cid:85)(cid:28)(cid:2)(cid:37)(cid:80)(cid:68)(cid:76)(cid:70)(cid:83)
(cid:52)(cid:88)(cid:66)(cid:83)(cid:78)
(cid:35)(cid:86)(cid:74)(cid:77)(cid:69)(cid:74)(cid:79)(cid:72)(cid:14)(cid:2)(cid:85)(cid:70)(cid:84)(cid:85)(cid:74)(cid:79)(cid:72)(cid:14)(cid:2)(cid:69)(cid:70)(cid:81)(cid:77)(cid:80)(cid:90)(cid:74)(cid:79)(cid:72)(cid:14)(cid:2)(cid:66)(cid:79)(cid:69)(cid:2)(cid:78)(cid:80)(cid:79)(cid:74)(cid:85)(cid:80)(cid:83)(cid:74)(cid:79)(cid:72)(cid:2)(cid:84)(cid:70)(cid:83)(cid:87)(cid:74)(cid:68)(cid:70)(cid:84)(cid:2)(cid:74)(cid:79)(cid:84)(cid:74)(cid:69)(cid:70)
(cid:37)(cid:80)(cid:68)(cid:76)(cid:70)(cid:83)(cid:2)(cid:52)(cid:88)(cid:66)(cid:83)(cid:78)(cid:2)(cid:68)(cid:77)(cid:86)(cid:84)(cid:85)(cid:70)(cid:83)(cid:84)
Viktor Farcic
BIRMINGHAM - MUMBAI
The DevOps 2.1 Toolkit: Docker Swarm
Copyright © 2017 Viktor Farcic
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the author, nor Packt Publishing, and its
dealers and distributors will be held liable for any damages caused or alleged to be caused
directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: May 2017
Production reference: 1020517
Published by Packt Publishing Ltd.
(cid:45)(cid:74)(cid:87)(cid:70)(cid:83)(cid:90)(cid:2)(cid:49)(cid:77)(cid:66)(cid:68)(cid:70)
(cid:21)(cid:23)(cid:2)(cid:45)(cid:74)(cid:87)(cid:70)(cid:83)(cid:90)(cid:2)(cid:52)(cid:85)(cid:83)(cid:70)(cid:70)(cid:85)
(cid:35)(cid:74)(cid:83)(cid:78)(cid:74)(cid:79)(cid:72)(cid:73)(cid:66)(cid:78)(cid:2)
(cid:35)(cid:21)(cid:2)(cid:20)(cid:49)(cid:35)(cid:14)(cid:2)(cid:54)(cid:44)(cid:16)
ISBN 978-1-78728-970-3
(cid:88)(cid:88)(cid:88)(cid:16)(cid:81)(cid:66)(cid:68)(cid:76)(cid:85)(cid:81)(cid:86)(cid:67)(cid:16)(cid:68)(cid:80)(cid:78)
Credits
Author Acquisition Editor
Viktor Farcic Frank Pohlmann
Technical Editors Production Coordinator
Joel Wilfred D'souza Shraddha Falebhai
Devesh Chugh
Indexer
Francy Puthiry
About the Author
Viktor Farcic is a senior consultant at CloudBees ((cid:73)(cid:85)(cid:85)(cid:81)(cid:84)(cid:28)(cid:17)(cid:17)(cid:88)(cid:88)(cid:88)(cid:16)(cid:68)(cid:77)(cid:80)(cid:86)(cid:69)(cid:67)(cid:70)(cid:70)(cid:84)(cid:16)(cid:68)(cid:80)(cid:78)(cid:17)), a member
(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
of the Docker Captains ((cid:73)(cid:85)(cid:85)(cid:81)(cid:84)(cid:28)(cid:17)(cid:17)(cid:88)(cid:88)(cid:88)(cid:16)(cid:69)(cid:80)(cid:68)(cid:76)(cid:70)(cid:83)(cid:16)(cid:68)(cid:80)(cid:78)(cid:17)(cid:68)(cid:80)(cid:78)(cid:78)(cid:86)(cid:79)(cid:74)(cid:85)(cid:90)(cid:17)(cid:69)(cid:80)(cid:68)(cid:76)(cid:70)(cid:83)(cid:15)(cid:68)(cid:66)(cid:81)(cid:85)(cid:66)(cid:74)(cid:79)(cid:84)) group,
(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
and an author.
He codes using a plethora of languages starting with Pascal (yes, he is old), Basic (before it
got the Visual prefix), ASP (before it got the .NET suffix), C, C++, Perl, Python, ASP.NET,
Visual Basic, C#, JavaScript, Java, Scala, and so on. He never worked with Fortran. His
current favorite is Go.
Viktor's big passions are Microservices, Continuous Deployment, and Test-Driven
Development (TDD).
He often speaks at community gatherings and conferences. Viktor wrote Test-Driven Java
Development by Packt Publishing, and The DevOps 2.0 Toolkit. His random thoughts and
tutorials can be found in his blog TechnologyConversations.com ((cid:73)(cid:85)(cid:85)(cid:81)(cid:84)(cid:28)(cid:17)(cid:17)(cid:85)(cid:70)(cid:68)(cid:73)(cid:79)(cid:80)(cid:77)(cid:80)(cid:72)(cid:90)(cid:68)(cid:80)(cid:79)(cid:87)(cid:70)(cid:83)
(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
(cid:84)(cid:66)(cid:85)(cid:74)(cid:80)(cid:79)(cid:84)(cid:16)(cid:68)(cid:80)(cid:78)(cid:17)).
(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
www.PacktPub.com
For support files and downloads related to your book, please visit (cid:88)(cid:88)(cid:88)(cid:16)(cid:49)(cid:66)(cid:68)(cid:76)(cid:85)(cid:49)(cid:86)(cid:67)(cid:16)(cid:68)(cid:80)(cid:78).
Did you know that Packt offers eBook versions of every book published, with PDF and
ePub files available? You can upgrade to the eBook version at (cid:88)(cid:88)(cid:88)(cid:16)(cid:49)(cid:66)(cid:68)(cid:76)(cid:85)(cid:49)(cid:86)(cid:67)(cid:16)(cid:68)(cid:80)(cid:78) and as a
print book customer, you are entitled to a discount on the eBook copy. Get in touch with us
at (cid:84)(cid:70)(cid:83)(cid:87)(cid:74)(cid:68)(cid:70)(cid:33)(cid:81)(cid:66)(cid:68)(cid:76)(cid:85)(cid:81)(cid:86)(cid:67)(cid:16)(cid:68)(cid:80)(cid:78) for more details.
At (cid:88)(cid:88)(cid:88)(cid:16)(cid:49)(cid:66)(cid:68)(cid:76)(cid:85)(cid:49)(cid:86)(cid:67)(cid:16)(cid:68)(cid:80)(cid:78), you can also read a collection of free technical articles, sign up for a
range of free newsletters and receive exclusive discounts and offers on Packt books and
eBooks.
(cid:73)(cid:85)(cid:85)(cid:81)(cid:84)(cid:28)(cid:17)(cid:17)(cid:88)(cid:88)(cid:88)(cid:16)(cid:81)(cid:66)(cid:68)(cid:76)(cid:85)(cid:81)(cid:86)(cid:67)(cid:16)(cid:68)(cid:80)(cid:78)(cid:17)(cid:78)(cid:66)(cid:81)(cid:85)
(cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt
books and video courses, as well as industry-leading tools to help you plan your personal
development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial
process.
If you'd like to join our team of regular reviewers, you can e-mail us at
(cid:68)(cid:86)(cid:84)(cid:85)(cid:80)(cid:78)(cid:70)(cid:83)(cid:83)(cid:70)(cid:87)(cid:74)(cid:70)(cid:88)(cid:84)(cid:33)(cid:81)(cid:66)(cid:68)(cid:76)(cid:85)(cid:81)(cid:86)(cid:67)(cid:16)(cid:68)(cid:80)(cid:78). We award our regular reviewers with free eBooks and
videos in exchange for their valuable feedback. Help us be relentless in improving our
products!
Table of Contents
Preface
1
Chapter 1: Continuous Integration with Docker Containers
5
Defining a fully Dockerized manual Continuous Integration flow 7
Running unit tests and building service binary 10
Building service images 12
Running staging tests 15
Pushing images to the registry 17
Chapter 2: Setting Up and Operating a Swarm Cluster
21
Scalability 22
Axis scaling 24
X-axis scaling 24
Y-axis scaling 26
Z-axis scaling 26
Clustering 26
Docker Swarm Mode 27
Setting up a Swarm cluster 29
Deploying services to the Swarm cluster 32
Scaling services 35
Failover 37
What now? 38
Chapter 3: Docker Swarm Networking and Reverse Proxy
39
Setting up a cluster 40
Requirements of secured and fault tolerant services running with high
availability 41
Running a database in isolation 42
Running a service through a reverse proxy 46
Creating a reverse proxy service in charge of routing requests
depending on their base URLs 48
Load balancing requests across all instances of a service 54
What now? 57
Chapter 4: Service Discovery inside a Swarm Cluster
58
What would Docker Swarm look like without? 58
What does standalone Docker Swarm look like with service discovery? 62
Service discovery in the Swarm cluster 64
Do we need service discovery? 65
Setting up Consul as service registry inside a Swarm cluster 65
Problems when scaling stateful instances 71
Using service registry to store the state 73
Discovering addresses of all instances that form a service 78
Using service registry or key value store to store service state 81
What now? 84
Chapter 5: Continuous Delivery and Deployment with Docker
Containers
85
Defining the Continuous Delivery environment 86
Setting up Continuous Delivery clusters 87
Using node labels to constrain services 90
Creating services 94
Walking through Continuous Delivery steps 97
Walking the extra mile from Continuous Delivery to Continuous
Deployment 103
What now? 105
Chapter 6: Automating Continuous Deployment Flow with Jenkins
106
Jenkins architecture 106
Production environment setup 107
Jenkins service 108
Jenkins failover 112
Jenkins agent 114
Creating services in production and production-like environments 124
Automating Continuous Deployment flow with Jenkins 125
Creating Jenkins Pipeline jobs 126
Defining Pipeline nodes 129
Defining Pipeline stages 131
Defining Pipeline steps 132
What now? 138
Chapter 7: Exploring Docker Remote API
140
Setting up the environment 140
Operating Docker Swarm through the Docker Remote API 142
Using Docker Remote API to automate proxy configuration 153
Combining the Swarm listener with the proxy 154
Automatically reconfiguring the proxy 155
Removing a service from the proxy 157
[ ii ]
What now? 158
Chapter 8: Using Docker Stack and Compose YAML Files to Deploy
Swarm Services
159
Swarm cluster setup 160
Creating Swarm services through Docker stack commands 161
Deploying more stacks 163
To stack or not to stack 166
Cleanup 166
Chapter 9: Defining Logging Strategy
167
The need for centralized logging 169
Setting up ElasticSearch as the logging database 170
Setting up LogStash as the logs parser and forwarder 172
Forwarding logs from all containers running anywhere inside a Swarm
cluster 176
Exploring logs 179
Discussing other logging solutions 185
What now? 186
Chapter 10: Collecting Metrics and Monitoring the Cluster
188
The requirements of a cluster monitoring system 188
Choosing the right database to store system metrics 190
Creating the cluster 191
Prometheus metrics 194
Exporting system wide metrics 195
Scraping, querying, and visualizing Prometheus metrics 201
Using Grafana to create dashboards 209
Exploring Docker Swarm and container overview dashboard in
Grafana 217
Adjusting services through dashboard metrics 225
Monitoring best practices 228
What now? 229
Chapter 11: Embracing Destruction: Pets versus Cattle
230
What now? 233
Chapter 12: Creating and Managing a Docker Swarm Cluster in
Amazon Web Services
234
Installing AWS CLI and setting up the environment variables 235
Setting up a Swarm cluster with Docker Machine and AWS CLI 239
Setting up a Swarm cluster with Docker for AWS 248
[ iii ]
