Table Of ContentT H E G U I D E T O
CONTINUOUS
DELIVERY
VOLUME II FEB 2015
BROUGHT TO YOU IN PARTNERSHIP WITH
tABlE of contEnts
DEAR READER,
3 Summary & Key Takeaways
I am very excited to welcome 2015 with the
release of our Guide to Continuous Delivery. 2014 4 Key Research Findings
was a huge year for DZone Research with the
release of six incredibly informative research 8 Choosing CD Tools That Bring Your Team
guides that have now been downloaded over Together By mAtthEw skElton
150,000 times.
12 Fostering DevOps Through A Testing Center
DZone’s 2014 Guide to Continuous Delivery was of Excellence By Jp moRGEnthAl
a huge success with over 30,000 downloads and
still counting. The enormous amount of positive 16 Are Containers Part of Your IT Strategy?
feedback we received last year from our audience
By Jim BuGwADiA
has motivated us to produce another edition of
the Continuous Delivery research guide, taking 20 Continuous Delivery: Visualized INFOGRAPHIC
an even further look at DevOps best practices.
24 Beyond Tooling and Process: the Culture of
In the 2015 edition of the research guide we have Continuous Delivery By AnDREw phillips
expanded the focus into other areas of DevOps
implementation and management, including a 28 Continuous Delivery: From Theory to Practice
look at how Continuous Delivery practices seep
By yARon pARAsol
into other aspects of the organization outside
the development team. 32 Continuous Delivery Maturity Checklist 2.0
We have focused on the role of containerization, 34 Solutions Directory
automation, testing, and other best practices
that move Continuous Delivery beyond simply 44 Glossary
an exercise in tooling. The solutions directory
provides a comprehensive listing of the various
ARA, CI, and CM tools beneficial for practitioners cREDits
in the DevOps space.
DZonE REsEARch mARkEtinG & coRpoRAtE
We are already looking forward to an exciting
sAlEs mAnAGEmEnt
year with several interesting new topics that we John Esposito
plan to cover, including Application Performance Editor-in-Chief Kellet Atkinson Rick Ross
Management, Software Quality, the Internet of Jayashree Gopal General Manager CEO
Director of Research
Things, and Big Data/Business Intelligence. We
Alex Crafts Matt Schmidt
are excited to hear your feedback on this guide, Mitch Pronschinske VP of Sales President & CTO
Sr. Research Analyst
thoughts on future topics, and ways you think
Ashley Slate Brandon Nokes
we can make our research guides better suited to Benjamin Ball
Director of Design VP of Sales, Demand
Research Analyst
your needs. Generation
Matt Werner Chelsea Bosworth
Market Researcher Marketing Associate Hernâni Cerqueira
Thanks again for your time and for being a Lead Software
part of our great community. Wish you all a John Walter Chris Smith Engineer
Content Curator Production Advisor
wonderful year ahead and happy reading!
Ryan Spain
Content Curator
Lauren Clapper
JAyAshREE GopAl Content Curator
DiREctoR of REsEARch
Special thanks to our topic experts Andrew Phillips, JP Morgenthal, Jim
[email protected]
Bugwadia, Matthew Skelton, Sharone Zitzman, and our trusted DZone Most
Valuable Bloggers for all their help and feedback in making this report a
great success.
2015 guide to continuous delivery dzone.com/research/continuousdelivery 2
Summary &
of Continuous Delivery [1], the number dropped to 18%
overall. It’s important to note that this is a 10% increase
Key Takeaways
over last year—more teams are getting it right.
continuous DElivERy foR DAtABAsE mAnAGEmEnt
AnD infRA stRuctuRE While we did see some growth
BusinEssEs ARE Alw Ays GoinG to BE lookinG to ADopt in Continuous Delivery adoption to different software
software delivery practices that help them come to market lifecycle environments, we’re still waiting for aggressive
on time, on budget, at high quality, and with few to no adoption with database management and infrastructure.
failures along the way. Nothing about these goals has Application build management is still the most popular
changed—what’s changing is the way we achieve them. CD environment, with 61% of all respondents reporting
Previously this meant a greater focus on the development its usage. We also saw a 9% growth in the adoption of
team, but it goes beyond that, farther even than the Continuous Delivery practices for databases—a significant
operations team and QA, and extends into how the entire growth for an observation period of less than a year. The
organization operates. Today’s Continuous Delivery growth of database implementation is not surprising
implementation isn’t just an exercise in tooling or a list considering increasing adoption of Continuous Delivery
of best practices, it reaches into every element of the for database lifecycle management [2]. True Continuous
business. The DZone 2015 Guide to Continuous Delivery has Delivery has to be present at every stage of the software
more insight than ever into the status of DevOps in the production process.
enterprise and the obstacles facing developers, not only in
their tooling, but within the organization as a whole. WHICH SOFTWARE LIFECYCLE ENVIRONMENTS DOES
The resources in this guide includes: YOUR CONTINUOUS DELIVERY PROCESS EXTEND TO?
• Side-by-side feature comparison of the best % APP. BUILD
61% 33% MANAGEMENT
6
configuration management, continuous integration, and
application release automation tools (selected based on 30% 51% 19% DATABASES
several criteria including solution maturity, technical
innovativeness, relevance, and data availability). 22% 57% 21% INFRASTRUCTURE
IMPLEMENTED WANT TO IMPLEMENT NO DESIRE
• Expert knowledge for implementing Continuous Delivery
and DevOps best practices
thE cultuRE of continuous DElivERy is An oBstAclE
• The tools that developers are using at every stage of the
woRth tAcklinG Organizational culture plays a major
Continuous Delivery pipeline
role in achieving Continuous Delivery, but CD can also
• Assessing how excellence in testing affects the entire causes improvements in cultural metrics like collaboration
enterprise and efficiency. 64% of our audience reported that
“corporate culture” and a lack of collaborative practices
• Analysis and forecasting based on research collected from
was the biggest obstacle facing their implementation
900+ IT professionals
of Continuous Delivery—and this is an obstacle
that can’t be overcome with proper tooling and
kEy tAkEAwAys
automation. Company culture also matters
continuous DElivERy: it’s DEfinitEly DO YOU to IT professionals when it comes to
A GRowinG pRActicE Exactly half BELIEVE YOUR measuring the success of Continuous
(50%) of the 900+ IT professionals ORGANIZATION 36% FOR SOME Delivery: 41% of our respondents
we surveyed believe that they have NO 50% HAS ACHIEVED PROYJEESCTS, reported that “cultural metrics” like
implemented Continuous Delivery - 9% FROM CONTINUOUS team collaboration and organizational
LAST YEAR
for some or all of their projects. This + 9% FROM efficiency play a role in measuring the
DELIVERY? LAST YEAR
represents an almost 10% growth impact of Continuous Delivery practices,
14%
from last year’s research. Our data which made it the third most important
YES
indicates that while many professionals metric of success after support ticket
SAME AS LAST YEAR
and organizations might be attempting to frequency (46%) and outage frequency (43%).
implement Continuous Delivery, they often aren’t meeting
[1] http://martinfowler.com/bliki/ContinuousDelivery.html
the ideal criteria. When we filtered respondents by stricter [2] https://www.simple-talk.com/sql/database-administration/continuous-delivery-
for-databases-microservices,-team-structures,-and-conways-law/
criteria based on three key traits in the definition
3 dzone.com/research/continuousdelivery 2015 guide to continuous delivery
kEy For example, development teams were only slightly
more likely to perform code deployments to production
(45%) than operations teams (32%). 58% of respondents
that said both development and operations were both
REsEARch
responsible for production support.
moRE pRofEssionAls ARE AchiEvinG continuous
finDinGs DElivERy
The authors of the Continuous Delivery methodology
defined three key traits to determine when an
organization has fully implemented its practices [1].
The panels below shows how many survey respondents
have these traits in their systems:
More than 900 IT professionals responded to
Is your software Does your team Do all of the
DZone’s 2015 Continuous Delivery Survey. perform push-button
confirmed to be in a stakeholders have
deployments of any
Here are the demographics for this survey: shippable state immediate visibility
desired version of your
every time a new into the production
software to any
feature or patch is environment readiness of your
• Developers and Engineers made up 65% of
added? on-demand? systems?
the total respondents.
43% SAID YES
47% SAID USUALLY
31% SAID YES
• 62% of respondents come from large
27% SAID FOR
24% SAID ALWAYS
organizations (100 or more employees) and SOME SYSTEMS
38% come from small organizations (under
100 employees).
Most organizations are somewhere in the process of
having adopted Continuous Delivery practices, but
• The majority of respondents are
not having fully achieved the primary principles. 36%
headquartered in Europe (48%) or the US
of those surveyed believe they have achieved CD for
(28%). some of their projects, and 14% believe they are doing
it in all of their projects. The combined statistic is that
50% of respondents have implemented Continuous
Division BEtwEEn DEv AnD ops is closinG
Delivery for some or all of their projects, which
Continuous Delivery has always been positioned
represents a 9% growth in implementation over the
as being part of the larger DevOps philosophy. To
last year. To determine the amount of respondents
implement this philosophy, many companies choose
performing a “textbook” implementation,
to create a team that is focused on cross-compatible
respondents were filtered by the three key traits of
skills for multiple disciplines between development
Continuous Delivery from the section above; 18% said
and operations. 35% of the
“yes” or “always” for all of the questions. While this
survey respondents
is smaller than the half of respondents who believe
have an officially
they’ve implemented Continuous Delivery, this
designated DevOps
number still represents a 10% growth compared to the
WHO IS team (up 5% from
number of respondents who had achieved “textbook”
58% RESPONSIBLE last year). For
Continuous Delivery last year (8%).
FOR PRODUCTION organizations
BOTH
DEVELOPMENT SUPPORT? without DevOps
& OPERATIONS
teams, the division 50% BELIEVE THEY’VE
22%
IMPLEMENTED CD
OTHER 3% of labor between
5% 12% DEVELOPMENT development and 18% PERFORM THE TEXTBOOK
DEVOPS operations teams DEFINITION OF CD
OPERATIONS
can become blurry.
2015 guide to continuous delivery dzone.com/research/continuousdelivery 4
Quick to REcovER AnD slow to fAil continuous DElivERy is spREADinG to o thER
Among other deployment and support metrics, EnviRonmEnts
we checked into the Mean Time to Recover (MTTR) Continuous Delivery isn’t a
and Mean Time Between Failure (MTBF) for our hard sell for developers.
respondents’ support and operations teams. The 61% say that they have
IS
survey indicated that recovery time after a failure already implemented CONTINUOUS 39% YES
(MTTR) averaged close to 6 hours. The respondents it for their application INTEGRATION
also reported that the average time between failures build environments, EXTENDED INTO THE
PRODUCTION
(MTBF) is just over 4 hours, with 13% reporting. and only 6% have no
DEPLOYMENT
61%
desire to do so. Database
PROCESS?
and infrastructure
24+ HOURS NO
MEAN TIME BETWEEN FAILURES (MTBF) 12-24 HOURS 5% environments are still
11% lagging behind, but they’ve seen decent growth
12% 0-2 HOURS
over the last year. 30% of respondents have
13% 3-7 HOURS
implemented it for database environments
5% 8-11 HOURS 8-11 11% MEAN TIME
8% 12-24 HOURS HOURS TO RECOVERY 38% 0-2 and 22% have implemented it for
HOURS infrastructure, which represents a combined
16% 2-7 DAYS (MTTR)
13% growth in these environments. Another
16% 1-3 WEEKS
positive sign is that over half of all respondents
16% 1-3 MONTHS
3-5 35% hope to implement CD for those environments.
14% 4+ MONTHS
HOURS The survey results do show some promising stats for
organizations taking the first steps toward complete
CD. 39% of respondents say they have extended their
cultuRE is Both oBstAclE AnD mEA suRE of CI practices to production deployments.
succEss
The three biggest barriers to Continuous Delivery
adoption are company culture (64%), a lack of time continuous DElivERy is BEcominG thE
(63%), and team skillsets (45%). This is the second univERsAl s tAnDARD
time that company culture and a lack of time have As popular as Continuous Delivery has become, it
topped the list of reasons why IT professionals are would be a stretch to say it’s currently a universal
having problems adopting CD practices. The healthy standard for software delivery, though it’s not that
growth of certain hard to say that it’s well on its way. Respondents
practices within a MAIN BARRIERS TO ADOPTING CD largely praised Continuous
IT’S NOT FOR EVERYONE
company culture has 64% COMPANY CULTURE Delivery as a near universal 20%
always been a major 63% LACK OF TIME standard, with 20% 12% ITS TWANONDA’TR BDE
focus within DevOps, saying that they think IS
so it seems natural 45% TEAM SKILLSETS Continuous Delivery CONTINUOUS
DELIVERY
that negative culture is currently 20%
BECOMING A
would be a barrier a universal IT IS UNIVERSAL
to implementation. MEASURES OF DEVOPS EXCELLENCE standard, and STANDARD STANDARD?
It’s not surprising 46% SUPPORT TICKET FREQUENCY 48% saying it
then that the 43% OUTAGE FREQUENCY will soon become 48% IT WILL BE
STANDARD
responsiveness 41% CULTURAL METRICS the standard; that’s a
of DevOps culture combined 68% of respondents. Even the negative
can also be used to responses were relatively tame—20% of respondents
measure the success of Continuous Delivery. Cultural just don’t think CD practices are appropriate for every
metrics (41%) are the third most important measure environment. 12% said they just don’t think it’ll be a
of success after support ticket frequency (46%) and universal standard.
outage frequency (43%).
[1] http://martinfowler.com/bliki/ContinuousDelivery.html
5 dzone.com/research/continuousdelivery 2015 guide to continuous delivery
6
sponsoRED opinion
Pre-Requisites for a 4. Share tools and procedures between teams
5. Make your application and infrastructure both
Successful Enterprise
production- and project-friendly: deployments should
be non-events - all the tooling and documentation
required to empower the development and QA teams
Continuous Delivery
and make them autonomous should be provided.
Implementation “Continuous delivery can be
seen as a natural evolution from
continuous integration and agile
Businesses today are moving toward continuous delivery
as a methodology to meet the ever-increasing demand to software development practices.”
deliver better software faster. Continuous delivery can be
seen as a natural evolution from continuous integration and
agile software development practices, but the cultural and
6. Make application versions ready to be shipped into
operational changes to achieve it are great. While these
production. Continuous delivery is not just about
changes are a huge challenge, there are many practical steps
a set of tools, ultimately it is also about the people
you can take to make real progress today. Here are six:
and organisational culture. Technology, people and
process all have to be aligned. If organisations are to
1. Make sure the development, QA and operations teams
reap the rewards of a more fluid, automated approach
all have shared goals and that they communicate.
to software development that can also provide them
business agility – they need to implement these best
2. Continuous integration is a prerequisite for
practice steps on the path to continuous delivery.
continuous delivery – get CI right, first.
3. Automate and version everything WRITTEn By
Cyrille Le Clerc, Director, Product Management , Cloudbees
Jenkins enterprise
by cloudBees continuous integration
CloudBees’ Enterprise Jenkins provides high availability for mission critical use of continuous delivery, as well as simplified
manageability of Jenkins to ease administration across distributed architectures, on-premise or in the cloud.
Does not require source control HostING tHIrd Party PluGINs
Push or pull agent model
through websites. On-premise or cloud-hosted Supported
Case study CloudBees worked with a top global financial institution in order to save time and improve suPPorted Platforms
quality of application development with Jenkins Enterprise. Prior to the Jenkins CI open source implementation, • Java Runtime Environment
applications were being developed and maintained by the corporate branch of the bank, but there was no
centralised management of the process, and developers couldn’t always access build assets. Implementing
Cm INteGratIoNs
Jenkins meant the bank gained the desired standardisation, along with other benefits of a centralised system that
were needed. The development organisation chose Jenkins because of prior experience with the software, its • Chef • Ansible
robustness, and the availability of useful plugins to match their requirements. Having installed Jenkins to support • Puppet • cfengine
continuous integration (CI) and monitor the build management process, the financial institution needed additional • SaltStack
features and functionality that the standard open source solution didn’t offer.To overcome these challenges, the
bank evaluated and ultimately selected Jenkins Enterprise by CloudBees. They had two primary reasons: the features
additional plugins that provide necessary functionality for large enterprises in the areas of backup, security, high
• Designate source control
availability and job organisation and the ability to obtain formal technical support for open source Jenkins and all
check-in as the mechanism
600+ open source Jenkins plugins.
to trigger a deployment
• Set new builds to be the
Customers
trigger for new deployments
• Intuit • Apple • Nokia • Alcatel Lucent • Automatic validation builds
• Target • Nordstrom • Liberty Mutual • ESPN for code changes to the
master branch
BLog blog.cloudbees.com twitter @cloudbees weBsite cloudbees.com
7 dzone.com/research/continuousdelivery 2015 guide to continuous delivery
support person,
choosing cD tools that and they feel more
empowered to say
“yes” or “no” to
a configuration
Bring your team together
change. They can sit
down and explain
to the developer
why they think
By mAtthEw skElton
the config change is unwise, based on comments in the pull
request and inline diffs of changes, all from a single web page.
with An EvER-incREAsinG ARRAy of tools AnD tEchnoloGiEs
When you emphasize change review via merge requests in
claiming to enable Continuous Delivery, how do you know
a web UI, version control can help to foster collaboration
which tools to try or to choose? In-house, open source, or
and communication between teams, especially with more
commercial? Ruby or shell? Dedicated or plugins? Highly
operations-focused teams that historically have less
collaborative practices such as DevOps and Continuous
experience in version control.
Delivery require new ways of assessing tools and technologies
in order to avoid creating new silos. This article identifies four
DEploymEnt pipElinE
guiding factors for choosing tools: collaboration, learning,
The deployment pipeline is a central concept in Continuous
singleton tools, and Conway’s law.
Delivery, providing visualization and orchestration of
all activities that take place between committing code to
Over the past few years, I have worked with many
version control and changes appearing in production. A good
organizations across different industry sectors to help
deployment pipeline allows you to deploy both new features
them improve their effectiveness in building and operating
and bug fixes using the same process, and you can choose
software systems. Part of my role was to help these
which kinds of testing to undertake before the changes go
organizations to adopt practices like DevOps and Continuous
live. Deployment pipelines are typically responsible for
Delivery. Choosing the right tools to support these practices
things like software builds, running unit tests, deployments,
is very important. Finding the right tool to use is not simply a
test execution, and so on.
case of comparing its advertised features or capabilities with
those of a competitor—you need to consider the ways that a
However, you can use the deployment pipeline for much
tool can affect multiple teams within an organization.
more than these technical tasks. A well-chosen deployment
pipeline tool allows us to model the existing workflow—
Rather than seeing a tool as purely achieving a particular
including team roles—and show the progress visually. This
task, you should look for ways in which a tool can be used to
means that a deployment pipeline can also be used to engage
achieve higher-level aims, such as aligning goals between
teams across the organization and reduce fear of change
Development and Operations, improving engagement with
by visualizing current processes and showing their gradual
business and commercial teams, and improving the quality of
evolution. This leads to increased collaboration and trust,
the software systems.
making changes easier and more rapid, while providing an
audit trace.
collABoRAtion
vERsion contRol
loG AGGREGAtion
When considering the purpose of version control systems,
Log aggregation has historically been seen as an
you usually think of things like source code, change history,
operations concern, not one for developers. However,
managing parallel work, audit logs, and so on. However,
with the emergence of modern log aggregation tools
version control systems can be used in ways that help with
(both on-premise and hosted), developers are realizing
communication and collaboration. Many organizations still
that logging has many benefits and is especially helpful
have a separate team responsible for deploying and releasing
for troubleshooting problems in production, where a
changes, and often these teams are not very comfortable with
debugger is not an option. Through the use of Event type
version control.
IDs and explicit transaction tracing, developers can ditch
the debugger and use logging instead, which means that
However, using Github-style pull requests (or merge
developers and operations teams can use the same tool for
requests) along with a web-based front-end, developers and
diagnosing problems in any environment.
deployment support can collaborate on the changes, with a
full history of discussions about the changes in the comments. The use of log aggregation across all environments opens up
In this scenario, Git feels less “scary” to the deployment collaboration between developers, testers, and operations
2015 guide to continuous delivery dzone.com/research/continuousdelivery 8
teams to enable much more rapid diagnosis and resolution between them, dictate the architecture of the software
of problems in production and upstream environments. It systems they build (Conway’s Law). If you want a three-
also creates more trust between teams because problems are tier system, then have three teams (front-end, services,
found earlier. database). If you want microservices, let the teams have
full-stack responsibility for one or more distinct business
lEARninG concepts, and so on. The impact of Conway’s Law has often
been focused on the application level, but less so at the level
The proliferation of new tools and techniques for DevOps
of the whole system, including operations and QA teams.
and Continuous Delivery can be difficult to keep pace with.
Many people involved in building and operating software
Some organizations—especially those with a single unified
systems are overwhelmed by the technologies and feel
product or the ability to choose when all features go live—
threatened when asked to use a new tool.
might want strong collaboration between development
You must be careful not to leave people behind when and operations teams so that a high change velocity is
choosing tools. Just as someone new to running might sustained. In this case, the development and operations
initially only run a few miles each day, increasing their teams would likely use many of the same tools—for
distance over a period of months, you should choose tools instance: version control, ticketing, deployment tools, and
in a way that encourages people and teams to “get started” configuration management.
with tools without insisting that they have to become an
expert within a week. This sometimes means choosing tools Alternatively, let’s say that another organization has an
that have fewer features than other tools, so that people internal Infrastructure-as-a-Service (IaaS) team, providing
can have time to learn, with the expectation that they may a cloud platform running on Xen and OpenStack. In this
replace or augment that tool with a more advanced tool in 12 case, you would want the development teams to interact
or 18 months’ time. with the IaaS team in a limited way, so that you preserve the
“as-a-Service” relationship and don’t blur the boundaries
This is particularly true for teams that have limited of responsibility. You would expect the use of shared
experience with automation. You need to allow teams to tools here to be very limited and for other tools, such as
learn which tools and approaches work for them, rather monitoring, metrics, logging, and configuration, to be
than imposing some kind of “best of breed” tools that might entirely separate between the two groups.
be too difficult to use.
In some cases, it is better for teams not to share the same
sinGlEton t ools tool, particularly if a boundary of responsibility needs to
be preserved. Collaboration through tools should match
Tools that exist in only one environment can be particularly
the boundaries of responsibility within and across teams,
problematic when organizations try to adopt DevOps
otherwise Conway’s Law will start to take effect and modify
or Continuous Delivery. A “singleton tool” might be a
the systems being built.
database technology, a logging or monitoring platform, a
load-balancer, or a virtualization technology that exists
only in the production environment, usually due to the high summARy
cost of the licenses. You can choose tools for DevOps and Continuous Delivery in
a way that significantly enhances the communication and
The problem with singleton tools is that they tend to create collaboration between teams, even if a tool’s main purpose
a divide between development and operations (aka “silos”). is not collaboration. You may need to avoid tools that are too
The special tool in production is seen by developers as complex for teams at a given point in time, and you should
“an Ops thing,” and operations people see no reason why prefer tools that can be present in all environments rather
developers should have access to certain data because “it’s than “singleton tools” that exist only in production. You
an Ops thing.” This breaks the vital feedback loop from should also make sure that the use of shared tools aligns
production to development, limiting the improvements with the responsibility boundaries between teams in order
that can be made to the software. to avoid confusion and the effect of Conway’s Law.
Singleton tools make upstream testing much more difficult,
meaning that high rates of failed deployments and buggy
releases are effectively guaranteed. mAtthEw skElton has been building, deploying, and
operating commercial software systems since 1998. Co-
founder and Principal Consultant at Skelton Thatcher
conwAy’s lAw Consulting, he specializes in helping organizations to adopt
It’s becoming increasing clear that the structure of teams and sustain good practices for building and operating
software systems: Continuous Delivery, DevOps, aspects of
within organizations, and the communication patterns
ITIL, and software operability.
9 dzone.com/research/continuousdelivery 2015 guide to continuous delivery
MARTIAL ARTS AUTOMATED TESTING
HAS BRUCE LEE. HAS SAUCE LABS.
Maybe you can’t do a one-fi ngered push-up, but you can master speed and scale
with Sauce Labs. Optimized for the continuous integration and delivery workfl ows
of today and tomorrow, our reliable, secure cloud enables you to run your builds
in parallel, so you can get to market faster without sacrifi cing coverage.
Try it for free at saucelabs.com and see
why these companies trust Sauce Labs.
10
