SlideShare a Scribd company logo
§Continuous delivery
Continuous delivery - takeaways
1. Antipatterns
2. Proposed model
3. Common delivery issues
Agenda
§Antipatterns
Deploying software manually
• production of extensive, detailed documentation that describes the steps to be
taken and the ways in which the steps may go wrong
• reliance on manual testing to confirm that the application is running correctly
• frequent calls to the development team to explain why a deployment is going
wrong on a release day
• frequent corrections to the release process during the course of a release
• environments in a cluster that differ in their configuration
• releases that take more than a few minutes to perform
• unpredictable releases that often have to be rolled back or run into unforeseen
problems
Deploying to a production-like environment
only after development complete
• testers, if involved by this stage, have tested the system on development
machines
• releasing into staging is the first time that operations people interact with the
new release
• dev team assembles the correct installers, configuration files, db migrations and
deployment documentation to pass to the deployment team without having tested
them in an environment that looks like production or staging
• little/no collaboration between the dev and deployment teams
Manual configuration management of
production environments
• having deployed successfully many times to staging, the deployment into
production fails
• different members of a cluster behave differently
• operations team take a long time to prepare an environment for a release
• cannot step back to an earlier configuration of your system
• servers in clusters have, unintentionally, different versions of OS, 3rd party
infrastructure, libraries or patch levels
§Proposed model
Software delivery principles
 Create a repeatable, reliable release process
 Automate almost everything
 Keep everything in version control
 If it hurts, do it more frequently and bring the pain forward
 Build quality in
 Done means released
 Everybody is responsible for delivery process
 Continuous improvement
The deployment pipeline
- deployment pipeline = an automated implementation of your application’s build,
deploy, test, and release process
- makes every part of the process of building, deploying, testing, and releasing software
visible to everybody involved
- improves feedback so that problems are identified and resolved as early as possible
- enables teams to deploy and release any version of their software to any environment
at will through a fully automated process
Blue-Green Deployments
- blue and green = 2 identical versions of production environment
- deploy new version to the blue environment
- when ready, change the router to point to blue – blue becomes production
- if something goes wrong, switch the router back to green
- careful with the db!
- if budget allows, blue and green can be completely separate replicas
- if not, use 2 copies of your application running side by side on the same environment
as blue and green
Canary Releasing
§
Common delivery issues
Infrequent/buggy deployments - symptoms
Problem - long time to deploy the build, brittle deployment process
Symptoms
• long time for bugs to be closed by testers
• long time for stories to be tested or signed off by the customer
• testers are finding bugs that developers fixed a long time ago
• nobody trusts the UAT, performance, or CI environments, and people are
skeptical as to when a release will be available
• showcases rarely happen
• the application can rarely be demonstrated to be working
• the team’s velocity (rate of progress) is slower than expected
Infrequent/buggy deployments - causes
• deployment process not automated
• not enough hardware available
• hardware and OS configuration not managed correctly
• deployment process depends on systems outside the team’s control
• not enough people understand the build and deployment process
• testers, developers, analysts, and operations personnel not collaborating
sufficiently during development
• developers not disciplined about keeping the application working by making
small, incremental changes, and so frequently break existing functionality
Poor application quality - symptoms
Problem - Delivery teams failing to implement an effective testing strategy
Symptoms
• regression bugs keep popping up
• the defect number keeps increasing even when your team spends most of its
time fixing them
• customers complain of a poor-quality product
• developers groan and look horrified whenever a new feature request arrives
• developers complain about the maintainability of the code
• an ever-increasing amount of time to implement new functionality, the team
starts falling behind
Poor application quality - causes
• testers do not collaborate with developers during feature development
• stories or features “DONE” without comprehensive automated tests, being signed
off by testers, or being showcased to users from a production-like environment
• defects routinely entered into a backlog without being fixed on the spot with an
automated test to detect regression problems
• developers or testers without enough experience developing automated test
suites
• team doesn’t understand the most effective types of tests to write for the
technology or platform
• developers working without enough test coverage
Poorly managed CI - symptoms
Problem - build process not properly managed
Symptoms
• developers don’t check in often enough (at least once a day)
• commit stage permanently broken
• high number of defects
• long integration phase before each release
Poorly managed CI - causes
• automated tests take too long to run
• commit stage takes too long to run (ideally under 5 minutes, more than 10
minutes unacceptable)
• automated tests fail intermittently, giving false positives
• nobody empowered to revert check-ins
• not enough people understand and can change the CI process
Poor configuration management - symptoms
Problem - environments can’t be commissioned, and applications installed
reliably, using an automated process
Symptoms
• mysterious failures in production environments
• new deployments are tense, scary events
• large teams dedicated to environment configuration and management
• deployments to production often have to be rolled back or patched
• unacceptable downtime of production environment
Poor configuration management - causes
• UAT and production environments are different
• poor or badly enforced change management process for making changes to
production and staging environments
• insufficient collaboration between operations, data management and delivery
teams
• ineffective monitoring of production and staging environments to detect
incidents
• insufficient instrumentation and logging built into applications
• insufficient testing of the non-functional requirements of applications
§
Thanks!

More Related Content

Continuous delivery - takeaways

  • 3. 1. Antipatterns 2. Proposed model 3. Common delivery issues Agenda
  • 5. Deploying software manually • production of extensive, detailed documentation that describes the steps to be taken and the ways in which the steps may go wrong • reliance on manual testing to confirm that the application is running correctly • frequent calls to the development team to explain why a deployment is going wrong on a release day • frequent corrections to the release process during the course of a release • environments in a cluster that differ in their configuration • releases that take more than a few minutes to perform • unpredictable releases that often have to be rolled back or run into unforeseen problems
  • 6. Deploying to a production-like environment only after development complete • testers, if involved by this stage, have tested the system on development machines • releasing into staging is the first time that operations people interact with the new release • dev team assembles the correct installers, configuration files, db migrations and deployment documentation to pass to the deployment team without having tested them in an environment that looks like production or staging • little/no collaboration between the dev and deployment teams
  • 7. Manual configuration management of production environments • having deployed successfully many times to staging, the deployment into production fails • different members of a cluster behave differently • operations team take a long time to prepare an environment for a release • cannot step back to an earlier configuration of your system • servers in clusters have, unintentionally, different versions of OS, 3rd party infrastructure, libraries or patch levels
  • 9. Software delivery principles  Create a repeatable, reliable release process  Automate almost everything  Keep everything in version control  If it hurts, do it more frequently and bring the pain forward  Build quality in  Done means released  Everybody is responsible for delivery process  Continuous improvement
  • 10. The deployment pipeline - deployment pipeline = an automated implementation of your application’s build, deploy, test, and release process - makes every part of the process of building, deploying, testing, and releasing software visible to everybody involved - improves feedback so that problems are identified and resolved as early as possible - enables teams to deploy and release any version of their software to any environment at will through a fully automated process
  • 11. Blue-Green Deployments - blue and green = 2 identical versions of production environment - deploy new version to the blue environment - when ready, change the router to point to blue – blue becomes production - if something goes wrong, switch the router back to green - careful with the db! - if budget allows, blue and green can be completely separate replicas - if not, use 2 copies of your application running side by side on the same environment as blue and green
  • 14. Infrequent/buggy deployments - symptoms Problem - long time to deploy the build, brittle deployment process Symptoms • long time for bugs to be closed by testers • long time for stories to be tested or signed off by the customer • testers are finding bugs that developers fixed a long time ago • nobody trusts the UAT, performance, or CI environments, and people are skeptical as to when a release will be available • showcases rarely happen • the application can rarely be demonstrated to be working • the team’s velocity (rate of progress) is slower than expected
  • 15. Infrequent/buggy deployments - causes • deployment process not automated • not enough hardware available • hardware and OS configuration not managed correctly • deployment process depends on systems outside the team’s control • not enough people understand the build and deployment process • testers, developers, analysts, and operations personnel not collaborating sufficiently during development • developers not disciplined about keeping the application working by making small, incremental changes, and so frequently break existing functionality
  • 16. Poor application quality - symptoms Problem - Delivery teams failing to implement an effective testing strategy Symptoms • regression bugs keep popping up • the defect number keeps increasing even when your team spends most of its time fixing them • customers complain of a poor-quality product • developers groan and look horrified whenever a new feature request arrives • developers complain about the maintainability of the code • an ever-increasing amount of time to implement new functionality, the team starts falling behind
  • 17. Poor application quality - causes • testers do not collaborate with developers during feature development • stories or features “DONE” without comprehensive automated tests, being signed off by testers, or being showcased to users from a production-like environment • defects routinely entered into a backlog without being fixed on the spot with an automated test to detect regression problems • developers or testers without enough experience developing automated test suites • team doesn’t understand the most effective types of tests to write for the technology or platform • developers working without enough test coverage
  • 18. Poorly managed CI - symptoms Problem - build process not properly managed Symptoms • developers don’t check in often enough (at least once a day) • commit stage permanently broken • high number of defects • long integration phase before each release
  • 19. Poorly managed CI - causes • automated tests take too long to run • commit stage takes too long to run (ideally under 5 minutes, more than 10 minutes unacceptable) • automated tests fail intermittently, giving false positives • nobody empowered to revert check-ins • not enough people understand and can change the CI process
  • 20. Poor configuration management - symptoms Problem - environments can’t be commissioned, and applications installed reliably, using an automated process Symptoms • mysterious failures in production environments • new deployments are tense, scary events • large teams dedicated to environment configuration and management • deployments to production often have to be rolled back or patched • unacceptable downtime of production environment
  • 21. Poor configuration management - causes • UAT and production environments are different • poor or badly enforced change management process for making changes to production and staging environments • insufficient collaboration between operations, data management and delivery teams • ineffective monitoring of production and staging environments to detect incidents • insufficient instrumentation and logging built into applications • insufficient testing of the non-functional requirements of applications