SlideShare a Scribd company logo

Retrofitting Continuous Delivery

•Download as PPTX, PDF•
•
Alan Norton
Alan Norton

This document describes Alan's journey to implement continuous delivery practices at Betterment through a DevOps transformation in three phases: Phase 1 focused on fixing issues with their initial infrastructure including slow builds, lack of automation, and fragile deployments. Phase 2 added practices like infrastructure as code and continuous integration to automate workflows. Phase 3 aims to scale innovation by having product teams adopt DevOps practices from the start so new features are delivered continuously without slowing down development. The transformation progressed from fixing problems to empowering teams to move fast through automated delivery pipelines.

1 of 60
Retrofitting Continuous Delivery A DevOps Tale
Hi. I’m Alan.
Retrofitting Continuous Delivery
Retrofitting Continuous Delivery
Retrofitting Continuous Delivery
betterment : investing :: devops : engineering
Phase 1: Snowflakes
Retrofitting Continuous Delivery
Betterment Core Applications “brochure” wordpress apache php mysql batch (trading) cron java spring hibernate mysql webapp apache tomcat java spring hibernate mysql
Rackspace Loadbalancer pre-prod apache tomcat mysql hudson jira confluence “warehouse” sendmail file locker backup
Look at the Mess You’ve Made ● Overburdened, inflexible infrastructure from Rackspace ● Only one pre-prod environment ○ And it runs five other things including Jira/Confluence/etc. ● Building and deploying artifacts manually, ad-hoc, as needed ○ “Why is your trunk .war 23 bytes larger than mine?” ● No Automated Functional Testing ● Deployments require downtime. ● Deployments happen at 3am ET ● Manual Provisioning on persistent servers ○ “This wordpress plugin needs php5.3 but we have 5.1” ● Branching (and merging) with Subversion requires significant effort ● …
yuck. so what’s the good news?
we shipped a ton of functionality.
delivery speed faster than innovation speed
Phase 2: Addicted to Automation
stop building features. fix what’s painful.
DevOp s
fix onboarding.
boxen. github // @jbarnette + @wfarr
mac osx + homebrew + puppet + facter
github.com/boxen/puppet-*
Retrofitting Continuous Delivery
fix tooling.
>
$ time ant clean test deployment 19m 22s $ time gradle clean test intTest war zipJar 2m 36s (~builds/day · Δt · devs · man_days/mo)/min = (6 · 17 · 19 · 20) / 60 = 646 hrs/mo
fix notifications & logging.
Retrofitting Continuous Delivery
Whoa.
fix parallel workflows.
Retrofitting Continuous Delivery
everybody loves jenkins build open pull requests* build develop/stage/master cobertura reports coverage * https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin
Retrofitting Continuous Delivery
fix crowded infrastructure.
AWS + Rackspace (sitting in a tree, N-E-T-W-O-R-K-I-N-G)
Retrofitting Continuous Delivery
<10ms !
“canary” ec2 webapps exposed via route53 weighting
fix provisioning.
Ansible (not manual) AMIs ● Playbooks are YML that execute over SSH ● Playbooks define necessary machine config o Results in AMI creation in AWS o Returns AMI_ID ● Commit playbooks and current AMI_ID ● Jenkins reads AMI_ID from build artifact o Spins-up necessary EC2 instances o Runs deploy playbook, unrelated to provisioning playbook ● If you need to modify a playbook o Run playbook to create new AMI_ID o Commit new playbook and AMI_ID to repo.
no more snowflakes.
everybody loves jenkins again deploys to rackspace deploys to ec2 bash or ansible playbooks
Retrofitting Continuous Delivery
fix database migrations.
Flyway Database Migrations ● Looks a lot like rake db:migrate o core/src/main/resources/db/migrations o core/src/main/resources/db/cleanups ● Kept track of in a schema_versions table. ● Cleanups are promoted to migrations when develop moves to stage. ● Run as part of weekly releases. ● http://flywaydb.org/
Database Subsetting with Jailer ● Walks the constraint graph from the mysql schema. ● Requires some manual intervention to cut off circular or “eager” relations. ● Exports a referentially valid subset of data. o User, Account(s), Trade(s), Etc. ● http://jailer.sourceforge.net/
Users. Duh.
+ = freshdb subsets live in s3. download and run flyway. instant db on any branch.
eos:better-core(develop)$ ./fresh -f Forcing download... Downloading fresh data... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 21.0M 100 21.0M 0 0 1004k 0 0:00:21 0:00:21 --:--:-- 1365k :setupFreshDb :setupFreshDb:complete __ _ / _| |_ ___ ____ _ _ _ | |_| | | | / / / _` | | | | | _| | |_| | V V / (_| | |_| | |_| |_|__, | _/_/ __,_|__, | |___/ by fido |___/ [stderr] INFO AbstractCLI.doMain(287) | cli_started [stderr] INFO FlywayDbMigrationCLI.executeBeforeSpring(164) | jdbc:mysql://127.0.0.1:3306/bettermentdbfresh [stderr] INFO FlywayWrapper.migrateTo(154) | Running ALL pending migrations ... [stderr] INFO ApacheCommonsLog.info(43) | Validated 146 migrations (execution time 00:00.616s) [stderr] INFO ApacheCommonsLog.info(43) | Current version of schema `bettermentdbfresh`: 20140617123653 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132438 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132439 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132440 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132441 [stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140625091709 ...
ah. that’s better. now what?
Phase 3: Innovation That Scales
innovating is easy. demoing is easy.
product teams move faster than infrastructure teams.
jruby on rails factory girl for Java multiple schemas webservices
delivery speed slower than innovation speed
DevOps
DevOps
Solution? Undercover DevOps
every team builds product features with delivery in mind
Questions? first one: can you start at phase 3?
alan@betterment.com github/twitter: @nonrational

More Related Content

Retrofitting Continuous Delivery

Editor's Notes

  1. 3+ years @ Betterment Lead Engineer Built some fun stuff @ Fintech Hackathons Been writing code for a while.
  2. Investing for everyone, if you want to know (or think you know) what you’re doing.
  3. multiple goals, multiple allocations, subsection of your account, all managed with the same finesse and care, whether you have $10 or $200,000.
  4. we show you exactly what’s going on, if you want to see it. automatic rebalancing tax loss harvesting
  5. very, very confusing at times and have the potential to waste a lot of your time before you figure out the right approach the dream of devops is to create an environment where the path of least resistance also yields the most efficient, sane result. betterment is totally onboard with that mission from the ground up.
  6. 2010, Java6, Spring MVC + Hibernate, Adobe Flex, Wordpress, MySQL Deployment was VERY manual and very infrequent Build Locally, Unpack War, Like playing Minecraft Properties Management - Secrets OTB. Sticky session management w/ JSESSIONID Necessitates 2am deployments Product features rule EVERYTHING. A quick look @ betterment archetecture. Deployment Process keeps up with Rate of Change Fourth S: Slow Enough
  7. This is what the site looked like when I signed up for Betterment.
  8. not a pretty picture, in more ways than one…
  9. No Persistent Sessions
  10. Git Flow Predictable Releases Boxen Trello Jenkins + Bash Ansible Brochure AWS AWS RDS RSpec + Capybara WebKit What sort of Innovation(s)? JRuby on Rails + Java Services Jailer + Flyway
  11. Git Flow Predictable Releases Boxen Trello Jenkins + Bash Ansible Brochure AWS AWS RDS RSpec + Capybara WebKit What sort of Innovation(s)? JRuby on Rails + Java Services Jailer + Flyway
  12. assemble a strikeforce / build a team
  13. Build a team to clean up the mess that you made getting the product out there. Make the easy way, the right way.I'd love to have rackspace mount it for us and outline the steps required to do so.
  14. assemble a strikeforce / build a team
  15. assemble a strikeforce / build a team
  16. assemble a strikeforce / build a team
  17. talk about
  18. assemble a strikeforce / build a team
  19. release code reliably, even through trading.
  20. automatically run tests and resolve dependencies every time code changes
  21. assemble a strikeforce / build a team
  22. Development Environments
  23. Development Environments
  24. assemble a strikeforce / build a team
  25. assemble a strikeforce / build a team
  26. assemble a strikeforce / build a team
  27. automatically run tests and resolve dependencies every time code changes
  28. assemble a strikeforce / build a team
  29. can run app against a remote database, but then the process of switching from local to remote is non-trivial. we haven’t gotten the “development on vm” down yet.
  30. central authority / decentralized knowledge. push built products out to the team
  31. lay the railroad track but don’t drive the train.