SlideShare a Scribd company logo

CI/CD: Lessons from LinkedIn and Mockito

•
•
C4Media
C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2Gtedjh. Szczepan Faber talks about two different use cases of implementing continuous delivery at scale: LinkedIn and Mockito. Yet the challenges, benefits and impact on the engineering culture are very similar. Filmed at qconsf.com. Szczepan Faber is a Tech Lead for LinkedIn Development Tools, responsible for developer productivity at LinkedIn. Previously, he was core engineer of Gradle 1.x and 2.x. and instructed numerous classes on build automation. He created Mockito framework in 2007, currently estimated user base of 2M, and has been giving classes on automated testing since.

Related slideshows

Pair Programming (2015)
Pair Programming (2015)Pair Programming (2015)
Pair Programming (2015)
 
IFG for SAP Integration, webinar on Automated Testing
IFG for SAP Integration, webinar on Automated TestingIFG for SAP Integration, webinar on Automated Testing
IFG for SAP Integration, webinar on Automated Testing
 
Trunk based development
Trunk based developmentTrunk based development
Trunk based development
 
1 of 30
@LinkedInEng | @mockitoguy CI/CD at scale Lessons from LinkedIn and Mockito QCon, San Francisco, 11-2017 ● Szczepan Faber @mockitoguy ● Born in Poland (we like our zzz’s) In US since 2015 ● Codes professionally since 2002 ● Creator of mockito.org in 2007 ● Core dev of gradle.org 1.x and 2.x in 2011-2015 ● Tech Lead at LinkedIn Dev Tools since 2015 http://bit.do/li-tools ● Author on LinkedIn: http://bit.do/mockitoguy ● How to build great code review culture? http://bit.do/li-code-review ● Want to write great tests? My workshop at QCon on Thursday: http://bit.do/qcon-testing ● Want to innovate and push CD in Open Source? Join shipkit.org New project used by Mockito!
InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ ci-cd-linkedin-mockito
Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
@LinkedInEng | @mockitoguy Imagine productivity without the release overhead
@LinkedInEng | @mockitoguy 2. CD in OSS Mockito (@MockitoJava) ● I created mocking framework Mockito in 2007 ● The team adopted CD in 2014 ● We ship every pull request to production to estimated 2M users Get ready to be excited about CD! Development Tools ● Build tools, Gradle, CI ● Code review, IDEs ● and more! ● I’m an architect, tech lead, engineer working on development tools at LinkedIn since 2015. 1. CD at LinkedIn (@LinkedInEng) ● linkedin.com adopted CD in 2015, shipping 3x day to 500M users. LinkedIn Engineering (3000+) Foundation team (300+) ● Kudos to great engineers working at LinkedIn, linkedin.com and the tools. I am a part of a team.
@LinkedInEng | @mockitoguy CD at linkedin.com by the numbers ○ 1000 commits, 300 unique committers across 4 main codebases ○ Web app, API server app, Android app, iOS app ○ The stats exclude other codebases (libraries, downstream microservices) ○ LinkedIn does not use (nor wants or needs) mono-repo. We currently have 7000+ codebases. ● Last quarter (Q3 ‘17): ○ 300 pushes to production (web app + api server) ○ 21 mobile app pushes (android + iOS, excluding Beta) ● LinkedIn engineering is more than linkedin.com ○ Other products exercise CD as well but don’t have such big codebase or number of committers ○ linkedin.com (flagship) is very progressive and paves the way for other LinkedIn software products ● Last week (5-11 Nov ‘17)
Part 1. CD is hard What do we do if something is hard?
@LinkedInEng | @mockitoguy Why CD in linkedin.com? ● reduce the lead time to make positive business impact ○ ship features to production faster ● increase engineering productivity and happiness ○ avoid release overhead ■ avoid wasted time on stabilizing (bugfixing) the release branch. Code should be always stable and ready to ship! ■ avoid wasted time on cherry-picking. Trunk based development. ○ avoid feature rush - last minute commit volume spike before the release ● improve quality ○ small incremental releases pose small risk and are easier to fix ○ avoid rollbacks and hotfixes. Instead we fix forward The arguments apply to every software project! In 2015 we launched 3x3 project - 3 releases per day, 3 hour max time commit-to-production. Our goals:
@LinkedInEng | @mockitoguy 3x3 @LI ○ Because we want to iterate fast. Plus it helps with resilience because we can afford to miss a release ● Why 3 hours max time commit-to-production? ○ Because to ship 3 times a day, the commit-to-production pipeline needs speed ○ And it forces us to sort out our testing strategy (not enough time for manual or slow tests) ● In 2015 we completed mobile web and mobile apps ● In 2016 we completed desktop web ● Today we release linkedin.com several times a day ● LinkedIn Mobile apps are released every week, with 3 beta releases per week (iOS Beta - once a week) ● Why 3 times per day?
@LinkedInEng | @mockitoguy ● Learning how to write great tests - what to test, how to test, how to write code that is easy to test ● Flaky test is worse than no test - detecting flaky tests automatically, overnight, using A/A testing ● Production grade tests and infrastructure - in the past, tests and build code were not considered equally important as production code ● Many existing tools, including OSS did not scale for CD ● Need for speed – testing in parallel and distributed ● Master branch always green - Running all validations before code is merged to master CD lessons @LI
@LinkedInEng | @mockitoguy Current 3x3 stats for linkedin.com Commits and unique committers in last week Avg. time commit-to-shippable in last week # of actual and target releases in Q3 2017 (*) in actuality, it is higher because commits to libraries are not included Web 317/115 150 min. 173/162 API 183/77 69 min. 123/162 iOS 164/69 165 min. 10/12 Android 241/72 72 min. 11/12 Combined 905*/333
@LinkedInEng | @mockitoguy Opportunities @LI ● Increase discipline of fixing flaky tests ● Avoid redundant work in the pipeline ● For Android: simulator service, speed ● For iOS: Beta channel, Swift compiler stability and speed ● For Web app: browser cache ● For API server: avoid redundant work (Gradle Distributed Cache) BTW. we use Play on Gradle for our API server: http://bit.do/play-on-gradle ● Minimize commit-to-shippable time ● Increase commit-to-shippable pipeline success rate (pipeline & tests stability) ● Hit the desired number of production releases And many more! We keep improving the system!!!
@LinkedInEng | @mockitoguy 3x3 resources LinkedIn Engineering Blog: http://bit.do/li-3x3
@LinkedInEng | @mockitoguy Development Workflow @LI ● Multi-codebase architecture, 7000 codebases, 60% active ● Every codebase is governed by our “Multiproduct” framework ● Every Multiproduct has independent release cadence ● Every code change produces new version ● Trunk based development ● Mandatory code review (code owner must approve every change) ● Automation of commit-to-production pipeline CD requires disciplined development workflow. How software is developed at LinkedIn? (all our software, not only linkedin.com)
@LinkedInEng | @mockitoguy Every codebase is a Multiproduct @LI ● One engineering culture ● Every engineer can contribute to any codebase ● Why matters? Easier to introduce CD into one culture
@LinkedInEng | @mockitoguy Every change is a new version @LI ● And every new version can be shipped to production ● Why matters? ○ Makes it impossible to defer quality. ○ Clean code every day!
@LinkedInEng | @mockitoguy Trunk based development @LI ● Incremental code changes, hiding incomplete features ● Feature toggles, “branch by abstraction” pattern http://bit.do/branch-by-abstraction ● Why matters? Forces small, incremental changes. Avoids merge and cherry-picking overhead. ● No long-lived feature branches ● All changes on main branch, which is always stable and ready to ship
@LinkedInEng | @mockitoguy Mandatory code review @LI ● Somebody reads my code (and wants me to fix it)! ● Culture of feedback, learning and improving http://bit.do/li-code-review ● Why matters? Clean, elegant code makes it easier to iterate
@LinkedInEng | @mockitoguy Downstream dependency testing @LI ● Building code that depends on my code ● Strong signal in our CI pipeline ● Flaky tests are a problem to the entire ecosystem ● Why matters? avoiding regressions, catching integration problems early
@LinkedInEng | @mockitoguy Automation of commit-to-production @LI ● The code I write or review goes to production within hours ● I am responsible for the quality ● Why it matters? Requires state of the art automated testing ○ BTW. I run a Java testing workshop on Thursday at QCon http://bit.do/qcon-testing
@LinkedInEng | @mockitoguy CI/CD pipeline @LI For all our software, not only components of linkedin.com ● Code change ● Code review (strong code ownership, owner must approve) ● Pre/post push validation (CI builds) ● Downstream dependency testing (test code that depend on me) ● New version ready! ● Staging ● Canary (deploy and test on single host) ○ Mobile uses Beta channel ● Ramp-up features (feature toggles) ○ Code push != feature push ● Remove feature toggle (dead code) ● Consumers that use wildcard versions pick up new version in the next build. ● Consumers that use pinned version can be updated automatically using “Push my Upgrade” system. ● We can deprecate/end-of-life previous versions Apps Libraries Dev workflow for all software at LinkedIn. Ready for part 2 (OSS)?
@mockitoguy Part 2. CD. in OSS Mockito Powered by shipkit.org
@mockitoguy Open Source Mocking Framework for Java ● Mockito started in 2007 ● 1.0 in 2008 ● Hit mainstream in 2010 ● Mostly manual releases...
Combined result puts Mockito on 3rd place!
@mockitoguy 2014 - Mockito adopts CD. ● Every merged pull request produces a new version and ships to public repo ● Scale: we estimate 2M users ● Prevent release procrastination (dreading to write release notes…)
@mockitoguy Benefits of CD in OSS ● Productivity – zero release overhead ● Happy users – get features faster ● Faster debugging – quickly identify bad version (MTTR) ● Sustainability – release & stay alive ● No waste - no unreleased code ● Quality – self-enforced craftsmanship of every change ● Thriving, engaged community – contributions are released quickly
@mockitoguy Community Feedback ● Quality anxiety - you ship every pull request to production, are you shipping every bug, too? - we ensure quality via immense battery of tests and rigorous code review ● What version of “mockito-core” to use? - currently 244 versions in Maven Central - use latest! We take compatibility VERY seriously. Sem ver! ● Dependency management cost - worry that any version upgrade may bring incompatibilities to the dependency graph - we get it. We strive to minimize Mockito dependencies. We understand that every dependency is a liability to our customers
@mockitoguy Mockito releases By Shipkit ● Shipkit - toolkit for shipping it for Java libraries ● Passionate about release automation? ● http://shipkit.org
@LinkedInEng | @mockitoguy Ready for CD? ● Imagine how fast you can ship changes that can create positive business impact ● Imagine unhindered productivity without the release overhead ● Imagine higher quality because smaller, incremental releases are a smaller risk ● Imagine that every code change is excellent, with clean code and great tests ● Imagine how reliable the commit-to-production pipeline is if it is battle tested daily ● At linkedin.com we land 1000 commits per week and ship to production several times a day to 500M+ users. ● In the Open Source, Mockito library ships to production every pull request to estimated 2M users Now it is your turn! Questions?
Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ci-cd- linkedin-mockito

More Related Content

CI/CD: Lessons from LinkedIn and Mockito

  • 1. @LinkedInEng | @mockitoguy CI/CD at scale Lessons from LinkedIn and Mockito QCon, San Francisco, 11-2017 ● Szczepan Faber @mockitoguy ● Born in Poland (we like our zzz’s) In US since 2015 ● Codes professionally since 2002 ● Creator of mockito.org in 2007 ● Core dev of gradle.org 1.x and 2.x in 2011-2015 ● Tech Lead at LinkedIn Dev Tools since 2015 http://bit.do/li-tools ● Author on LinkedIn: http://bit.do/mockitoguy ● How to build great code review culture? http://bit.do/li-code-review ● Want to write great tests? My workshop at QCon on Thursday: http://bit.do/qcon-testing ● Want to innovate and push CD in Open Source? Join shipkit.org New project used by Mockito!
  • 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ ci-cd-linkedin-mockito
  • 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  • 5. @LinkedInEng | @mockitoguy 2. CD in OSS Mockito (@MockitoJava) ● I created mocking framework Mockito in 2007 ● The team adopted CD in 2014 ● We ship every pull request to production to estimated 2M users Get ready to be excited about CD! Development Tools ● Build tools, Gradle, CI ● Code review, IDEs ● and more! ● I’m an architect, tech lead, engineer working on development tools at LinkedIn since 2015. 1. CD at LinkedIn (@LinkedInEng) ● linkedin.com adopted CD in 2015, shipping 3x day to 500M users. LinkedIn Engineering (3000+) Foundation team (300+) ● Kudos to great engineers working at LinkedIn, linkedin.com and the tools. I am a part of a team.
  • 6. @LinkedInEng | @mockitoguy CD at linkedin.com by the numbers ○ 1000 commits, 300 unique committers across 4 main codebases ○ Web app, API server app, Android app, iOS app ○ The stats exclude other codebases (libraries, downstream microservices) ○ LinkedIn does not use (nor wants or needs) mono-repo. We currently have 7000+ codebases. ● Last quarter (Q3 ‘17): ○ 300 pushes to production (web app + api server) ○ 21 mobile app pushes (android + iOS, excluding Beta) ● LinkedIn engineering is more than linkedin.com ○ Other products exercise CD as well but don’t have such big codebase or number of committers ○ linkedin.com (flagship) is very progressive and paves the way for other LinkedIn software products ● Last week (5-11 Nov ‘17)
  • 7. Part 1. CD is hard What do we do if something is hard?
  • 8. @LinkedInEng | @mockitoguy Why CD in linkedin.com? ● reduce the lead time to make positive business impact ○ ship features to production faster ● increase engineering productivity and happiness ○ avoid release overhead ■ avoid wasted time on stabilizing (bugfixing) the release branch. Code should be always stable and ready to ship! ■ avoid wasted time on cherry-picking. Trunk based development. ○ avoid feature rush - last minute commit volume spike before the release ● improve quality ○ small incremental releases pose small risk and are easier to fix ○ avoid rollbacks and hotfixes. Instead we fix forward The arguments apply to every software project! In 2015 we launched 3x3 project - 3 releases per day, 3 hour max time commit-to-production. Our goals:
  • 9. @LinkedInEng | @mockitoguy 3x3 @LI ○ Because we want to iterate fast. Plus it helps with resilience because we can afford to miss a release ● Why 3 hours max time commit-to-production? ○ Because to ship 3 times a day, the commit-to-production pipeline needs speed ○ And it forces us to sort out our testing strategy (not enough time for manual or slow tests) ● In 2015 we completed mobile web and mobile apps ● In 2016 we completed desktop web ● Today we release linkedin.com several times a day ● LinkedIn Mobile apps are released every week, with 3 beta releases per week (iOS Beta - once a week) ● Why 3 times per day?
  • 10. @LinkedInEng | @mockitoguy ● Learning how to write great tests - what to test, how to test, how to write code that is easy to test ● Flaky test is worse than no test - detecting flaky tests automatically, overnight, using A/A testing ● Production grade tests and infrastructure - in the past, tests and build code were not considered equally important as production code ● Many existing tools, including OSS did not scale for CD ● Need for speed – testing in parallel and distributed ● Master branch always green - Running all validations before code is merged to master CD lessons @LI
  • 11. @LinkedInEng | @mockitoguy Current 3x3 stats for linkedin.com Commits and unique committers in last week Avg. time commit-to-shippable in last week # of actual and target releases in Q3 2017 (*) in actuality, it is higher because commits to libraries are not included Web 317/115 150 min. 173/162 API 183/77 69 min. 123/162 iOS 164/69 165 min. 10/12 Android 241/72 72 min. 11/12 Combined 905*/333
  • 12. @LinkedInEng | @mockitoguy Opportunities @LI ● Increase discipline of fixing flaky tests ● Avoid redundant work in the pipeline ● For Android: simulator service, speed ● For iOS: Beta channel, Swift compiler stability and speed ● For Web app: browser cache ● For API server: avoid redundant work (Gradle Distributed Cache) BTW. we use Play on Gradle for our API server: http://bit.do/play-on-gradle ● Minimize commit-to-shippable time ● Increase commit-to-shippable pipeline success rate (pipeline & tests stability) ● Hit the desired number of production releases And many more! We keep improving the system!!!
  • 13. @LinkedInEng | @mockitoguy 3x3 resources LinkedIn Engineering Blog: http://bit.do/li-3x3
  • 14. @LinkedInEng | @mockitoguy Development Workflow @LI ● Multi-codebase architecture, 7000 codebases, 60% active ● Every codebase is governed by our “Multiproduct” framework ● Every Multiproduct has independent release cadence ● Every code change produces new version ● Trunk based development ● Mandatory code review (code owner must approve every change) ● Automation of commit-to-production pipeline CD requires disciplined development workflow. How software is developed at LinkedIn? (all our software, not only linkedin.com)
  • 15. @LinkedInEng | @mockitoguy Every codebase is a Multiproduct @LI ● One engineering culture ● Every engineer can contribute to any codebase ● Why matters? Easier to introduce CD into one culture
  • 16. @LinkedInEng | @mockitoguy Every change is a new version @LI ● And every new version can be shipped to production ● Why matters? ○ Makes it impossible to defer quality. ○ Clean code every day!
  • 17. @LinkedInEng | @mockitoguy Trunk based development @LI ● Incremental code changes, hiding incomplete features ● Feature toggles, “branch by abstraction” pattern http://bit.do/branch-by-abstraction ● Why matters? Forces small, incremental changes. Avoids merge and cherry-picking overhead. ● No long-lived feature branches ● All changes on main branch, which is always stable and ready to ship
  • 18. @LinkedInEng | @mockitoguy Mandatory code review @LI ● Somebody reads my code (and wants me to fix it)! ● Culture of feedback, learning and improving http://bit.do/li-code-review ● Why matters? Clean, elegant code makes it easier to iterate
  • 19. @LinkedInEng | @mockitoguy Downstream dependency testing @LI ● Building code that depends on my code ● Strong signal in our CI pipeline ● Flaky tests are a problem to the entire ecosystem ● Why matters? avoiding regressions, catching integration problems early
  • 20. @LinkedInEng | @mockitoguy Automation of commit-to-production @LI ● The code I write or review goes to production within hours ● I am responsible for the quality ● Why it matters? Requires state of the art automated testing ○ BTW. I run a Java testing workshop on Thursday at QCon http://bit.do/qcon-testing
  • 21. @LinkedInEng | @mockitoguy CI/CD pipeline @LI For all our software, not only components of linkedin.com ● Code change ● Code review (strong code ownership, owner must approve) ● Pre/post push validation (CI builds) ● Downstream dependency testing (test code that depend on me) ● New version ready! ● Staging ● Canary (deploy and test on single host) ○ Mobile uses Beta channel ● Ramp-up features (feature toggles) ○ Code push != feature push ● Remove feature toggle (dead code) ● Consumers that use wildcard versions pick up new version in the next build. ● Consumers that use pinned version can be updated automatically using “Push my Upgrade” system. ● We can deprecate/end-of-life previous versions Apps Libraries Dev workflow for all software at LinkedIn. Ready for part 2 (OSS)?
  • 22. @mockitoguy Part 2. CD. in OSS Mockito Powered by shipkit.org
  • 23. @mockitoguy Open Source Mocking Framework for Java ● Mockito started in 2007 ● 1.0 in 2008 ● Hit mainstream in 2010 ● Mostly manual releases...
  • 24. Combined result puts Mockito on 3rd place!
  • 25. @mockitoguy 2014 - Mockito adopts CD. ● Every merged pull request produces a new version and ships to public repo ● Scale: we estimate 2M users ● Prevent release procrastination (dreading to write release notes…)
  • 26. @mockitoguy Benefits of CD in OSS ● Productivity – zero release overhead ● Happy users – get features faster ● Faster debugging – quickly identify bad version (MTTR) ● Sustainability – release & stay alive ● No waste - no unreleased code ● Quality – self-enforced craftsmanship of every change ● Thriving, engaged community – contributions are released quickly
  • 27. @mockitoguy Community Feedback ● Quality anxiety - you ship every pull request to production, are you shipping every bug, too? - we ensure quality via immense battery of tests and rigorous code review ● What version of “mockito-core” to use? - currently 244 versions in Maven Central - use latest! We take compatibility VERY seriously. Sem ver! ● Dependency management cost - worry that any version upgrade may bring incompatibilities to the dependency graph - we get it. We strive to minimize Mockito dependencies. We understand that every dependency is a liability to our customers
  • 28. @mockitoguy Mockito releases By Shipkit ● Shipkit - toolkit for shipping it for Java libraries ● Passionate about release automation? ● http://shipkit.org
  • 29. @LinkedInEng | @mockitoguy Ready for CD? ● Imagine how fast you can ship changes that can create positive business impact ● Imagine unhindered productivity without the release overhead ● Imagine higher quality because smaller, incremental releases are a smaller risk ● Imagine that every code change is excellent, with clean code and great tests ● Imagine how reliable the commit-to-production pipeline is if it is battle tested daily ● At linkedin.com we land 1000 commits per week and ship to production several times a day to 500M+ users. ● In the Open Source, Mockito library ships to production every pull request to estimated 2M users Now it is your turn! Questions?
  • 30. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ci-cd- linkedin-mockito