Migrating to a bazel based CI system: 6 learnings
•
1 like•202 views
Two years ago, we were given a big challenge - Transform Wix Build System, then based on Maven and Teamcity, to a new system that will support our exponentially growing scale. But, how could we move to a system so different in so many ways than the existing one? Furthermore, we were required not to break the current build system, as we migrate to the new one. Fast forward to today: Wix backend CI system is fully migrated to Bazel! The system builds in a fracture of the time - even with our largest codebases. In this talk, we will describe how we achieved this, why it took us so long, what tools we had to build on the way (and what we already have, and will, open source!), and share the principles that helped us.
Report
Share
Migrating to a bazel based CI system: 6 learnings
- 1. Or Shachar, Backend Engineer, Wix-CI Migrating to a Bazel-based CI system: 6 learnings ors@wix.com linkedin/or-shachar github.com/or-shachar
- 2. Check emailCoffee break Ping pong Reading news CAll your mom <Pending mode>
- 3. Check emailCoffee break Ping pong Reading news CAll your mom Building...
- 4. Hi, I'm Or. CI Build with Ant and Makefiles Maven CI Pipelines Backend Engineering … Backend Engineering for2017 2011 2014 2019
- 6. Codebase + build definition Build server feedback Build tool deployables
- 7. ~5M LOC > 1000 repos 2000+ Maven modules 2017 Teamcity Maven feedback deployables
- 8. ~5M LOC > 1000 repos 2000+ Maven modules 2017 Teamcity Maven feedback deployables building / broken blockedblocked blocked blocked
- 9. 2017 Master stability Push to feedback Push to release Often broken Can take 1 hour Can take hours
- 11. Find new machinery that’s stable, fast, and can scale.
- 12. Master stability Push to feedback Push to release 2019 Often broken Can take 1 hour Can take hours 2017 Master stability Push to feedback Push to release Mostly stable 5-10 minutes 15-25 minutes
- 13. 6 LEARNINGS FROM MIGRATING OUR BUILD SYSTEM TO BAZEL.
- 14. #1 Use a Migration tool Massage it, validate it.
- 15. Migration 101 Bazel>>Maven >>
- 16. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Bazel>>Maven >>
- 17. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
- 18. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
- 19. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity
- 20. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity 1:1:1 idiom - one target per one directory, representing a single package
- 21. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules java_library java_library java_test java_image
- 22. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> internal deps java_library java_library java_test java_image
- 23. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary @junit @jetty @guava20 Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> external deps java_library java_library java_test java_image @guava28
- 24. . ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] └── FooServer.java java_library java_library java_test java_binary @junit @jetty Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> external deps java_library java_library java_test java_image @guava28
- 25. . [WORKSPACE.bazel] ├── foo-core [pom.xml] │ └── src │ ├── main │ │ └── java │ │ ├── [com.wix.foo.api] [BUILD.bazel] │ │ │ └── FooBar.java │ │ └── [com.wix.foo.utils] [BUILD.bazel] │ │ ├── TimeUtils.java │ │ └── NameUtils.java │ └── test │ └── java │ └── [com.wix.foo.utils] [BUILD.bazel] │ ├── TimeUtilsTest.java │ └── NameUtilsTest.java └── foo-server [pom.xml] └── src └── main └── java └── [com.wix.foo.server] [BUILD.bazel] └── FooServer.java Migration 101 Maven Bazel>> >> Choose Granularity >> Convert to Bazel rules >> Add deps >> Write Bazel files
- 27. Code graph >> Targets graph Third party definitions >> >> ✓ wix/exodus >>
- 28. Automatic tooling is not enough >> >> ! bazel build bazel test migration trustees
- 29. ✖ Missing compile/runtime deps ✖ Third party alignment ✖ Trying to read from ./src/test/resources ✖ Tests that try to download during test runtime ✖ More... TECHNICAL DEBT == MORE WORK exodus/troubleshooting
- 30. >> >> >> ?! >> bazel build bazel test Did I lose anyting?
- 31. >> << Test reports Test reports compare! >> >> >> ?! >>
- 32. >> >> >> ?! >> >> ✓ >> Exodus Overrides
- 33. 1 Use a migration tool
- 34. #2 Measure your system but measure right.
- 35. 2 3 4 5 1 2 3 4 1 2 3 1 2 7 0 1 0 1 WIX FRAMEWORK (2017) Lines of code 2 5 0 7 Maven modules 3 5 2 3 Bazel Targets 2 5
- 36. 89% reduction Correct Build Times in Minutes * 8-cores machine
- 37. 89% reduction Not Good Enough Correct Build Times in Minutes * 8-cores machine
- 38. Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
- 39. Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
- 40. Push-to-deployable times per commit Stability rate "Super-Artifact" CI Quality Board
- 41. 1 Use a migration tool 2 Measure the system
- 42. #3 Local dev experience don’t forget your users!
- 43. CI & Console runs 🤝 Build Tool
- 44. ✖ Dependency management ✖ Code completion ✖ Refactoring ✖ MacOS cache issues Developers and IDEs 🤝 Build Tool
- 45. Local-dev velocity matters! Bad CILocal dev CIBad Local dev
- 46. ✓ Bazel training ✓ IDE issues ✓ Develop solutions! Think local development! CIOkay Local dev Wix Bazel Local dev team: ○Tooling alignment ○Dev Cache ○Wix IntelliJ Bazel Plugin
- 47. 1 Use a migration tool 2 Measure the system 3 Think local dev experience
- 48. #4 Introduce a new CI pipline but keep the old one too.
- 49. Code push Trigger Artifact Deployment backoffice inform GA/Testbed deployment CI Pipeline Production Server
- 50. Code push Trigger Artifact Deployment backoffice Inform GA/Testbed Deployment CI Pipeline Production Server Replace This part
- 51. ✓ Remote cache and remote execution ✓ PR prechecks (no push to master) ✓ Automatic discovery of deployables ✓ Cloud based build server - no build queues New Bazel CI Pipeline
- 52. Inform Code push Trigger Artifact Deployment backoffice GA/Testbed Deployment CI Pipeline Production Server Trigger Bazel Artifact
- 53. Bazel Maven ✓ Two CI pipelines, side by side Self-service switch Bazel Maven✓
- 54. 1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines
- 55. #5 Optimize for Speed e.g., remote execution cache
- 56. How to make faster bazel builds?
- 57. ✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput 5-8 minutes - fully cached build
- 58. ✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput ✓ Repository cache - Saves downloading http binaries ✓ Docker cache - Saves downloading docker images
- 59. ✓ Remote cache - saves ~90% of execution ✓ Remote execution - improves throughput ✓ Repository cache - Saves downloading http binaries ✓ Docker cache - Saves downloading docker images ✓ Bazel flags - builds without the bytes, multithread digest...
- 60. When we started we compiled Bazel from sources Bazel keeps evolving Each release brought a lot of value, but also required us to adjust Tap to important issues, influence the priorities Become a Bazel activist!
- 61. 1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed
- 62. #6 Migrating poly repo is hard but possible!
- 63. Bazel works well with mono repo All code in one place Single bazel execution ✓ Single CI job
- 64. It's too much for me! I'm not ready yet Mono-repo on Git You change too much! Mono RepoMercurial Poly-Repo on Git ✓
- 65. from 1000+ repos ...to just about 50.
- 66. Goal: Virtual mono repository java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@framework//src/main/java/com/wix/external", ], ) git_repository( name = "framework", url = "git@github.com:wix/framework.git", commit = "<auto-updated>", )
- 67. Virtual mono-repo: Virtual Mono-Repo Service holds all repos of wix with their latest HEAD version Bazel wrapper generates a local bzl file with `git_repository` definition of all repositories Each WORKSPACE file loads the definition from the generated bzl file. framework(031d53ce983d26a861d053c760190de669aa458d) users(1202d41c7dc0739deddde3176c7de34ed9b978b7) events(ab6d1d07477e3742fbc9f9c1e81e0ec402650b1b) external_repos.bzl git_repository( name = "framework", Commit = "031d53ce983d2 ) ... WORKSPACE.bazel 1 2 3
- 68. ~50 migrations
- 70. External Dependencies Guava:25 Commons:7 Junit:3.2 wix-framework:1.0-SNAPSHOT wix-utils:1.0.SNAPSHOT ... java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], )
- 71. Migration in "Isolation Mode" ✓ Faster move to Bazel CI ✓ Faster feedback ✓ Faster deployables ✖ Keep Maven builds green ✖ Reproducibility hit ✖ Stale SNAPSHOT locally
- 72. Isolation Mode Social Mode java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) scala_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@framework//src/java/com/wixpress/lib", ], )
- 73. ✖ Complex ✖ Non-revertible java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@framework//src/java/com/wixpress/lib", ], )
- 74. java_library( name = "core", srcs = [":sources"], deps = [ "//src/main/java/com/wix/local", "@com_google_guava", "@com_wix_framework", ], ) external_binary( name = "com_wix_framework", artifact = "com.wix.framework:1.0-SNAPSHOT" ) external_source( name = "com_wix_framework", exports = [ "@framework//src/java/com/wixpress/lib" ] ) workspace_rule
- 75. external_binary external_source @com_wix_framework Current repo in social mode? Requested module in social mode? no no yes yes
- 76. Social mode ✓ Global Third party alignment ✓ Missing runtime deps ✓ Testonly targets ✓ ...
- 77. 1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed 6 Migrate poly-repo
- 78. Where are we now?
- 79. Safe haven CLOSED ✓ No more SNAPSHOT consumption ✓ All deployable from bazel ✓ All developers use bazel only
- 80. 1 Use a migration tool 2 Measure the system 3 Think local dev experience 4 Run parallel CI pipelines 5 Optimize for speed 6 Migrate poly-repo
- 81. Thank You ors@wix.com 6 LEARNINGS FROM MIGRATING OUR BUILD SYSTEM TO BAZEL
- 82. Q&A ors@wix.com