Paris container day june17
•
0 likes•1,064 views
BlaBlaCar is moving its infrastructure to be fully containerized using rkt and CoreOS. Key tools developed include dgr for building and running containers and ggn for managing services on fleet clusters. Service discovery is handled by go-nerve and go-synapse which monitor services in Zookeeper. The infrastructure aims for standardization, simplicity and removing unique configurations ("snowflakes"). Over 300 servers across multiple data centers are now managed this way.
Report
Share
Paris container day june17
- 1. Take a BlaBlaCar to the stars with rkt ! Simon Lallemand System Engineer @slallema
- 2. New infrastructure ecosystem Agenda Key principles 100% rkt powered carpooling Directions for driving change BlaBlaCar Facts, Figures & History
- 3. Founded in 2006 21 million mobile app downloaded (iPhone + Android) Facts and figures 12 million travelers per quarter 1 million tonnes less CO2 in the past year 40 million members Currently in 22 countries France, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey. CO2
- 4. Our server growth The evolution of the infrastructure from the begining to now: 2006 2008 2010 2012 2013 2014 2015 2016 Web hosting Dedicated servers Baremetal 1 rack Baremetal 3 racks Baremetal 5 racks Baremetal 8 racks Baremetal 14 racks 150 servers 2 DC Baremetal 17 racks 300 servers 3 DC
- 5. ~300 bare-metal servers ~400 container images 4000+ running containers
- 6. Tech evolution The major tech changes in our infrastructure : Virtu 2012 Chef 2013 Foreman 2014 Hardware uniformization 2015 Containers 2015 2016
- 7. Key principles Leading the industrialization at BlaBlaCar
- 8. Metal is invisible Decouple hardware management and what runs on it
- 9. Route everything Keep the network simple and scalable
- 10. Remove snowflakes Be service oriented
- 11. New Infrastructure Ecosystem 100% rkt powered carpooling
- 13. CoreOS Container Linux On 100% of our new servers
- 15. New Infrastructure Ecosystem The tools
- 16. dgr Container build and runtime tool Harmonize the way we build Quick build One way of doing things Easy to understand for newcomers As little code replication as possible Templating at container start A good integration with rkt github.com/blablacar/dgr
- 17. dgr : build directory of an ACI ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl A standardized structure Ease maintenance and teamwork Inspired by config management Separation of templates, attributes and scripts.
- 18. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf Simpler manifest format Fill only what is important YAML <3 One process per ACI Composition is done only with POD
- 19. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf dependencies: - debian:8.6 Use of dependencies Composition Lighter images
- 20. dgr : runlevel build ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh ├── templates │ └── etc │ └── redis │ └── redis.conf.tmpl └── tests Scripts executed in the container Build from inside the container with all the dependencies
- 21. dgr : runlevel build #!/bin/bash apt-get install -y redis-server Install packages and stuff For debian based containers you can install packages using apt-get for instance.
- 22. dgr : runlevel builder name: example.com/aci-redis-dicator:1 builder: dependencies: - example.org/aci-go - example.org/aci-git aci: app: exec: - /bin/dictator Build from outside of the container Dedicated builder image with go & git
- 23. dgr : runlevel builder #!/dgr/bin/busybox sh git clone https://github.com/blablacar/redis-dictator cd redis-dictator go build cp dist/dictator ${ROOTFS}/bin Builder script Clone, build and copy binary to target ACI Only /bin/dictator in final ACI
- 24. dgr : runlevel builder name: example.org/gentoo-redis:1 builder: dependencies: - example.org/gentoo-stage4 aci: dependencies: - example.org/base app: exec: [ /usr/bin/redis-server ] Other example Builder with gentoo’s stage4
- 25. dgr : runlevel builder #!/dgr/bin/busybox sh emerge -v redis-server Install packages from outside Use emerge to install your app and its dependencies. Redis with dependencies only in the final ACI
- 26. dgr : templates & attributes ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl Render configuration files Templates stored in the aci Default attributes stored in the aci Overridable when used as dependencies Overridable by environment var
- 27. dgr : templates & attributes # templates/etc/redis.conf.tmpl daemonize no port {{ .redis.port }} … # attributes/redis.yml default: redis: port: 6379
- 28. dgr : runlevel prestart ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── prestart-late │ └── 10-init-db.sh └── templates └── etc └── redis └── redis.conf.tmpl Initialize container prestart-early and prestart-late scripts Before and after templating Initialize environment before exec
- 29. dgr : testing ├── aci-manifest.yml ├── attributes ├── runlevels ├── templates └── tests └── wait.sh └── my_cool_tests.bats Testing Bats as default tester wait.sh Wait for service to be ready
- 30. dgr : testing #!/dgr/bin/bats -x @test "Redis should be running" { run bash -c "ps -aux | grep redis-server" [ "$status" -eq 0 ] echo ${lines[0]} [[ "${lines[0]}" =~ "/usr/bin/redis-server" ]] } @test "Redis should listen on port: 6379" { run bash -c "netstat -peanut | grep redis-server" [ "$status" -eq 0 ] [[ "${lines[0]}" =~ ":6379" ]] [[ "${lines[0]}" =~ "redis-server" ]] }
- 31. dgr : subcommands ~ # dgr init ~ # dgr try ~ # dgr build ~ # dgr test ~ # dgr install ~ # dgr push github.com/blablacar/dgr
- 32. ggn Manage services in fleet clusters Pods Services and Environments Different services that use the same pods Environments (1 per DC for prod + dev…) Abstraction of fleet commands github.com/blablacar/ggn INSERT LOGO HERE
- 33. ggn ├── prod-dc1 │ └── attributes │ └── services │ └── redis-quota │ │ └── attributes │ │ └── service-manifest.yml │ │ └── unit.tmpl │ └── redis-rating ├── preprod │ ... Single directory tree Describes the services The environments Versioned in git
- 34. ggn : service manifest containers: - blablacar.com/aci-redis:3 - blablacar.com/aci-redis-exporter:1 nodes: - hostname: redis-quota1 - hostname: redis-quota2 - hostname: redis-quota3 Define the service ACI images and versions combined to make a pod Nodes composing the service
- 35. ggn : attributes # attributes/redis.yml override: redis: maxmemory: 4GB TEMPLATER_OVERRIDE={“redis”:{“max memory”: “4GB”}…} Override attributes Inject run context attributes as environment var
- 36. ggn : unit.tmpl [Unit] Description=Redis POD {{. [Service] KillMode=mixed Restart=always ExecStart=/usr/bin/rkt run --set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}' {{.acis}}
- 37. ggn : run a POD $ ggn prod-dc1 redis-quota update $ ggn prod-dc1 redis-quota1 journal -f github.com/blablacar/ggn
- 38. Service discovery with nerve & synapse AirBnB’s smartstack Dynamic topology of services Scalability requires service discovery We started with smartstack of AirBnB Since then we improved it and rewrote it in go github.com/blablacar/go-nerve github.com/blablacar/go-synapse
- 39. /database/node1 go-nerve does health checks and reports to zookeeper in service keys HAProxy node1 Applications hit their local haproxy to access backends Service Discovery /database go-synapse watches zookeeper service keys and reloads haproxy if changes are detected go-nerve Zookeeper go-synapse
- 40. bare-metal servers 1 type of hardware 3 disk profiles fleet machines CoreOS fleet etcd“Distributed init system” Hardware Container Registry ggn dgr Service Codebase rkt PODs build run store host create Our infrastructure ecosystem nerve mysqld monitoring mysql-main_1 nerve php monitoring nginx synapse front_1 synapse nerve zookeeper Service Discovery
- 41. Work In Progress ( We’re hiring )