Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos

Editor's Notes

1. Hi, My name is Gilad and along here with is Kiril and we are a part of Vmware’s CPSBU or Cloud provider software business unit which a fancy way of saying the we build software for cloud providers.
2. Vmware is transitioning from a product based company to a services based company. More and more teams are developing services, and need to interact with internal backoffice system such as identity and billing. As development moved forward, we’ve noticed two things: No one like to write integrations with billing or identity  developers prefer to write services! Not integrations Every service implements its integrations in its own way, and if different services wants to share this integration, most of the time it’s too domain specific Like all good engineers we want to share code and not waste time on reinventing the wheel. So, our main goal with CSP is to create a platform that will enable acceleration of internal services development and standardize the way a service interacts withthe various intergations
3. How do you design such a platform? When designing CSP we’ve decided on a set of design principles:1. Run on any infrastructure 2. High availability – self explanatory 3. Scalable – support N nodes 4. Public APIs dogfooding – we believe that a good API experience is only achievable when you consume your own APIs 5. Modular – add capabilities to the platform easily and be able to not use certain capabilities 6. Ease of operability / development – try to limit the tech zoo, and be able to run the platform with a single “click” How does it looks in practice? Our lowest common denominator is container support. If a provider can support containers, we can run on it. Our platform is distributed and Stateful. we use tunable consistency in which most of our data is eventually consistent In order to be scalable, we use gossip or to me more precise, SWIM protocol to be highly available No internal APIs, if you don’t have them, you need to consume the public ones Our capabilities or modules are just jars in the class path. Coupling between modules is done at the public API level Our executable is a JAR, not a web / application server which is easy on development and operations. We limited our tech zoo to technologies that are aligned with our design principles. Most of these principles are provided by Vmware’s own Xenon framework, a distributed control plane. More on xenon in a few seconds. When we sticked to our guns with the design princples (and it wasn’t easy) we had a big win:
4. When deployed in production, CSP looks like this. (also in Dev) the number of nodes can scale. A lot.How did we achieve this? Vmware’s xenon framework
5. So how do we upgrade our customer envs? Upgrading services to a new version is not a new concept, All of us are familiar with the popular strategies Rolling upgrade inside an existing cluster Blue/Green Even hybrid solutions exists We had two main goals when designing the upgrade mechanism, other than the oblivious one of actually upgrading the code base: We must support schema transformation (renaming of fields) since adding or subtracting fields is free in Xenon. The other goal is that the customer should not feel service interruptions CSP has some challenges that needed to be addressed when we designed our upgrade mechanism: CSP is stateful and the state and the business logic reside together in the same tier. This causes a challenge when considering a rolling upgrade. You can’t seprate the schema changes and the business logic changes since they both reside in the same jar.And you you can’t modify API paths and or logic since our cluster is symmetrical. So what did we do?
6. Since rolling upgrades are not easily achievable for now, we went with a green / blue strategy. Our goal here is to migrate most of the data while the platform is live. Once the migration is almost done, we queue the incoming traffic,copy the remaining data, and then reroute the traffic to the new cluster. In order to achieve that, we run in cycles. When a cycle is finished, we examine its telemetry and pass it to a threshold mechanism. The threshold’s mechanism purpose it to determine whether it is safe to queue the external traffic and migrate the remaining data.If the last cycle took too long, we start a new cycle picking up from where the last cycle finished in terms of state. (the platform is still live so data is modified in runtime and we need to address these changes) So, we migrate, check and repeat until we’ve crossed a certain threshold. Once the threshold is crossed we queue the traffic, perform a final Cycle and reroute the traffic.Let’s see an example.
7. What is Xenon? Xenon is a framework for writing small REST-based services. (Some people call them microservices.) The runtime is implemented in Java and acts as the host for the lightweight, asynchronous services. The programming model is language agnostic (does not rely on Java specific constructs) so implementations in other languages are encouraged. The services can run on a set of distributed nodes. Xenon provides replication, synchronization, ordering, and consistency for the state of the services. Because of the distributed nature of Xenon, the services scale well and highly available. Xenon is a "batteries included" framework. Unlike some frameworks that provides just consistent data replication or just a microservice framework, Xenon provides both. Xenon services have REST-based APIs and are backed by a consistent, replicated document store.
8. When you build a modern service today you’ll probably need the following checklist: Orchestration code and container – you’ll probably go with Spring Boot HA Distributed DB – Cassandra / Mongo And an ORM layer to go with it A way to keep your cluster in sync – Zookeeper / ETCD UI serving – Node.js / Apache You’ll want to go Stateful at one point for performance / throughput issues – Redis And some message bus / pipeline – Kafka? In my opinion, this checklist looks good . All of the techs listed here work. Industry standard. But, you have to admit, it is a bit complex to manage and deploy.You have to deploy and bootstrap in a certain order, wait for things to get settled in…. You get it. But, there’s another way:
9. Each Xenon runtime provides the following abilities: An Orchestration and a restful layer Persistency and replication layer Total asyc processing with a single thread per core Tunable consistency per service with leader election Publish / Subscribe mechansims And UI sevices, telemetry data, tracing and more….