TechBeats #2
•
0 likes•550 views
Applause's journey from a monolith to a microservice environment running on Mesosphere DC/OS by Christopher Gianelloni
Report
Share
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
More Related Content
What's hot
What's hot (20)
Similar to TechBeats #2
Similar to TechBeats #2 (20)
Recently uploaded
Recently uploaded (20)
TechBeats #2
- 1. Journey from monolith to microservices Utilizing microservice patterns with monoliths Chris Gianelloni @wolf31o2 1
- 2. In the beginning… Applause had several ways to deploy and manage software. • Custom system management tool (sysdeploy) • Basically a SSH wrapper for manually created systems • Custom Docker image management tool (Platypus) • Standardized AMIs, built w/ Packer, including Docker daemon • Services in Docker containers w/ configuration using SaltStack • Provides A/B testing and health checks • Packer + Chef + Terraform • Packer + Chef to bake AMIs • Terraform to deploy using ASGs • Mesosphere DC/OS • OSS orchestration for “Docker” containers 2
- 3. Typical “old school” configuration system, written completely in-house, limited in capabilities, author has long-since departed the company • SSH wrapper to copy files and run commands • No instance management • No user management • No rollback features • No documentation • Unfamiliar code base to everyone • Unable to look up problems sysdeploy 3
- 4. In-house microservice deployment and service management • Leverages CloudFormation for infrastructure • Template-based system • INI-style configuration files • Output lookups • Leverages SaltStack for some configuration management • Uses roles for service management • Services in Docker containers • Supports health checking • Supports A/B deployments • Supports manual rollback • Tied to AWS • Lots of ELBs Platypus 4
- 5. Utilizes common, public, OSS tools • Common tools with existing user bases and communities • Basically “best of breed” tools • Packer for AMIs • Chef for installing and configuring software • Terraform for deploying baked AMIs • Plethora of documentation for each tool • Chef Server optional • Composable and reusable pieces • Output lookups (Chef + Terraform) Packer + Chef + Terraform 5
- 6. Mesos, Marathon, and Metronome (and more) • Consolidated and unified platform • Leverages common OSS technologies • Standardized application and service management • Health checking for services • Supports single-shot, or scheduled tasks • Service discovery • Metrics and log collection • Integrated data services • Configuration rollbacks • Canary deployments • Universe packages Mesosphere DC/OS 6
- 7. 7 Applause chose DC/OS to leverage previous work while also moving to a scalable system using open source components. This frees the Platform Delivery team to provide new capabilities to the Applause Hosting Platform which provide for our business needs. • Open source with a vibrant and active community • Strong feature set around an integrated platform • Ability to colocate diverse workloads • Microservices • Data services • AI / Machine learning / Analytics • Simple interfaces using API, CLI, and GUI • Enterprise features and support • Appreciation for memes Why DC/OS?
- 8. 8 DC/OS Architecture Software layer is where containers execute to provide services. This includes Marathon applications, Metronome jobs, and Mesos frameworks. Platform layer is Mesosphere DC/OS services execute, which run in the host operating system. Infrastructure layer provides the host and operating system which hosts our stack, such as Amazon Web Services.
- 9. 9 DC/OS Node Types Master nodes host DC/OS services and provide the orchestration layer, service discovery, and administrative interfaces. Public agent nodes are public facing and contain API routing and load balancing of incoming requests to backend services. These are agent nodes with a public role. Private agent nodes are internal and host all other services. Services communicate via East-West load balancing.
- 11. Mesosphere Universe packages + Application services 11 Packages and services which provide base value to the platform to be used by all Applause services: • ecr-login - AWS Elastic Container Registry login process • Provides and updates credentials for fetching images • marathon-lb - North-South load balancer • Provides ingress load balancing from public slaves to services running in private slaves • hdfs - Hadoop Distributed File System • Provides shared storage for artifacts, logs, etc. • Provides storage layer for AI/ML and analytics processes • linkerd - HTTP proxy • Provides service discovery and service mesh • Provides East-West load balancing across private slaves • kong - API gateway • Provides API routing to specific endpoints • spark - Data processing framework • Provides processing framework for AI/ML and analytics
- 12. • Chef cookbook wrapping community cookbook: https://supermarket.chef.io/cookbooks/dcos • Custom recipes • Monitoring agent • Docker Engine installation and configuration • Enhanced Networking (ena) driver • Logging aggregation agent • System users via data bag • DC/OS volumes (volume0, etc) • DC/OS workdir configuration • Cookbook “bake_time” • Packer templates to create “shared” images • Start from “official” CentOS base images • Patch • Reboot • Remove old kernels • Run Chef • Cleanup How do we build DC/OS? 12
- 13. Chef wrapper “secret sauce” 13 Disable some Chef resources by modifying resources at converge time: # These are resources which need to be modified in the upstream dcos # cookbook to prevent them from executing at bake time [ { template: '/usr/src/dcos/genconf/config.yaml' }, { execute: 'dcos-genconf' }, { file: '/usr/src/dcos/genconf/serve/dcos_install.sh' }, { execute: 'preflight-check' }, { execute: 'dcos_install' }, ].each do |res| ruby_block "action-nothing-#{res.keys.first}[#{res.values.first}]" do block do r = resources(res) r.action([:nothing]) end only_if { node['chef-applause-dcos']['bake_time'] } end end
- 14. Packer template 14 Runs Chef wrapper cookbook to produce a shared AMI: { "type": "chef-solo", "version": "13.8.5", "run_list": ["recipe[chef-applause-base]", "recipe[chef-applause-dcos]"], "json": { "chef-applause-dcos": { "bake_time": true, "chef_config_actions": ["nothing"] }, "dcos": { "dcos_enterprise": "{{user `dcos_enterprise`}}", "dcos_role": "nil", "dcos_version": "{{user `dcos_version`}}" } }, "cookbook_paths": [ "chef/cookbooks" ], "data_bags_path": "chef/pubkeys/data_bags" },
- 15. • Terraform • Derived from Mesosphere’s AWS CloudFormation templates • Originally a 1:1 translation • Evolved over time, more customizations • VPC per cluster • Masters have public addresses / ELB for discovery • Private slaves have only internal addresses • Public slaves are behind ALB • Autoscaling Groups + Launch Configs • One group per DC/OS role • Launch Configs write out node-specific Chef configuration • Executes Chef client in cloud-init at boot • IAM instance profiles used • One profile per DC/OS role How do we deploy DC/OS? 15
- 16. Launch Configs template 16 Provide outputs from terraform to Chef client: #cloud-config write_files: - content: | { "dcos": { "config": { "aws_region": "${region}", "cluster_name": "${stack_name}", "exhibitor_address": "${internal_elb_dns_name}", "master_external_loadbalancer": "${public_elb_dns_name}", "num_masters": "${num_masters}", "s3_bucket": "${exhibitor_s3_bucket}", "superuser_username": "${superuser_username}", "superuser_password_hash": "${superuser_password_hash}" }, "dcos_enterprise": ${dcos_enterprise}, "dcos_role": "${dcos_role}", "dcos_version": "${dcos_version}" } } path: |- /etc/chef/client-config.json
- 17. • Chef wrapper changes • Pull Request made • Tested with ChefDK and Test Kitchen • Merged to master • Tested again • Pushed to Chef server • Packer job executed • Creates AMIs in AWS accounts • Terraform job executed • Creates AWS resources • IAM accounts, profiles, instance profiles • VPC, subnets, security groups • ASGs, ELBs/ALBs • Creates terraform outputs Development to deployment workflow for DC/OS 17
- 18. • Application terraform updated • Databases, caches, storage buckets, etc • Service repository updated • Pull Request made • Unit tests, integration tests • Merge to master (or deployment branch) • Unit tests • Docker image • Integration tests • Code coverage • Push image • Service deployment / promotion to DC/OS • Metronome • Marathon • Kong Development to deployment workflow for Applause services 18
- 19. • Migrate more workloads from legacy hosting • Data science • Analytics • Build and test • Other products • Integrate services with in-cluster resources • Data services • Migrate scheduled jobs to Metronome • Chronos • cron • Applause platform • Migrate long-running tasks to Metronome • Kubernetes in-cluster What now? 19
- 20. Check out our careers page: https://www.applause.com/working-at-applause/We’re hiring 20
- 21. Questions? 21