OSDC 2015: Mitchell Hashimoto | Automating the Modern Datacenter, Development to Production
•
0 likes•190 views
Physical, virtual, containers. Public cloud, private cloud, hybrid cloud. IaaS, PaaS, SaaS. These are the choices that we're faced with when architecting a datacenter of today. And the choice is not one or the other; it is often a combination of many of these. How do we remain in control of our datacenters? How do we deploy and configure software, manage change across disparate systems, and enforce policy/security? How do we do this in a way that operations engineers and developers alike can rejoice in the processes and workflow? In this talk, I will discuss the problems faced by the modern datacenter, and how a set of open source tools including Vagrant, Packer, Consul, and Terraform can be used to tame the rising complexity curve and provide solutions for these problems.
Report
Share
OSDC 2015: Mitchell Hashimoto | Automating the Modern Datacenter, Development to Production
- 1. Automa'ng the Modern Datacenter
- 3. Powering the so9ware-‐managed datacenter. HashiCorp hashicorp.com
- 9. Virtualiza'on Datacenter Server Server VM VM VM VM VM VM VM VM
- 11. Service Prolifera'on Datacenter Server Server Server Server Server Server Server Server Server DNS Database CDN
- 12. Etc… • Hybrid cloud: Physical datacenter vs. Cloud provider • Mul'-‐paradigm: Physical, virtual, container • IaaS, PaaS, SaaS depending on the app • Opera'ng systems: Windows, Linux, Mac, Other • Realis'cally a mixture of everything just shown
- 13. But… why?
- 14. Common Goal: Efficiently deliver and maintain applica9ons.
- 15. Applica'on Delivery • Consistent • Shareable • Readily Available • High producCon parity. • Start and configure servers / services • Deploy and run applicaCon • Update servers or applicaCons • Reconfigure, feature flag • Monitor health • Orchestrate complex changes Development Deployment Maintenance
- 16. HashiCorp’s Open Source Tools Development Deployment Maintenance
- 17. Taming the Datacenter Deployment + Maintenance
- 18. Deployment + Maintenance 1. Acquisi'on 2. Provision 3. Update 4. Destroy
- 19. Historically • Servers: Days, weeks • Provisioning: Hours, days • SaaS: <didn’t exist>
- 20. Today • Servers: Minutes • Provisioning: Minutes • SaaS: Minutes
- 21. Historically • Rela'vely fixed set of servers • Fewer applica'ons to deploy • Fewer SaaS • Less demanding web traffic
- 22. Today • Poten'ally elas'c set of servers of varying sizes • Push towards SoA • SaaS for everything • More internet connected devices than ever before => higher traffic
- 23. What do we need? • Zero to deployed in one command • Resiliency through distributed systems • Autoscaling, autohealing • Beder teamwork through codified knowledge
- 25. HashiCorp’s Open Source Tools Development Deployment Maintenance
- 26. Automa9ng the Datacenter Deployment + Maintenance
- 27. terraform.io
- 28. Build, combine, and launch infrastructure safely and efficiently. terraform.io
- 29. What If I asked you to… • create a completely isolated second environment to run an applica'on (staging, QA, dev, etc.)? • deploy a complex new applica'on? • update an exis'ng complex applica'on? • document how our infrastructure is architected? • delegate some ops to smaller teams? (Core IT vs. App IT)
- 30. What If I asked you to… • create a completely isolated second environment to run an applica'on (staging, QA, dev, etc.)? One command. • deploy a complex new applica'on? Code it, diff it, pull request. • update an exis'ng complex applica'on? Code it, diff it, pull request. • document how our infrastructure is architected? Read the code. • delegate some ops to smaller teams? (Core IT vs. App IT) Modules, code reviews.
- 31. But how?
- 32. Terraform • Create infrastructure with code: servers, load balancers, databases, email providers, etc. • One command to create, update infrastructure. • Preview changes to infrastructure, save diffs. • Use code + diffs to treat infrastructure change just like code change: make a pull request, show the differences, review it, and accept. • Break infrastructure into modules to encourage/allow teamwork without risking stability.
- 33. Infrastructure as Code DigitalOcean Droplet with DNS in DNSimple resource "digitalocean_droplet" "web" { name = "tf-web" size = "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" }
- 34. Infrastructure as Code DigitalOcean Droplet with DNS in DNSimple resource "digitalocean_droplet" "web" { name = "tf-web" size = "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" }
- 35. Infrastructure as Code DigitalOcean Droplet with DNS in DNSimple resource "digitalocean_droplet" "web" { name = "tf-web" size = "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" }
- 36. Infrastructure as Code DigitalOcean Droplet with DNS in DNSimple resource "digitalocean_droplet" "web" { name = "tf-web" size = "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" }
- 37. Infrastructure as Code • Human friendly config, JSON compa'ble • Text format makes it version-‐able, VCS-‐friendly • Declara've • Infrastructure as code on a level not before possible
- 38. Zero to Done in One Command Terraform Apply $ terraform apply digitalocean_droplet.web: Creating… dnsimple_record.hello: Creating… Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
- 39. Zero to Done in One Command • Idempotent • Highly parallelized • Will only do what the plan says
- 40. Safely Change/Iterate Terraform Plan + digitalocean_droplet.web backups: "" => "<computed>" image: "" => "centos-5-8-x32" ipv4_address: "" => "<computed>" ipv4_address_private: "" => "<computed>" name: "" => "tf-web" private_networking: "" => "<computed>" region: "" => "sfo1" size: "" => "512mb" status: "" => "<computed>" + dnsimple_record.hello domain: "" => "example.com" domain_id: "" => "<computed>" hostname: "" => "<computed>" name: "" => "test" priority: "" => "<computed>" ttl: "" => "<computed>" type: "" => "A" value: "" => "${digitalocean_droplet.web.ipv4_address}"
- 41. Safely Change/Iterate Terraform Plan + digitalocean_droplet.web backups: "" => "<computed>" image: "" => "centos-5-8-x32" ipv4_address: "" => "<computed>" ipv4_address_private: "" => "<computed>" name: "" => "tf-web" private_networking: "" => "<computed>" region: "" => "sfo1" size: "" => "512mb" status: "" => "<computed>" + dnsimple_record.hello domain: "" => "example.com" domain_id: "" => "<computed>" hostname: "" => "<computed>" name: "" => "test" priority: "" => "<computed>" ttl: "" => "<computed>" type: "" => "A" value: "" => "${digitalocean_droplet.web.ipv4_address}"
- 42. Safely Change/Iterate Terraform Plan + digitalocean_droplet.web backups: "" => "<computed>" image: "" => "centos-5-8-x32" ipv4_address: "" => "<computed>" ipv4_address_private: "" => "<computed>" name: "" => "tf-web" private_networking: "" => "<computed>" region: "" => "sfo1" size: "" => "512mb" status: "" => "<computed>" + dnsimple_record.hello domain: "" => "example.com" domain_id: "" => "<computed>" hostname: "" => "<computed>" name: "" => "test" priority: "" => "<computed>" ttl: "" => "<computed>" type: "" => "A" value: "" => "${digitalocean_droplet.web.ipv4_address}"
- 43. Safely Change/Iterate • Plan shows you what will happen • Save plans to guarantee what will happen • Plans show reasons for certain ac'ons (such as re-‐create) • Prior to Terraform: Operators had to “divine” change ordering, paralleliza'on, rollout effect.
- 44. Workflow • Make code changes • `terraform plan` • Pull request with code changes + plan to make changes • Review and merge • `terraform apply pr1234.pplan`
- 45. Knowledge Sharing: Modules Terraform Plan module “consul” { source = “github.com/hashicorp/consul/terraform/aws” servers = 3 } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = “${module.consul.server_address}” type = "A" }
- 46. Knowledge Sharing: Modules Terraform Plan module “consul” { source = “github.com/hashicorp/consul/terraform/aws” servers = 3 } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = “${module.consul.server_address}” type = "A" }
- 47. Knowledge Sharing: Remote Modules Terraform Plan resource “terraform_remote_state” “consul” { type = “atlas” name = “hashicorp/consul” } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = “${terraform_remote_state.consul.outputs.server_address}” type = "A" }
- 48. Knowledge Sharing: Modules • Self-‐contained infrastructure components • Allows delega'on of responsibility to mul'ple teams • Some teams create modules, other teams consume modules • Remote modules let teams share outputs, but not affect infrastructure
- 49. Terraform • Zero to fully deployed in one command • Change/maintain infrastructure predictably • Teamwork-‐oriented workflow to infrastructure • Goal: Sta'c deploy/provisioning of infrastructure. Real'me monitoring, discovery, configura'on provided by Consul (discussed next).
- 50. consul.io
- 51. Service discovery, configura9on, and orchestra9on made easy. Distributed, highly available, and datacenter-‐aware.
- 52. Ques'ons that Consul Answers • Where is the service foo? (ex. Where is the database?) • What is the health status of service foo? • What is the health status of the machine/node foo? • What is the list of all currently running machines? • What is the configura'on of service foo? • Is anyone else currently performing opera'on foo?
- 53. Service Discovery Where is service foo?
- 54. Service Discovery Service Discovery via DNS or HTTP $ dig web-frontend.service.consul. +short 10.0.3.89 10.0.1.46 $ curl http://localhost:8500/v1/catalog/service/web-frontend [{ “Node”: “node-e818f1”, “Address”: “10.0.3.89”, “ServiceID”: “web-frontend”, … }]
- 55. Service Discovery • DNS is legacy-‐friendly. No applica'on changes required. • HTTP returns rich metadata. • Discover both internal and external services (such as service providers)
- 56. Failure Detection Is service foo healthy/available?
- 57. Failure Detec'on
- 58. Failure Detec'on • DNS won’t return non-‐healthy services or nodes. • HTTP has endpoints to list health state of catalog.
- 59. Key/Value Storage What is the config of service foo?
- 60. Key/Value Storage Serng and Gerng a Key $ curl –X PUT –d ‘bar’ http://localhost:8500/v1/kv/foo true $ curl http://localhost:8500/v1/kv/foo?raw bar
- 61. Key/Value Storage • Highly available storage of configura'on. • Turn knobs without big configura'on management process. • Watch keys (long poll) for changes • ACLs on key/value to protect sensi've informa'on
- 63. Mul'-‐Datacenter Service Discovery $ dig web-frontend.singapore.service.consul. +short 10.3.3.33 10.3.1.18 $ dig web-frontend.germany.service.consul. +short 10.7.3.41 10.7.1.76
- 64. Mul'-‐Datacenter Serng and Gerng a Key $ curl http://localhost:8500/v1/kv/foo?raw&dc=asia true $ curl http://localhost:8500/v1/kv/foo?raw&dc=eu false
- 65. Mul'-‐Datacenter • Local by default • Can query other datacenters however you may need to • Can view all datacenters within one UI
- 66. Orchestration Events, Exec, Watches
- 67. Events, Exec, Watches Dispatching Custom Events $ consul event deploy 6DF7FE … $ consul watch -type event -name deploy /usr/bin/deploy.sh … $ consul exec -service web /usr/bin/deploy.sh …
- 68. Events, Exec, Watches • Powerful orchestra'on tools • Pros/cons to each approach, use the right tool for the job • All approaches proven to scale to thousands of agents
- 69. Easiest Distributed System Deploy Deploy Consul to AWS $ terraform apply github.com/hashicorp/consul/terraform/aws var.servers The number of Consul servers to launch. Default: 3 Enter a value: 3 …
- 70. Easiest Distributed System Deploy Deploy Consul to AWS (manually) $ consul agent -atlas-join -atlas=USERNAME/NAME -atlas-token=API_TOKEN
- 71. Workflow • Server is started (via Terraform, etc.) • Consul agent is started, joins cluster • Star'ng services (ex. web app) query Consul for configura'on • Once healthy, services are discovered via DNS!
- 72. Opera'onal Bullet Points • Leader elec'on via Ra9 • Gossip protocol for aliveness • Three consistency models: default, consistent, and stale • Encryp'on, ACLs available • Real world usage to thousands of agents per datacenter