SlideShare a Scribd company logo

Consul - service discovery and others

•
•
Walter Liu
Walter Liu

Shortly describe how Consul could help us and propose some minor changes in department for future continuous delivery.

1 of 18
Download to read offline
Consul -­‐ Service discovery and others By Walter Liu 2014/11/11
Similar services • Consul • SkyDNS • Others • Zookeeper • etcd • Serf
What’s Consul? • Service Discovery • DNS • HTTP (JSON) • Health Checking (Service Discovery need this) • Key/Value Store • dynamic configuraMon • feature flagging • coordinaMon • leader elecMon • (Very like the funcMonality of Zookeeper we used) • MulM Datacenter • Network parMMon • If one DC off, no impact to another DC. • Data duplicate to another DC automaMcally.
Consul - service discovery and others
User Scenarios in Our Infra • Simplify config (even with SaltStack) • Dynamic server cluster • Faster failure recovery • Dynamic cluster configuraMon • Launching a node in a container cloud. (Future maybe?)
Scenario: Simplify con>ig • ENV (from hostname or specified in Salt) • $TYPE=“prod” • $GROUP=“gg” • $DATACENTER=“mydc1” • Config could be • HOST_POSTFIX = $GROUP+$TYPE+$DATACENTER = • “prod.gg.mydc1.consul” • Config in Group • “db.prod.gg.mydc.consul” • Config for a datacenter • “redis.prod.mydc1.consul” • So, every config for prod/stg/dev could be the same. • Do config in service provider instead of every client. • Service provider could automaMcally register if supported.
Scenario: Dynamic server cluster • Background • A server cluster without load balancer (Cannot apply LB) • An applicaMon that use this service • Sequence • Query Consul will return a list of servers. • When a server failed, Consul will aware that very quickly and remove it from the list. • An applicaMon that use this cluster will be interrupted and need to re-­‐query Consul. Consul will return the new list of servers.
Scenario: Faster failure recovery • Background • A master/slave DB without automaMc failover • Sequence • The master DB is dead. • Nagios found it. • OPS aware it and log-­‐in slave DB. • OPS restart slave DB as master and run script to register new DB • All applicaMons that use this DB will change to new DB.
Scenario: Dynamic cluster con>iguration • If a service cluster needs dynamic/automaMc change configuraMon during run-­‐Mme.
Scenario: Launching a node in a container cloud. • Assume the container image of the service is ready. • Specify correct hostname of the new container. • Launch the new container. • The new container start the new service. • If it is a simple service provider, it registers itself to Consul. • If it uses other services, it queries these services from Consul. • If the configuraMon is all the same, the new node is ready.
OTHERS
Use pip + virtualenv • Why, • Reduce package conflict. (Use egg is not very friendly.) • Some Python packages doesn’t have RPM. • Install using requirement.txt (pip install –r requirement.txt) • (?) private pip repository • Below for dev • Use virtualenvwrapper to be more convenient. • How to produce requirement more easily? • pip freeze > stable-­‐req.txt • Pip freeze –r dev-­‐req.txt > stable-­‐req.txt # Keep old comments
Example: requirement.txt • Beaker==1.3.1 • Chameleon==2.11 • DecoratorTools==1.7 • DistuMls2==1.0a3 • Django==1.5.1 • M2Crypto==0.20.2 • Magic-­‐file-­‐extensions==0.1 • Mako==0.8.1 • MarkupSafe==0.9.2 • MySQL-­‐python==1.2.3c1 • PasteDeploy==1.5.0 • PyXML==0.8.4 • Pygments==1.1.1 • SOAPpy==0.11.6 • SSSDConfig==1.11.6 • Scrapy==0.18.4
Store SHOULD-­‐BE-­‐CHANGED con>ig in environment var • Why, • Leak secrets in version control system. • Check-­‐in always config to version control system accidentally. • Example, • DB username/password, AWS API Keys • Secret-­‐token (web framework like Django) • URI resource string, Used service host, port …..
Explicitly store prod/stg/dev settings in version control • Why, • Check-­‐in always config to version control system accidentally. • All sevngs in all environment are visible to everyone. • For example, • DEBUG=True in Django • Log level • How (Django for example) • sevngs/ • prod.py • staging.py • dev.py • ENV[“DJANGO_SETTINGS_MODULE”] = “sevngs.prod” • “sevngs.prod” could automaMcally come from hostname. • If this env variable is not set, service will print log and exit.
Consul vs. SkyDNS • Pros • Beyer health check (SkyDNS uses heartbeat + TTL) • MulMple datacenter support by using “region”. • A beyer web view. (hyp://demo.consul.io/ui/) • Beyer RAFT implementaMon (unverified) • Cons • lesser contributors (1/3) • More immature (0.41 now) and young.
Problem needs attention • For only one server service like DB, what if a second server register the service accidentally? • Use DNS+DNS forwarding approach may not be good for DNS heavy service. (Like Odin)
Other References • Service registrator for Docker • hyps://github.com/progrium/registrator • Set env by Consul • hyps://github.com/hashicorp/envconsul

More Related Content

Consul - service discovery and others

  • 1. Consul -­‐ Service discovery and others By Walter Liu 2014/11/11
  • 2. Similar services • Consul • SkyDNS • Others • Zookeeper • etcd • Serf
  • 3. What’s Consul? • Service Discovery • DNS • HTTP (JSON) • Health Checking (Service Discovery need this) • Key/Value Store • dynamic configuraMon • feature flagging • coordinaMon • leader elecMon • (Very like the funcMonality of Zookeeper we used) • MulM Datacenter • Network parMMon • If one DC off, no impact to another DC. • Data duplicate to another DC automaMcally.
  • 5. User Scenarios in Our Infra • Simplify config (even with SaltStack) • Dynamic server cluster • Faster failure recovery • Dynamic cluster configuraMon • Launching a node in a container cloud. (Future maybe?)
  • 6. Scenario: Simplify con>ig • ENV (from hostname or specified in Salt) • $TYPE=“prod” • $GROUP=“gg” • $DATACENTER=“mydc1” • Config could be • HOST_POSTFIX = $GROUP+$TYPE+$DATACENTER = • “prod.gg.mydc1.consul” • Config in Group • “db.prod.gg.mydc.consul” • Config for a datacenter • “redis.prod.mydc1.consul” • So, every config for prod/stg/dev could be the same. • Do config in service provider instead of every client. • Service provider could automaMcally register if supported.
  • 7. Scenario: Dynamic server cluster • Background • A server cluster without load balancer (Cannot apply LB) • An applicaMon that use this service • Sequence • Query Consul will return a list of servers. • When a server failed, Consul will aware that very quickly and remove it from the list. • An applicaMon that use this cluster will be interrupted and need to re-­‐query Consul. Consul will return the new list of servers.
  • 8. Scenario: Faster failure recovery • Background • A master/slave DB without automaMc failover • Sequence • The master DB is dead. • Nagios found it. • OPS aware it and log-­‐in slave DB. • OPS restart slave DB as master and run script to register new DB • All applicaMons that use this DB will change to new DB.
  • 9. Scenario: Dynamic cluster con>iguration • If a service cluster needs dynamic/automaMc change configuraMon during run-­‐Mme.
  • 10. Scenario: Launching a node in a container cloud. • Assume the container image of the service is ready. • Specify correct hostname of the new container. • Launch the new container. • The new container start the new service. • If it is a simple service provider, it registers itself to Consul. • If it uses other services, it queries these services from Consul. • If the configuraMon is all the same, the new node is ready.
  • 12. Use pip + virtualenv • Why, • Reduce package conflict. (Use egg is not very friendly.) • Some Python packages doesn’t have RPM. • Install using requirement.txt (pip install –r requirement.txt) • (?) private pip repository • Below for dev • Use virtualenvwrapper to be more convenient. • How to produce requirement more easily? • pip freeze > stable-­‐req.txt • Pip freeze –r dev-­‐req.txt > stable-­‐req.txt # Keep old comments
  • 13. Example: requirement.txt • Beaker==1.3.1 • Chameleon==2.11 • DecoratorTools==1.7 • DistuMls2==1.0a3 • Django==1.5.1 • M2Crypto==0.20.2 • Magic-­‐file-­‐extensions==0.1 • Mako==0.8.1 • MarkupSafe==0.9.2 • MySQL-­‐python==1.2.3c1 • PasteDeploy==1.5.0 • PyXML==0.8.4 • Pygments==1.1.1 • SOAPpy==0.11.6 • SSSDConfig==1.11.6 • Scrapy==0.18.4
  • 14. Store SHOULD-­‐BE-­‐CHANGED con>ig in environment var • Why, • Leak secrets in version control system. • Check-­‐in always config to version control system accidentally. • Example, • DB username/password, AWS API Keys • Secret-­‐token (web framework like Django) • URI resource string, Used service host, port …..
  • 15. Explicitly store prod/stg/dev settings in version control • Why, • Check-­‐in always config to version control system accidentally. • All sevngs in all environment are visible to everyone. • For example, • DEBUG=True in Django • Log level • How (Django for example) • sevngs/ • prod.py • staging.py • dev.py • ENV[“DJANGO_SETTINGS_MODULE”] = “sevngs.prod” • “sevngs.prod” could automaMcally come from hostname. • If this env variable is not set, service will print log and exit.
  • 16. Consul vs. SkyDNS • Pros • Beyer health check (SkyDNS uses heartbeat + TTL) • MulMple datacenter support by using “region”. • A beyer web view. (hyp://demo.consul.io/ui/) • Beyer RAFT implementaMon (unverified) • Cons • lesser contributors (1/3) • More immature (0.41 now) and young.
  • 17. Problem needs attention • For only one server service like DB, what if a second server register the service accidentally? • Use DNS+DNS forwarding approach may not be good for DNS heavy service. (Like Odin)
  • 18. Other References • Service registrator for Docker • hyps://github.com/progrium/registrator • Set env by Consul • hyps://github.com/hashicorp/envconsul