There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly
•
0 likes•390 views
From ChefConf 2015. https://youtu.be/FI5sQQh8aKw When we first began using chef at Parse, we fell in love with it. Chef became our source of truth for everything. Bootstrapping, config files, package management, deploying software, service registration & discovery, db provisioning and backups and restores, cluster management, _everything_. But at some point we reached Peak Chef and realized our usage model was starting to cause more problems than it was solving for us. We still love the pants off of chef, but it is not the right tool for every job in every environment. I'll talk about the evolution of Parse's chef infrastructure, what we've opted to move out of chef, and some of the tradeoffs involved in using chef vs other tools.
Report
Share
There and Back Again: How We Drank the Chef Kool-Aid, Sobered Up, and Learned to Cook Responsibly
- 3. There and back again: a Chef tale How we drank the Kool-Aid, sobered up, and learned to cook responsibly.
- 4. Mobile apps platform 500k+ apps AWS MongoDB, Cassandra, Mysql, Redis ruby & rails => golang
- 5. Our mission: • Support relentless growth • Ship products fast • Solve mobile apps naively at scale
- 6. Active monthly Parse installations
- 7. API requests per second
- 8. • Support relentless growth • Ship products fast • Solve mobile apps naively at scale Our mission:
- 12. Chef the Base System!! • bootstrapping nodes with knife-ec2 • configuring system packages • managing deb versions • ec2 hostname tags from chef node names • route53 DNS records from hostname tags • cron jobs, batch jobs
- 13. Chef the Services!! • haproxy configs • generate yaml files • generate host lists • manage config files for Parse services • monitoring and graphing based off roles
- 14. Chef the Databases!! • creating/managing mongo replica sets • provisioning & assembling RAID devices • assigning cassandra initial tokens • backups, snapshotting & restores • community cookbooks for mysql, redis
- 15. Chef the Deploys!! • deploy Parse services? ….??????
- 16. wait …
- 18. 1) Things we did with chef badly 2) Things that chef was not the right tool for
- 19. mistakes were made … • Overloading roles with too much work • Confusion between role vs instantiation of service • Using definitions instead of providers • Using lots of data bags • One attribute per config entry instead of a hash of all entries • Using knife search extensively
- 20. mistakes were made … • Forking + modifying community cookbooks • Importing community cookbooks with too many custom dependencies • Not using repo-per-cookbook / Berkshelf • Not investing the time into vagrant, unit tests, staging environment, versioning • Where is my source of truth?!
- 21. but these are all solvable problems.
- 22. but these are all solvable problems. what isn’t?
- 23. sometimes, chef just ain’t enough.
- 24. • Provisioning from scratch • Service registration & discovery • Managing software & configs • Databases Problem areas
- 25. bootstrapping from vanilla AMIs launching instances with knife-ec2 Provisioning
- 26. bootstrapping from vanilla AMIs launching instances with knife-ec2 Provisioning Solution: bake AMI with chef, use ASGs
- 27. realtime search needs realtime data Service discovery
- 28. realtime search needs realtime data Service discovery Solution: zookeeper, consul, etcd, etc
- 29. Service discovery avoid snowflake hosts use distributed locking for cron jobs
- 30. Managing software & configs • System software (debs, rpms) • Developer-owned services • Internal operations software
- 31. Managing software & configs System software
- 32. Managing software & configs Developer-owned services • Do not tie code deploys to system changes • Perform the minimal set of changes • Configs *are* software. Version together.
- 33. Managing software & configs Internal operations software • Treat software engineering like software engineering • Treat systems-y packages like systems packages • Package and version “util” scripts • Manage package versions with Chef
- 35. Databases DBA operations Not really what chef is best at. Imperative commands Automatic remediation Coordinating actions across nodes
- 36. Databases DBA operations • Create, tear down replica sets or nodes • Verify backups • Rolling version upgrade • Elect new primary / switch masters • Enable/disable query killer • Change schemas or indexes • Compaction, rotation • Version replica set state • Etc
- 37. Databases DBA operations If you don’t have to do a ton of DBA ops, Chef can manage databases. Don’t over-engineer in advance of your actual needs.
- 38. Databases Separation of configuration and state Base system => chef Detect and publish state changes => chef, zk Generate monitoring configs => chef Imperative commands => db tooling
- 40. We chef for: • Building base AMIs • Generating monitoring configs • Storing encrypted secrets • Cron jobs (with zk lock) • Inferring and publishing db state changes
- 41. Things we still suck at • Single source of truth (git / chef-server) • Isolated staging environment • Full continuous testing for cookbooks
- 42. • Realtime data • Internal software packaging & management • Database administration at scale Things we don’t chef