Bloated Chefs: A Tale of Gluttony, & the Path to Enlightenment - ChefConf 2015
•
0 likes•365 views
Evan Gilman presented on optimizing Chef resources at PagerDuty that had led to bloated Chef runs. As PagerDuty grew, Chef runs experienced CPU spikes, long run times, and out of memory errors. To address this, they measured Chef run times and resource usage, identified inefficiencies like full node searches and redundant API calls, and implemented optimizations like partial searches, result memoization, and centralized API access. These changes reduced Chef run memory usage by 88% and time by 84%, allowing faster convergence and less resource intensive runs.
Report
Share
Bloated Chefs: A Tale of Gluttony, & the Path to Enlightenment - ChefConf 2015
- 1. 4/3/15 @evan2645 EVAN GILMAN Bloated Chefs A Tale of Gluttony, and the Path to Enlightenment
- 3. 4/3/15 Agenda BLOATED CHEFS 1. Chef resources in use at PD 2. Problems encountered as we grew 3. Measuring chef-client run 4. How we fixed it 5. How fast is it now?
- 4. 4/3/15BLOATED CHEFS CHEF @ PAGERDUTY
- 11. 4/3/15BLOATED CHEFS ALL WAS NOT WELL
- 12. 4/3/15 As we grew… BLOATED CHEFS
- 13. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs
- 14. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs • Awkward pauses at the beginning of the run
- 15. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs • Awkward pauses at the beginning of the run • chef-client run took several minutes
- 16. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs • Awkward pauses at the beginning of the run • chef-client run took several minutes • chef-client OOM
- 17. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs • Awkward pauses at the beginning of the run • chef-client run took several minutes • chef-client OOM
- 21. 4/3/15 Measuring Run Time BLOATED CHEFS
- 22. 4/3/15 Measuring Run Time BLOATED CHEFS https://github.com/joemiller/chef-handler-profiler
- 23. 4/3/15 Measuring Resources BLOATED CHEFS • Total number of resources per run, by type • Number of updated resources per run, by type
- 24. 4/3/15 Measuring Memory BLOATED CHEFS • Gather proc stats with sys-proctable • Gather GC stats • Can be emitted as statsd
- 26. 4/3/15BLOATED CHEFS WHAT WE FOUND AND WHAT WE DID
- 29. 4/3/15 Step-through Searches BLOATED CHEFS From this To this
- 30. 4/3/15 Step-through Searches BLOATED CHEFS 417MB -> 190MB
- 31. 4/3/15 Step-through Searches BLOATED CHEFS 417MB -> 190MB ~54%
- 33. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results
- 34. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled
- 35. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled • hash2node
- 36. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled • hash2node • Two searches touched
- 37. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled • hash2node • Two searches touched 90s -> 60s
- 38. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled • hash2node • Two searches touched 90s -> 60s 30%
- 40. 4/3/15 Result Memoization BLOATED CHEFS • Common search data
- 41. 4/3/15 Result Memoization BLOATED CHEFS • Common search data • API-backed LWRP’s
- 42. 4/3/15 Result Memoization BLOATED CHEFS • Common search data • API-backed LWRP’s • Can be generalized
- 43. 4/3/15 Result Memoization BLOATED CHEFS • Common search data • API-backed LWRP’s • Can be generalized
- 47. 4/3/15 Other Nasties BLOATED CHEFS • Too many conditional guards
- 48. 4/3/15 Other Nasties BLOATED CHEFS • Too many conditional guards • tmpfs storage
- 49. 4/3/15 Other Nasties BLOATED CHEFS • Too many conditional guards • tmpfs storage • Multiple package resources (Chef 12)
- 50. 4/3/15 Other Nasties BLOATED CHEFS • Too many conditional guards • tmpfs storage • Multiple package resources (Chef 12) Six seconds for twelve packages
- 53. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After:
- 54. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After: ~60MB
- 55. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After: ~60MB 88% less memory!
- 57. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After:
- 58. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After: ~30s/run
- 59. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After: ~30s/run ~84% faster!