Practical service level objectives with error budgeting
•Download as PPTX, PDF•
3 likes•1,595 views
This document summarizes Fred Moyer's presentation on practical service level objectives with error budgeting. It discusses how to calculate error budgets based on service level indicators, objectives, and agreements. It presents methods to calculate error budgets using log files by counting errors and slow requests, and using metrics like counters and histograms to track errors and response time distributions over time. Maintaining an error budget allows managing risk while releasing new code by setting a target for acceptable errors or slow requests.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Practical service level objectives with error budgeting
Similar to Practical service level objectives with error budgeting (20)
More from Fred Moyer
More from Fred Moyer (15)
Recently uploaded
Recently uploaded (20)
Practical service level objectives with error budgeting
- 1. Practical Service Level Objectives With Error Budgeting Fred Moyer @phredmoyer BayLISA May 16, 2019
- 4. How many errors in your app last week? @phredmoyer
- 5. How many requests over 500ms last week? @phredmoyer
- 6. Your error/request ratio last week? @phredmoyer
- 7. Are slow requests errors? @phredmoyer
- 8. Hi I’m Fred ● @phredmoyer ● Monitoring Nerd ● Writing code 20 years ● And breaking prod ● Likes Go, Perl, C, Pg ● Likes SLOs ● Doesn’t like errors @phredmoyer
- 9. Talk Agenda ● SLOs and Error Budgets ● Calculating Error Budgets with Logs ● Calculating Error Budgets with Metrics @phredmoyer
- 10. What is an Error Budget? @phredmoyer Zero Errors! Happy Users!
- 11. What is an Error Budget? @phredmoyer Too much risk = Too many errors Too many errors = Unhappy users Too little risk = No code shipped No code shipped = Unhappy users
- 12. What is an Error Budget? @phredmoyer Too much risk = Too many errors Too many errors = Unhappy users Too little risk = No code shipped No code shipped = Unhappy users
- 13. What is an Error Budget? @phredmoyer Too much risk = Too many errors Too many errors = Unhappy users Too little risk = No code shipped No code shipped = Unhappy users
- 14. What is an Error Budget? @phredmoyer Too much risk = Unhappy users Just enough risk = Happy users Too little risk = Unhappy users
- 15. What is an Error Budget? @phredmoyer Error budget = Acceptable risk Acceptable risk = 100%-SLO Error budget = 100%-SLO
- 16. @phredmoyer SLOs, How Do They Work?
- 17. SLOs, How Do They Work? @phredmoyer SLIs, SLOs, SLAs, oh my! https://www.youtube.com/watch?v=tEylFyxbDLE @lizthegrey ⇔ @sethvargo SLI: 95th %ile requests over 5 min < 300ms SLO: 95th %ile SLI for 1 month succeeds 99.9% SLA: 95th %ile SLI for 1 month succeeds 99.5% or you have to refund money
- 18. What is an Error Budget? @phredmoyer SLI: 95th %ile req over 5 min < 300ms SLO: 95th %ile SLI for 1 month succeeds 99.9% 1M reqs in one month Error Budget = (1-0.999)*1M = 1k requests 1k requests can exceed 300ms
- 19. What is an Error Budget? @phredmoyer Chapter 3 Embracing Risk
- 20. Talk Agenda ● SLOs and Error Budgets ● Calculating Error Budgets with Logs ● Calculating Error Budgets with Metrics @phredmoyer
- 21. Calculating Error Budgets with Logs @phredmoyer Latency
- 22. Calculating Error Budgets with Logs - Latency @phredmoyer Error Budget = 100%-SLO = (1-0.999)*1M = 1k Error Budget = 1k requests/day > 300ms EventLog "%h %l %u %O "%{User-Agent}i" %D" %D - Request duration in milliseconds For each request: If duration > SLI (300ms), error_budget++
- 23. Calculating Error Budgets with Logs - Errors @phredmoyer Errors
- 24. Calculating Error Budgets with Logs - Errors @phredmoyer Error Budget = 1k requests/day > 300ms [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server: /export/home/live/ap/htdocs/test For each error log entry, error_budget++ If req duration > SLI (300ms), error_budget++ Alert if error_budget/total_reqs > 80% * 1-SLO
- 25. Calculating Error Budgets with Logs @phredmoyer Cumulative sum functionality required ● Splunk ● ELK ● Mtail ○ https://github.com/google/mtail ● Honeycomb.io ● Circonus Logwatch ○ https://github.com/circonus- labs/circonus-logwatch
- 26. Talk Agenda ● SLOs and Error Budgets ● Calculating Error Budgets with Logs ● Calculating Error Budgets with Metrics @phredmoyer
- 27. Calculating Error Budgets with Metrics @phredmoyer Errors
- 28. Calculating Error Budgets with Metrics @phredmoyer Use a counter metric (uint32/uint64) Error Budget = 1k requests/day > 300ms For each app error, error_budget++ If req duration > SLI (300ms), error_budget++ Alert if error_budget/total_reqs > 80% * 1-SLO
- 29. Calculating Error Budgets with Metrics (and Logs) @phredmoyer Problems: ● SLI fixed threshold ● Inability to introspect historical data ● Difficult to compare different SLI behavior
- 30. Calculating Error Budgets with Metrics - Histograms @phredmoyer Use a histogram Image source http://www.brendangregg.com/FrequencyTrails/modes.html
- 31. Calculating Error Budgets with Metrics - Histograms @phredmoyer Linear, Cumulative, Log-Linear, Approximate… High dynamic range, log-linear recommended http://hdrhistogram.org/ https://github.com/circonus/-labs/circonusllhist
- 32. Calculating Error Budgets with Metrics - Histograms @phredmoyer Error Budget = 1k requests/day > Xms For each histogram bin >= X: error_budget += bin_sample_count Alert if error_budget/total_reqs > 80% * 1-SLO
- 33. Calculating Error Budgets with Metrics - Histograms @phredmoyer Choose bin boundary for SLI (preferred) or interpolate within boundaries
- 34. Calculating Error Budgets with Metrics - Histograms @phredmoyer Error Budget ~ 1k requests/day > 1,800µs
- 35. Calculating Error Budgets with Metrics - Histograms @phredmoyer Error Budget ~ 1k requests/day > 2,400µs
- 36. Calculating Error Budgets with Metrics - Histograms @phredmoyer Benefits: ● SLI variable threshold ● Ability to analyze historical data ● Examine error budgets for different SLIs
- 37. Talk Agenda ● SLOs and Error Budgets ● Calculating Error Budgets with Logs ● Calculating Error Budgets with Metrics @phredmoyer
- 40. Appendix - SLOs, How Do They Work? @phredmoyer ● Chapter 4 ○ Service Level Objectives ● 99% Get RPC calls < 100ms ● https://landing.google.com/sre/sre-book/toc/index.html
- 41. @phredmoyer ● Ch 2: Implementing SLOs ● Ch 3: SLO Eng case studies ● Ch 5: Alerting on SLOs ● https://landing.google.com/sre/workbook/toc Appendix - SLOs, How Do They Work?
- 42. @phredmoyer ● Chapter 21 ○ The Art and Science of The Service Level Objective Appendix - SLOs, How Do They Work?