More Related Content
What's hot
What's hot (20)
Similar to Using Machine Learning on K8s Logs to Find Root Cause Faster
Similar to Using Machine Learning on K8s Logs to Find Root Cause Faster (20)
More from LibbySchulze
More from LibbySchulze (20)
Recently uploaded
Recently uploaded (20)
Using Machine Learning on K8s Logs to Find Root Cause Faster
- 1. Z E B R I U M Webinar: Using Machine Learning on K8s Logs to Find Root Cause Faster Zebrium Proprietary & Confidential 7 Best Log Management Tools for Kubernetes LARRY LANCASTER Founder/CTO Zebrium ARAN KHANNA Co-Founder/CEO Reserved.ai GAVIN COHEN VP Product Zebrium
- 2. Your app is broken, now what?
- 3. Your app is broken, now what? Demo Part 1 Microservices demo app Deployed on a K8s cluster using minikube
- 4. Detected: incident opened PagerDuty Incident #1652: HTTP Status: 500 Internal Server Error Priority: P1 Description: Service not responding Monitoring dashboard Incident management tool
- 5. Time to hunt through your logs… ▪ Look for “bad” (errors, critical, alert, etc.) & rare events ▪ Focus on clusters of these, especially across services/hosts ▪ Leverage intuition and experience to keep exploring…
- 6. Imagine Automating the RCA Process? Send your logs, zero training… Identifies root cause without hunting
- 7. Your app is broken, now what? Demo Part 2 What the machine learning found when analyzing the logs from the broken app
- 8. Welcome! Aran Khanna Co-Founder/CEO @ Reserved.ai @arankhanna
- 9. About Reserved.ai Reserved.AI plans, forecasts, and manages your cloud resources for you ▪ Provides automated attribution, alerting and scenario modeling for complex multi-cloud environment ▪ Enables automated commitment management, cashflow forecasting and tax optimization of cloud resources ▪ Guarantees buyback of over-committed resources ▪ Requires only a simple 5-minute IAM role installation
- 10. The problem – Before Zebrium ▪ Cloud native stack in AWS using Kubernetes ▪ Flask, Celery, Redis, PostgreSQL, Airflow, React and more ▪ Each instance of each component generates lots of log events ▪ Prior to Zebrium, manual log searching when there was a problem ▪ vi, grep, awk, custom scripts ▪ Lots of time wasted by critical engineers to find root cause ▪ Potential for problems to go unnoticed
- 11. I was skeptical! ARE YOU TELLING ME AI CAN SHOW ROOT CAUSE?
- 12. Very early success ▪ Soon after trial started ▪ Unexpected AWS API change occurred ▪ Would have caused a service disruption had it not been caught ▪ Zebrium picked up the problem and correlated it with events from all the services that were impacted. ▪ Root cause report showed us what we needed ▪ We resolved problem quickly
- 13. Root cause reports are amazing
- 14. GPT-3 is amazing ▪ Some recent examples
- 15. Using Zebrium to Find Root Cause Automatic root cause analysis ! Monitoring, APM, Logging, Help Desk or any other tool ML Root Cause Report 2 3 1 With incident management tool (PagerDuty, Opsgenie, Slack, etc.) When something breaks: Scan root cause summaries around time of interest (existing or on-demand) Without incident management tool
- 16. How the ML works ML-driven structuring, event categorization & pattern learning 1 2 Anomaly scoring for each incoming event Raw log streams Cluster of abnormally correlated anomalies across log streams 3 Metrics (optional) Identify correlated metrics anomalies 5 Automatic root cause report Optional Feedback 6 4 Correlated Rare & Error Events
- 17. ML-driven Root Cause Analysis : Examples of outages, crashes, failures, etc. Bugs Undefined variables, Undefined index, mismatched data structures Missing function arguments, input validation failed SQL syntax errors Inter-Service Interactions Auth & connection failures Rate limit failures API version mis-match errors Container Orchestration Missing containers, pods Crashloopbackoffs failed leader elections Kafka, Redis, Databases Kafka broker pod failure SQL schema errors Redis performance issues Database connectivity issues Security Excessive auth failures Invalid Root login attempts WAF attacks XSS vulnerabilities Infrastructure OOM process failures Out of space, CPU saturation Object storage failures Network corruption, VM failovers Note: No pre-built rules
- 18. Thank you! hello@zebrium.com @ZebriumAI Free trial: www.zebrium.com “Zebrium caught a vendor API change which caused problems downstream.” - Aran Khanna, CEO “Zebrium helped us root cause obscure error messages that didn’t seem to match our code." – Yosef Deray, Lead Software Engineer “Zebrium shrunk MTTR from 3 hours to <10 mins, and caught a Redis performance issue that was silently impacting users.” - Jason Johnson – SVP of InfoTech Zebrium saves us hours per major incident, and in aggregate, has saved us days.” Mike Heyeck - Cloud Lead “Zebrium identified root cause for every failure induced using Litmus Chaos Engine.” - Murat Karslioglu, VP of Product Top 50 AI Companies 7 Best Log Tools for Kubernetes Zebrium named a Gartner Cool Vendor