Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016
•
2 likes•2,044 views
"Elasticsearch has come a long way: Started as a distributed search engine in 2009, it's now the tool of choice for even the largest websites (e.g. Facebook, Github, Ebay). Half-way to 2016 the ELK stack helped it become firmly embedded in many centralised log management systems (e.g. Netflix, Uber). We're now midair in the next step, with the first folks using it for metrics. NASA is using it to monitor the Curiosity rover, Blizzard and Riot to monitor vast online gaming worlds. This talk will focus on what makes this transition from more unstructured to structured data possible. "
Report
Share
Related slideshows
Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016
- 2. BATTLE PLAN 15:05 – 15:40 SLIDES 15:40 – 15:55 COFFEE 15:55 – 16:55 HANDS ON
- 4. Unstructured Data Structured Data Search Logging Security MetricsAnalytics Social SensorLogsDocuments User Activity Location
- 5. Elastic Cloud Security X-Pack Kibana User Interface ElasticsearchStore, Index, & Analyze Ingest Logstash Beats + Elastic Stack Alerting Monitoring Reporting Graph
- 6. DATA STORE FAST SCALEABLE FULL TEXT Numbers need 60% less memory in 5.0
- 7. USER INTERFACE VISUALIZATIONS DASHBOARDS REAL TIME CHARTING LANGUAGE (5.0)
- 8. DATA PROCESSING REAL TIME PIPELINE 200+ PLUGINS SYSLOG REGEX MULTILINE LOOKUPS GEOIP
- 9. DATA COLLECTION AT THE SOURCE LIGHTWEIGHT REAL TIME Preconfigured Dashboards
- 10. Filebeat Tail files Packetbeat Capture network packets Metricbeat System (Topbeat), Apache, Docker, HAProxy, MongoDB, MySQL, Nginx, PostgreSQL, Redis, Zookeeper Winlogbeat Windows event logs
- 11. apachebeat httpd server-status burrowbeat kafka consumer lag cassandrabeat nodetool cfstats cloudtrailbeat aws events dockerbeat container stats elasticbeat elasticsearch status execbeat shell cmd output flowbeat sflow factbeat puppet facter httpbeat poll http(s) endpoints jmxproxybeat tomcat jmx metrics lmsensorsbeat temp, voltage, fans mysqlbeat any query -> elasticsearch packagebeat dpkg, rpm pingbeat icmp ping w/ rtt redisbeat redis info twitterbeat tweets by screen name wmibeat windows monitoring https://www.elastic.co/guide/en/beats/libbeat/master/community-beats.html
- 13. DEPLOYING
- 14. Beats Log Files Metrics Wire Data Elasticsearch Kibana Master / Data Nodes (3) Ingest Nodes (X) your{beat} X-Pack X-Pack Simple Beats to Ingest Node Instances (X)
- 15. Beats Logstash Elasticsearch Kibana X-Pack X-Pack Additional Processing Nodes (X) Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes – Hot (X) Data Notes – Warm (X) Log Files Metrics Wire Data your{beat} LDAP Authentication AD Notification SSO
- 16. Deployment at Scale Kibana X-Pack Instances (X) Beats Log Files Metrics Wire Data Datastore Web APIs Social Sensors Kafka Redis Messaging Queue Logstash Elasticsearch Nodes (X) Master Nodes (3) Ingest Nodes (X) Data Nodes – Hot (X) Data Notes – Warm (X) your{beat} X-Pack LDAP Authentication AD Notification SSO
- 17. Beats Log Files Metrics Wire Data Datastore Web APIs Social Sensors Kafka Redis Messaging Queue Logstash ES-Hadoop Elasticsearch Kibana Nodes (X) Master Nodes (3) Ingest Nodes (X) Data Nodes – Hot (X) Data Notes – Warm (X) Instances (X) your{beat} X-Pack X-Pack Custom UI LDAP Authentication AD Notification SSO Deployment in the Enterprise Hadoop Ecosystem
- 18. HANDS ON