SlideShare a Scribd company logo

Elastic Meetup Belgium - December 2018

•Download as PPTX, PDF•
•
Arthur Eyckerman
Arthur Eyckerman

The document discusses a presentation given by Arthur Eyckerman on what's new in the Elastic Stack, including updates to Elasticsearch, Kibana, Logstash, Beats, and solutions. It provides an overview of recent features and acquisitions by Elastic as well as a history and timeline of the Elastic Stack. The presentation was given at Beta Cowork in Turnhout, Belgium.

1 of 73
{ ”host” : “data.be”, “topic” : “What’s new in the Elastic Stack”, “speaker” : “Arthur Eyckerman”, “location” : “@betacowork” }
Arthur Eyckerman Sr. Solutions Architect Living in Turnhout, center of NL | BE | LU [Consultant, Freelancer, Architect] Joined Elastic in May 2018 tuurleyck
Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018
ELK Feb 2010 Elasticsearch first release Kibana joins Logstash joins January 2013 August 2013 Feb 2014 Jan 2015 Feb 2015 May 2015 June 2015 Feb 2014 Sept 2016 Oct 2016 May 2017 May 2017 Sept 2017 Oct 2017 Feb 2018 April 2018 July 2017 May 2018 https://www.elastic.co/about/history-of-elasticsearch
Elasticsearch Heart of the Elastic Stack Distributed & Scalable Highly Available Multi-tenancy Developer Friendly Real-time, Full-text Search Aggregations
Logstash Data processing pipeline Ingest data of all shapes, sizes, and sources Parse and dynamically transform data Transport data to any output Secure and encrypt data inputs Build your own pipeline More than 200+ plugins
Kibana Window into the Elastic Stack Visualize and analyze Geospatial Customize and Share Reports Graph Exploration UX to secure and manage the Elastic Stack Build Custom Apps
Elastic Stack Feb 2010 Elasticsearch first release Kibana joins Logstash joins Monitoring feature first release (XP) Found acquired; now Elastic Cloud Packetbeat joins; Beats introduced Elasticsearch 1.0 January 2013 August 2013 Feb 2014 Jan 2015 Feb 2015 May 2015 June 2015 Feb 2014 Sept 2016 Prelert acquired for ML capabilities Elastic Stack 5.0 first release with X-Pack Elastic Cloud Enterprise (ECE) release Oct 2016 May 2017 May 2017 Sept 2017 Oct 2017 Feb 2018 April 2018 July 2017 May 2018 Security feature first release (XP) Alerting feature first release (XP) https://www.elastic.co/about/history-of-elasticsearch
Beats Lightweight data shippers Ship data from the source Ship and centralize in Elasticsearch Ship to Logstash for transformation and parsing Single command modules to visualize data Libbeat: API framework to build custom beats 70+ community Beats
Elastic Stack + Solutions Feb 2010 Elasticsearch first release Kibana joins Logstash joins Monitoring feature first release (XP) Found acquired; now Elastic Cloud Packetbeat joins; Beats introduced Elasticsearch 1.0 January 2013 August 2013 Feb 2014 Jan 2015 Feb 2015 May 2015 June 2015 Feb 2014 Sept 2016 Prelert acquired for ML capabilities Elastic Stack 5.0 first release with X-Pack Elastic Cloud Enterprise (ECE) release Opbeat acquired for APM GCP on Elastic Cloud release Swiftype acquired APM first release Opening of X- Pack code Machine learning first release (XP) Oct 2016 May 2017 May 2017 Sept 2017 Oct 2017 Feb 2018 April 2018 July 2017 App Search first release May 2018 Security feature first release (XP) Alerting feature first release (XP) https://www.elastic.co/about/history-of-elasticsearch
SaaS Elastic Cloud Metrics Logging APM Site Search App Search Business Analytics Elastic Stack Kibana Elasticsearch Beats Logstash Self Managed Elastic Cloud Enterprise Standalone Visualize & Manage Store, Search, & Analyze Ingest Enterprise Search Security Analytics Future Deployment Solutions
Elastic was in the perfect place to capitalize on this industry shift. Elasticsearch, together with a couple of smaller but strategically critical software components - Logstash and Kibana - were quickly becoming the industry standard known as the ELK Stack, and today called the Elastic Stack. What proved to be so useful about the Elastic Stack was its enormous flexibility and ease of use. At its heart, Elasticsearch is a search engine, which means that it can read structured and unstructured documents (often referred to as “schema- free” in the industry lingo), index them (an index is like a map of where the content rests), find the right content, and return a search result. What makes Elasticsearch special is the fact that it’s distributed - it allows clusters to act like one big system; it’s multi-tenant - so many developers of parts of an organization can use it without interfering with each other; and it has a programmable web (HTTP) interface - so that developers can write programs that query Elasticsearch rather than just respond to typed searches. https://www.indexventures.com/blog/elastic-the-evolution-of-open-source TL;DR - Elastic -- The Evolution of Open Source
Community, Open Source .. and SaaS What do you expect ?
Why use open source ? • Lots of resources - discuss.elastic.co • Open Code - github.com/elastic • Meetups - and great people • Free, no license required. • Get started now. • Your laptop looks better with stickers !
Why pay for open source ?
Elastic Meetup Belgium - December 2018
Open X-Pack
Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018
22
23
Elastic Meetup Belgium - December 2018
Elastic Meetup Belgium - December 2018
10 DECEMBER 2018 NEWS Elastic doubles down on cloud native with Helm charts and CNCF membership docker.elastic.co https://www.elastic.co/blog/alpha-helm-charts-for-elasticsearch-kibana-and-cncf-membership https://raw.githubusercontent.com/cncf/trailmap/master/CNCF_TrailMap_latest.png helm.elastic.co Logs Metrics APM Distributed & Scalable Joining Cloud Native Computing Foundation (CNCF) is a natural evolution for us, given our open source history of building products and ongoing efforts around integrating with cloud native technologies.
Helm Charts To get started with Elasticsearch and Kibana Helm charts: 1. Add the Elastic Helm Chart Repo: helm repo add elastic https://helm.elastic.co 2. Install Elasticsearch: helm install --name elasticsearch elastic/elasticsearch 3. Install Kibana: helm install --name kibana elastic/kibana
Bring Events + Traces + Metrics together for 100% Observability in IT Operations TRACES (APM) EVENT S METRICS
Elastic Meetup Belgium - December 2018
Getting started with Elasticsearch Everybody loves CURL. Right ? # When you need to index a lot of docs, you should use the bulk API curl -XPOST "https://39d98b388e194025888558780eaf266b.europe-west1.gcp.cloud.es.io:9244/inspections/_doc/_bulk" -H 'Content-Type: application/json' -d’ { "index": { "_id": 1 }} {"business_address":"315 California St","business_city":"San Francisco","business_id":"24936","business_latitude":"37.793199","business_location":{"type":"Point","coordinates":[-122.400152,37.793199]},"business_longitude":"- 122.400152","business_name":"San Francisco Soup Company","business_postal_code":"94104","business_state":"CA","inspection_date":"2016-06- 09T00:00:00.000","inspection_id":"24936_20160609","inspection_score":77,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Improper food labeling or menu misrepresentation","violation_id":"24936_20160609_103141"} { "index": { "_id": 2 }} {"business_address":"10 Mason St","business_city":"San Francisco","business_id":"60354","business_latitude":"37.783527","business_location":{"type":"Point","coordinates":[-122.409061,37.783527]},"business_longitude":"- 122.409061","business_name":"Soup Unlimited","business_postal_code":"94102","business_state":"CA","inspection_date":"2016-11- 23T00:00:00.000","inspection_id":"60354_20161123","inspection_type":"Routine", "inspection_score": 95} { "index": { "_id": 3 }} {"business_address":"2872 24th St","business_city":"San Francisco","business_id":"1797","business_latitude":"37.752807","business_location":{"type":"Point","coordinates":[-122.409752,37.752807]},"business_longitude":"- 122.409752","business_name":"TIO CHILOS GRILL","business_postal_code":"94110","business_state":"CA","inspection_date":"2016-07- 05T00:00:00.000","inspection_id":"1797_20160705","inspection_score":90,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unclean nonfood contact surfaces","violation_id":"1797_20160705_103142"} { "index": { "_id": 4 }} {"business_address":"1661 Tennessee St Suite 3B","business_city":"San Francisco Whard Restaurant","business_id":"66198","business_latitude":"37.75072","business_location":{"type":"Point","coordinates":[-122.388478,37.75072]},"business_longitude":"- 122.388478","business_name":"San Francisco Restaurant","business_postal_code":"94107","business_state":"CA","inspection_date":"2016-05- 27T00:00:00.000","inspection_id":"66198_20160527","inspection_type":"Routine","inspection_score":56 } { "index": { "_id": 5 }} {"business_address":"2162 24th Ave","business_city":"San Francisco","business_id":"5794","business_latitude":"37.747228","business_location":{"type":"Point","coordinates":[-122.481299,37.747228]},"business_longitude":"- 122.481299","business_name":"Soup House","business_phone_number":"+14155752700","business_postal_code":"94116","business_state":"CA","inspection_date":"2016-09- 07T00:00:00.000","inspection_id":"5794_20160907","inspection_score":96,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unapproved or unmaintained equipment or utensils","violation_id":"5794_20160907_103144"} { "index": { "_id": 6 }} {"business_address":"2162 24th Ave","business_city":"San Francisco","business_id":"5794","business_latitude":"37.747228","business_location":{"type":"Point","coordinates":[-122.481299,37.747228]},"business_longitude":"- 122.481299","business_name":"Soup-or-Salad","business_phone_number":"+14155752700","business_postal_code":"94116","business_state":"CA","inspection_date":"2016-09- 07T00:00:00.000","inspection_id":"5794_20160907","inspection_score":96,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unapproved or unmaintained equipment or utensils","violation_id":"5794_20160907_103144"}'
Data Visualizer for Files Experimental | Basic (free) To get the most out of Machine Learning you need to understand your data Data Visualizer now provides the ability to upload files (up to 100MB) and use ML File Structure API for field identification and to create indices and index patterns Works with CSV, text delimited, or JSON files
Elasticsearch
ODBC REST Translate API CLI JDBC ODBC SQL Endpoint Basic (free)
sql> SELECT description FROM features WHERE product = 'elasticsearch' AND version >= '6.3.0'; description --------------- SQL Data Rollups Java 10 & 11 Support
• API for creating an Elasticsearch process to periodically store aggregate statistics • Primary benefit is space savings ‒ Faster queries ‒ Potentially less nodes to manage ‒ Smaller snapshots ‒ Longer retention times ‒ etc. • Query rolled up data and “live” data together in a single query. Data Rollups Basic (free)
Raw Minute Hour Day Docs: 9,041,000 1,448,285 49,554 8,447 Size: 2.23gb 1.25gb 48.40mb 9.10mb Docs % change: -83.98% -99.45% -99.91% Size % change: -43.68% -97.84% -99.59% (avg ~200 docs per minute, 32 days of data, single host) (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics) Rolling up Metricbeat dataRolling up metricbeat data
Data Rollups API // Define rollup job, start/stop /_xpack/rollup/job/{job_id} /_xpack/rollup/job/{job_id}/_start /_xpack/rollup/job/{job_id}/_stop // rollup jobs configured for an index or index pattern /_xpack/rollup/data/{index}/ // Search endpoint using standard ES Query DSL /{index}/_rollup_search
Cluster 1 Cluster 2 _source Only Snapshot Basic (free) _source only snapshots can be 50% smaller than full snapshots Requires a reindex to make the data searchable again Trades off restore time for smaller storage space / costs source only snapshot full snapshot Snapshot Restore to a different cluster Restore to the same cluster
Java Updates OSS Adds support for Java 11 Adds support for G1 garbage collector (supported on Java 10+) Even with G1, we don’t recommend extremely large heaps
Kibana
Canvas: Create live pixel-perfect presentations Beta | Basic (free)
Canvas: Create live pixel-perfect presentations Beta | Basic (free)
Canvas Beta | Basic (free) Showcase your data, live & pixel-perfect Tell the story of your data, your way Showcase your work on the big screen Support Elasticsearch SQL Automate reporting (adios copy/paste) Built for the community, extend Canvas with plugins
Spaces Basic (free) / Gold Organize Kibana visuals, dashboards, etc into separate, independent spaces Control user access to spaces using role-based access control Simplify Kibana multitenant use Use Cases: ● Organization ● Phasing (dev, stage, prod, etc) ● Security (restrict access)
Rollups in Kibana Beta | Basic (free) Automatically roll up data into coarser time buckets as it ages - Save on storage space & costs - Smaller indices = faster analytics 6.3 - Rollups API in Elasticsearch 6.5 - Rollups support in Kibana - Job management UI - Visualize rolled up indices Aggregation functions: - Avg, min, max, sum, count
Beats
Beats Central Management Beta | Gold Centrally manage your fleet of Beats • Enroll & unenroll Beats • Add, modify & delete configs Manage via UI and APIs Currently supports: • Filebeat (inputs, modules) • Metricbeat (modules) Metricbeat Metricbeat Filebeat Filebeat Filebeat Beats Admin Metricbeat
Beats Central Management Beta | Gold Centrally manage your fleet of Beats • Enroll & unenroll Beats • Add, modify & delete configs Manage via UI and APIs Currently supports: • Filebeat (inputs, modules) • Metricbeat (modules)
Functionbeat Beta | Basic (free) New Beat type that deploys as a function in serverless platforms Easily collect and stream cloud data to Elasticsearch 6.5 release supports AWS Lambda Collect events from Cloudwatch Logs, SQS, and more to come soon Automate via single AWS CLI command
Heartbeat GA OSS New config reload provide an easy way to create, update, delete Heartbeat monitors Autodiscovery for Docker & Kubernetes Add Data tutorials in Kibana
Logstash
Multiple Pipelines● Building on Multiple Pipelines ● Reduce repetition, isolate blockages and simplify pipelines ● Usage patterns (also in docs!) ○ Distributor ○ Output Isolator ○ Forked Path ○ Collector ● Don’t create cycles Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline Logstash Inter-pipeline Communication Beta
Inter-connected Pipelines: Distributor Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline Distributor Pipeline OR OR Logstash Inter-pipeline Communication Beta
Inter-connected Pipelines: Output Isolation Logstash S3-Out Pipeline HTTP-Out Pipeline ES-Out Pipeline Apache Pipeline AND AND Logstash Inter-pipeline Communication Beta
Inter-connected Pipelines: Forking Logstash Redacted Pipeline Normal Pipeline Apache Pipeline AND Logstash Inter-pipeline Communication Beta
Inter-connected Pipelines: Collector (Common Output) Logstash Inter-pipeline Communication Beta Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline Output Pipeline
Azure Monitoring Module OSS • Easily monitor your Azure deployments with the new Azure monitoring module • Consume from Azure Event Hubs with a new input plugin ‒ Basic configuration generally recommended ‒ Advanced configuration enables more granular tuning when consuming from multiple Event Hubs • Amazon Linux OS support input { azure_event_hubs { event_hub_connections => ["Endpoint=sb:// threads => 8 decorate_events => true consumer_group => "logstash" storage_connection => "DefaultEndpointsPr } }
SNMP Input Plugin OSS Centrally poll SNMP devices like routers, switches, phones, and printers. Great for network monitoring & management use cases Support SNMP v1, v2c, v3 over TCP/UDP Servers, Switches, Routers, Gateways, etc SNMP Agents
Infrastructure UI
Infrastructure Solution Beta | Basic (free) Curated experience for infra operators Bird’s eye view of 10K+ infra elements Native support for Kubernetes, Docker Drill down to metrics, logs, APM views Ad hoc and structured search
Logs UI
Logs Solution Beta | Basic (free) Compact log viewer optimized for live log event troubleshooting Console-like display Live log streaming (like tail -f) Infinite scroll for historical logs Ad hoc and structured search
APM
Supported Languages & Frameworks
Track key application metrics 65 ● Response time for requests ● Unhandled errors & exceptions ● Visualize call hierarchy (waterfall chart) ● Identify code bottlenecks ● Drill down to the code level
66 APM App (UI) ● Tailored for APM ● Made for developers ● Free (Basic license) ● Integrated Search Bar
Mix APM with other data & features 67 ● APM data is just another Elasticsearch index ● Customize dashboards with other visuals to show what YOU want ● Mix with other Elastic Stack features, such as machine learning, alerting… ● Built-in integration with ML & Alerting
RUM (Real User Monitoring) The only way to measure actual end-user experience Browser interactionAgent Response from web server Agent Request to web serverAgent Browser renderingAgent Time to interaction: ~6s
69 RUM ● See where the browser spends its time ● Similar waterfall view ● Annotations at key DOM events
Java & Go Agents GA OSS Java support: Java 7-10, Several technologies/frameworks: Servlet API, Spring Web MVC, Spring Boot, Tomcat, WildFly, Jetty, Websphere, JDBC & more Go support: Go 1.8+, httprouter, Echo, Gin,gorilla/mux, database/sql, GORM, gocql, gRPC
Distributed Tracing Beta | Basic (free) Consolidated waterfall showing all instrumented services Ability to jump to full trace context when in a sub-context OpenTracing compatible
Elastic APM Monitoring Basic (free) Track the health of your Elastic APM deployments Monitoring stats include rates, events, and requests.
Become an Elastic Pioneer 7.0 is shipping with many highly requested features, and we welcome your testing and feedback during the preview cycles. When our community gets involved, good things happen. Install a 7.0 preview version, try new features, and provide feedback via GitHub to become an Elastic Pioneer. Learn more about the Elastic Pioneer Program. Oh, and Elastic Pioneers may win some limited edition Elastic swag (and bragging rights, of course).
Elastic Meetup Belgium - December 2018

More Related Content

Elastic Meetup Belgium - December 2018

  • 1. { ”host” : “data.be”, “topic” : “What’s new in the Elastic Stack”, “speaker” : “Arthur Eyckerman”, “location” : “@betacowork” }
  • 2. Arthur Eyckerman Sr. Solutions Architect Living in Turnhout, center of NL | BE | LU [Consultant, Freelancer, Architect] Joined Elastic in May 2018 tuurleyck
  • 6. Elasticsearch Heart of the Elastic Stack Distributed & Scalable Highly Available Multi-tenancy Developer Friendly Real-time, Full-text Search Aggregations
  • 7. Logstash Data processing pipeline Ingest data of all shapes, sizes, and sources Parse and dynamically transform data Transport data to any output Secure and encrypt data inputs Build your own pipeline More than 200+ plugins
  • 8. Kibana Window into the Elastic Stack Visualize and analyze Geospatial Customize and Share Reports Graph Exploration UX to secure and manage the Elastic Stack Build Custom Apps
  • 9. Elastic Stack Feb 2010 Elasticsearch first release Kibana joins Logstash joins Monitoring feature first release (XP) Found acquired; now Elastic Cloud Packetbeat joins; Beats introduced Elasticsearch 1.0 January 2013 August 2013 Feb 2014 Jan 2015 Feb 2015 May 2015 June 2015 Feb 2014 Sept 2016 Prelert acquired for ML capabilities Elastic Stack 5.0 first release with X-Pack Elastic Cloud Enterprise (ECE) release Oct 2016 May 2017 May 2017 Sept 2017 Oct 2017 Feb 2018 April 2018 July 2017 May 2018 Security feature first release (XP) Alerting feature first release (XP) https://www.elastic.co/about/history-of-elasticsearch
  • 10. Beats Lightweight data shippers Ship data from the source Ship and centralize in Elasticsearch Ship to Logstash for transformation and parsing Single command modules to visualize data Libbeat: API framework to build custom beats 70+ community Beats
  • 11. Elastic Stack + Solutions Feb 2010 Elasticsearch first release Kibana joins Logstash joins Monitoring feature first release (XP) Found acquired; now Elastic Cloud Packetbeat joins; Beats introduced Elasticsearch 1.0 January 2013 August 2013 Feb 2014 Jan 2015 Feb 2015 May 2015 June 2015 Feb 2014 Sept 2016 Prelert acquired for ML capabilities Elastic Stack 5.0 first release with X-Pack Elastic Cloud Enterprise (ECE) release Opbeat acquired for APM GCP on Elastic Cloud release Swiftype acquired APM first release Opening of X- Pack code Machine learning first release (XP) Oct 2016 May 2017 May 2017 Sept 2017 Oct 2017 Feb 2018 April 2018 July 2017 App Search first release May 2018 Security feature first release (XP) Alerting feature first release (XP) https://www.elastic.co/about/history-of-elasticsearch
  • 12. SaaS Elastic Cloud Metrics Logging APM Site Search App Search Business Analytics Elastic Stack Kibana Elasticsearch Beats Logstash Self Managed Elastic Cloud Enterprise Standalone Visualize & Manage Store, Search, & Analyze Ingest Enterprise Search Security Analytics Future Deployment Solutions
  • 13. Elastic was in the perfect place to capitalize on this industry shift. Elasticsearch, together with a couple of smaller but strategically critical software components - Logstash and Kibana - were quickly becoming the industry standard known as the ELK Stack, and today called the Elastic Stack. What proved to be so useful about the Elastic Stack was its enormous flexibility and ease of use. At its heart, Elasticsearch is a search engine, which means that it can read structured and unstructured documents (often referred to as “schema- free” in the industry lingo), index them (an index is like a map of where the content rests), find the right content, and return a search result. What makes Elasticsearch special is the fact that it’s distributed - it allows clusters to act like one big system; it’s multi-tenant - so many developers of parts of an organization can use it without interfering with each other; and it has a programmable web (HTTP) interface - so that developers can write programs that query Elasticsearch rather than just respond to typed searches. https://www.indexventures.com/blog/elastic-the-evolution-of-open-source TL;DR - Elastic -- The Evolution of Open Source
  • 14. Community, Open Source .. and SaaS What do you expect ?
  • 15. Why use open source ? • Lots of resources - discuss.elastic.co • Open Code - github.com/elastic • Meetups - and great people • Free, no license required. • Get started now. • Your laptop looks better with stickers !
  • 16. Why pay for open source ?
  • 21. 22
  • 22. 23
  • 25. 10 DECEMBER 2018 NEWS Elastic doubles down on cloud native with Helm charts and CNCF membership docker.elastic.co https://www.elastic.co/blog/alpha-helm-charts-for-elasticsearch-kibana-and-cncf-membership https://raw.githubusercontent.com/cncf/trailmap/master/CNCF_TrailMap_latest.png helm.elastic.co Logs Metrics APM Distributed & Scalable Joining Cloud Native Computing Foundation (CNCF) is a natural evolution for us, given our open source history of building products and ongoing efforts around integrating with cloud native technologies.
  • 26. Helm Charts To get started with Elasticsearch and Kibana Helm charts: 1. Add the Elastic Helm Chart Repo: helm repo add elastic https://helm.elastic.co 2. Install Elasticsearch: helm install --name elasticsearch elastic/elasticsearch 3. Install Kibana: helm install --name kibana elastic/kibana
  • 27. Bring Events + Traces + Metrics together for 100% Observability in IT Operations TRACES (APM) EVENT S METRICS
  • 29. Getting started with Elasticsearch Everybody loves CURL. Right ? # When you need to index a lot of docs, you should use the bulk API curl -XPOST "https://39d98b388e194025888558780eaf266b.europe-west1.gcp.cloud.es.io:9244/inspections/_doc/_bulk" -H 'Content-Type: application/json' -d’ { "index": { "_id": 1 }} {"business_address":"315 California St","business_city":"San Francisco","business_id":"24936","business_latitude":"37.793199","business_location":{"type":"Point","coordinates":[-122.400152,37.793199]},"business_longitude":"- 122.400152","business_name":"San Francisco Soup Company","business_postal_code":"94104","business_state":"CA","inspection_date":"2016-06- 09T00:00:00.000","inspection_id":"24936_20160609","inspection_score":77,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Improper food labeling or menu misrepresentation","violation_id":"24936_20160609_103141"} { "index": { "_id": 2 }} {"business_address":"10 Mason St","business_city":"San Francisco","business_id":"60354","business_latitude":"37.783527","business_location":{"type":"Point","coordinates":[-122.409061,37.783527]},"business_longitude":"- 122.409061","business_name":"Soup Unlimited","business_postal_code":"94102","business_state":"CA","inspection_date":"2016-11- 23T00:00:00.000","inspection_id":"60354_20161123","inspection_type":"Routine", "inspection_score": 95} { "index": { "_id": 3 }} {"business_address":"2872 24th St","business_city":"San Francisco","business_id":"1797","business_latitude":"37.752807","business_location":{"type":"Point","coordinates":[-122.409752,37.752807]},"business_longitude":"- 122.409752","business_name":"TIO CHILOS GRILL","business_postal_code":"94110","business_state":"CA","inspection_date":"2016-07- 05T00:00:00.000","inspection_id":"1797_20160705","inspection_score":90,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unclean nonfood contact surfaces","violation_id":"1797_20160705_103142"} { "index": { "_id": 4 }} {"business_address":"1661 Tennessee St Suite 3B","business_city":"San Francisco Whard Restaurant","business_id":"66198","business_latitude":"37.75072","business_location":{"type":"Point","coordinates":[-122.388478,37.75072]},"business_longitude":"- 122.388478","business_name":"San Francisco Restaurant","business_postal_code":"94107","business_state":"CA","inspection_date":"2016-05- 27T00:00:00.000","inspection_id":"66198_20160527","inspection_type":"Routine","inspection_score":56 } { "index": { "_id": 5 }} {"business_address":"2162 24th Ave","business_city":"San Francisco","business_id":"5794","business_latitude":"37.747228","business_location":{"type":"Point","coordinates":[-122.481299,37.747228]},"business_longitude":"- 122.481299","business_name":"Soup House","business_phone_number":"+14155752700","business_postal_code":"94116","business_state":"CA","inspection_date":"2016-09- 07T00:00:00.000","inspection_id":"5794_20160907","inspection_score":96,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unapproved or unmaintained equipment or utensils","violation_id":"5794_20160907_103144"} { "index": { "_id": 6 }} {"business_address":"2162 24th Ave","business_city":"San Francisco","business_id":"5794","business_latitude":"37.747228","business_location":{"type":"Point","coordinates":[-122.481299,37.747228]},"business_longitude":"- 122.481299","business_name":"Soup-or-Salad","business_phone_number":"+14155752700","business_postal_code":"94116","business_state":"CA","inspection_date":"2016-09- 07T00:00:00.000","inspection_id":"5794_20160907","inspection_score":96,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unapproved or unmaintained equipment or utensils","violation_id":"5794_20160907_103144"}'
  • 30. Data Visualizer for Files Experimental | Basic (free) To get the most out of Machine Learning you need to understand your data Data Visualizer now provides the ability to upload files (up to 100MB) and use ML File Structure API for field identification and to create indices and index patterns Works with CSV, text delimited, or JSON files
  • 33. sql> SELECT description FROM features WHERE product = 'elasticsearch' AND version >= '6.3.0'; description --------------- SQL Data Rollups Java 10 & 11 Support
  • 34. • API for creating an Elasticsearch process to periodically store aggregate statistics • Primary benefit is space savings ‒ Faster queries ‒ Potentially less nodes to manage ‒ Smaller snapshots ‒ Longer retention times ‒ etc. • Query rolled up data and “live” data together in a single query. Data Rollups Basic (free)
  • 35. Raw Minute Hour Day Docs: 9,041,000 1,448,285 49,554 8,447 Size: 2.23gb 1.25gb 48.40mb 9.10mb Docs % change: -83.98% -99.45% -99.91% Size % change: -43.68% -97.84% -99.59% (avg ~200 docs per minute, 32 days of data, single host) (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics) Rolling up Metricbeat dataRolling up metricbeat data
  • 36. Data Rollups API // Define rollup job, start/stop /_xpack/rollup/job/{job_id} /_xpack/rollup/job/{job_id}/_start /_xpack/rollup/job/{job_id}/_stop // rollup jobs configured for an index or index pattern /_xpack/rollup/data/{index}/ // Search endpoint using standard ES Query DSL /{index}/_rollup_search
  • 37. Cluster 1 Cluster 2 _source Only Snapshot Basic (free) _source only snapshots can be 50% smaller than full snapshots Requires a reindex to make the data searchable again Trades off restore time for smaller storage space / costs source only snapshot full snapshot Snapshot Restore to a different cluster Restore to the same cluster
  • 38. Java Updates OSS Adds support for Java 11 Adds support for G1 garbage collector (supported on Java 10+) Even with G1, we don’t recommend extremely large heaps
  • 40. Canvas: Create live pixel-perfect presentations Beta | Basic (free)
  • 41. Canvas: Create live pixel-perfect presentations Beta | Basic (free)
  • 42. Canvas Beta | Basic (free) Showcase your data, live & pixel-perfect Tell the story of your data, your way Showcase your work on the big screen Support Elasticsearch SQL Automate reporting (adios copy/paste) Built for the community, extend Canvas with plugins
  • 43. Spaces Basic (free) / Gold Organize Kibana visuals, dashboards, etc into separate, independent spaces Control user access to spaces using role-based access control Simplify Kibana multitenant use Use Cases: ● Organization ● Phasing (dev, stage, prod, etc) ● Security (restrict access)
  • 44. Rollups in Kibana Beta | Basic (free) Automatically roll up data into coarser time buckets as it ages - Save on storage space & costs - Smaller indices = faster analytics 6.3 - Rollups API in Elasticsearch 6.5 - Rollups support in Kibana - Job management UI - Visualize rolled up indices Aggregation functions: - Avg, min, max, sum, count
  • 45. Beats
  • 46. Beats Central Management Beta | Gold Centrally manage your fleet of Beats • Enroll & unenroll Beats • Add, modify & delete configs Manage via UI and APIs Currently supports: • Filebeat (inputs, modules) • Metricbeat (modules) Metricbeat Metricbeat Filebeat Filebeat Filebeat Beats Admin Metricbeat
  • 47. Beats Central Management Beta | Gold Centrally manage your fleet of Beats • Enroll & unenroll Beats • Add, modify & delete configs Manage via UI and APIs Currently supports: • Filebeat (inputs, modules) • Metricbeat (modules)
  • 48. Functionbeat Beta | Basic (free) New Beat type that deploys as a function in serverless platforms Easily collect and stream cloud data to Elasticsearch 6.5 release supports AWS Lambda Collect events from Cloudwatch Logs, SQS, and more to come soon Automate via single AWS CLI command
  • 49. Heartbeat GA OSS New config reload provide an easy way to create, update, delete Heartbeat monitors Autodiscovery for Docker & Kubernetes Add Data tutorials in Kibana
  • 51. Multiple Pipelines● Building on Multiple Pipelines ● Reduce repetition, isolate blockages and simplify pipelines ● Usage patterns (also in docs!) ○ Distributor ○ Output Isolator ○ Forked Path ○ Collector ● Don’t create cycles Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline Logstash Inter-pipeline Communication Beta
  • 52. Inter-connected Pipelines: Distributor Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline Distributor Pipeline OR OR Logstash Inter-pipeline Communication Beta
  • 53. Inter-connected Pipelines: Output Isolation Logstash S3-Out Pipeline HTTP-Out Pipeline ES-Out Pipeline Apache Pipeline AND AND Logstash Inter-pipeline Communication Beta
  • 54. Inter-connected Pipelines: Forking Logstash Redacted Pipeline Normal Pipeline Apache Pipeline AND Logstash Inter-pipeline Communication Beta
  • 55. Inter-connected Pipelines: Collector (Common Output) Logstash Inter-pipeline Communication Beta Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline Output Pipeline
  • 56. Azure Monitoring Module OSS • Easily monitor your Azure deployments with the new Azure monitoring module • Consume from Azure Event Hubs with a new input plugin ‒ Basic configuration generally recommended ‒ Advanced configuration enables more granular tuning when consuming from multiple Event Hubs • Amazon Linux OS support input { azure_event_hubs { event_hub_connections => ["Endpoint=sb:// threads => 8 decorate_events => true consumer_group => "logstash" storage_connection => "DefaultEndpointsPr } }
  • 57. SNMP Input Plugin OSS Centrally poll SNMP devices like routers, switches, phones, and printers. Great for network monitoring & management use cases Support SNMP v1, v2c, v3 over TCP/UDP Servers, Switches, Routers, Gateways, etc SNMP Agents
  • 59. Infrastructure Solution Beta | Basic (free) Curated experience for infra operators Bird’s eye view of 10K+ infra elements Native support for Kubernetes, Docker Drill down to metrics, logs, APM views Ad hoc and structured search
  • 61. Logs Solution Beta | Basic (free) Compact log viewer optimized for live log event troubleshooting Console-like display Live log streaming (like tail -f) Infinite scroll for historical logs Ad hoc and structured search
  • 62. APM
  • 63. Supported Languages & Frameworks
  • 64. Track key application metrics 65 ● Response time for requests ● Unhandled errors & exceptions ● Visualize call hierarchy (waterfall chart) ● Identify code bottlenecks ● Drill down to the code level
  • 65. 66 APM App (UI) ● Tailored for APM ● Made for developers ● Free (Basic license) ● Integrated Search Bar
  • 66. Mix APM with other data & features 67 ● APM data is just another Elasticsearch index ● Customize dashboards with other visuals to show what YOU want ● Mix with other Elastic Stack features, such as machine learning, alerting… ● Built-in integration with ML & Alerting
  • 67. RUM (Real User Monitoring) The only way to measure actual end-user experience Browser interactionAgent Response from web server Agent Request to web serverAgent Browser renderingAgent Time to interaction: ~6s
  • 68. 69 RUM ● See where the browser spends its time ● Similar waterfall view ● Annotations at key DOM events
  • 69. Java & Go Agents GA OSS Java support: Java 7-10, Several technologies/frameworks: Servlet API, Spring Web MVC, Spring Boot, Tomcat, WildFly, Jetty, Websphere, JDBC & more Go support: Go 1.8+, httprouter, Echo, Gin,gorilla/mux, database/sql, GORM, gocql, gRPC
  • 70. Distributed Tracing Beta | Basic (free) Consolidated waterfall showing all instrumented services Ability to jump to full trace context when in a sub-context OpenTracing compatible
  • 71. Elastic APM Monitoring Basic (free) Track the health of your Elastic APM deployments Monitoring stats include rates, events, and requests.
  • 72. Become an Elastic Pioneer 7.0 is shipping with many highly requested features, and we welcome your testing and feedback during the preview cycles. When our community gets involved, good things happen. Install a 7.0 preview version, try new features, and provide feedback via GitHub to become an Elastic Pioneer. Learn more about the Elastic Pioneer Program. Oh, and Elastic Pioneers may win some limited edition Elastic swag (and bragging rights, of course).