2020 07-30 elastic agent + ingest management
•
0 likes•285 views
This document provides an agenda and overview for a technical deep dive presentation on Elastic Agent and Ingest Manager. The presentation will include demos of enrolling an agent, collecting data out of the box, and configuring data collection. It will also provide a technical overview of components like the new indexing strategy, Elastic Agent, config options, and the Elastic Package Registry for centralized package management. Questions from the audience will be taken at the end.
Report
Share
2020 07-30 elastic agent + ingest management
- 1. Nicolas Ruflin (@ruflin) Engineer at Elastic 2020-07-30 Technical deep dive into Elastic Agent + Ingest Manager
- 2. Agenda • Teaser on all the parts that are new • Demo • Technical overview • Questions / Discussions
- 3. About me • Engineer at Elastic for 5 years • Tech Lead Ingest Management project • Switzerland based • @ruflin
- 4. Reimagine the way how to ingest data into the Elastic Stack
- 6. Beats to Elastic Agent
- 7. Heartbeat Winlogbeat Metricbeat Filebeat Security APM Many Beats & agents today Elastic Agent Easier to install Easier to upgrade Easier to scale One unified Agent tomorrow
- 9. Beats modules today Integrations tomorrow ● One click, right in Kibana ● Recommendations ● Share with the community
- 10. Config UI
- 11. Configuration files today Configuration UI tomorrow ● Minimal input ● Out of the box defaults ● Logs & metrics combined
- 13. Config management today You’re on your own... ● Powershell / Bash ● Ansible ● Puppet ● Chef ● ServiceNow Fleet central mgmt tomorrow ● Configuration updates automatically ● Binaries update automatically ● View status of Agents
- 14. Password to API Keys
- 15. User/pass today Keys tomorrow ● No need to input user/pass ● More secure
- 16. Demo
- 17. Demo • Enroll agent • Show collecting out of the box • Collect nginx data • Everything is API based
- 19. Our components
- 22. Current Indexing strategy • filebeat-*, metricbeat-* • Beats load all the assets ‒ Index Template ‒ ILM Policy
- 23. Problems with the current strategy • Too many fields • ILM only for logs or metrics • Query always on all data • Bootstrapping tricky • User modifications break things
- 24. New Indexing Strategy • {type}-{dataset}-{namespace} • type: generic type of data (logs, metrics, traces, ...) • dataset: Set of data with the same structure / mapping (nginx.access) • namespace: Use configurable namespace (prod, testing) • Example: logs-nginx.access-default • Default: logs-generic-default
- 25. Indexing Strategy: Bootstrapping and Templates • Generic templates ‒ logs-*-*, metrics-*-* ‒ ECS Based ‒ Loaded by Elasticsearch • Dataset specific templates ‒ logs-nginx.access-* ‒ Ingest pipeline attached to index ‒ Loaded by Ingest Manager
- 26. Indexing Strategy: Elasticsearch Features • Problem: Could not be built on the existing Elasticsearch features • New features ‒ constant_keywords ‒ component templates ‒ Data streams • More details on these in a separate talk
- 27. Elastic Agent One agent to rule them all
- 28. Elastic Agent
- 29. Elastic Agent • Runs and manages processes ‒ Today: Metricbeat, Filebeat, Endpoint ‒ Potential future: *beat, Apm-Server • Communication through GRPC • Two modes ‒ Managed by Fleet ‒ Standalone • Unified configuration for all inputs • Upgrade
- 31. Agent Config
- 32. Agent Config inputs: - type: nginx/metrics hosts: http://127.0.0.1 streams: - dataset.name: nginx.stub_status metricset: stub_status - type: logs streams: - dataset.name: nginx.access paths: /var/log/nginx/access.log* - dataset.name: nginx.error paths: /var/log/nginx/error.log*
- 33. Elastic Package Registry Central place for all packages
- 34. Package Registry
- 36. Packages
- 37. Packages • Contains assets and configs for the Elastic Stack • Package manager knows how to install, upgrade, remove each asset • Delivered as .tar.gz file • Logs, metrics together in one package
- 38. Package Structure: Supported Assets Today • Elasticsearch ‒ Index Template v2 ‒ ILM Policy ‒ Ingest Pipeline • Kibana ‒ Dashboard ‒ Visualization, Map ‒ Search ‒ Index Pattern • Agent ‒ Stream Template Future • Elasticsearch ‒ ML Job ‒ Data • Kibana ‒ Alert ‒ Action ‒ Canvas Template ‒ SIEM rules Note: Any asset type in the Stack can be added to this list
- 39. Config Builder
- 40. Packages
- 41. Agent Config: You can build your own config UI # Stream template dataset.name:nginx.access {{#each paths}} paths: "{{this}}" {{/each}} exclude_files: [".gz$"] processors: - add_locale: ~ # dataset manifest streams: - input: logs title: ... description: ... template_path: stream.yml.hbs vars: - name: paths required: true default: - /var/../access.log* # Stream config - dataset: nginx.access paths: - /var/log/nginx/access.log* exclude_files: - .gz$ processors: - add_locale: ~ + = Note: In case there is a shared config across streams, the package manifest has vars # package manifest datasources: - name: nginx inputs: - type: logs
- 42. Agent Config Built inputs: - type: nginx/metrics hosts: http://127.0.0.1 streams: - dataset.name: nginx.stub_status metricset: stub_status - type: logs streams: - dataset.name: nginx.access paths: /var/log/nginx/access.log* - dataset.name: nginx.error paths: /var/log/nginx/error.log*
- 43. FAQ
- 44. FAQ • How can I start using this? ‒ Start 7.8 Cloud cluster with xpack.ingestManager.enabled: true ‒ Download 7.8 Elastic Agent ‒ Soon: Run 7.9 • Do you have an API? ‒ Yes, it is all API based • Indexing strategy ‒ Can I use it also without agent? Yes
- 45. Try it out!