What's new in v11 - Fluentd Casual Talks #3 #fluentdcasual
•
9 likes•9,225 views
Fluentd version 11 includes several new features and improvements including non-stop restart capability, support for multiprocess architecture with separate worker processes, dedicated error stream handling, improved plugin version management, ability to set log levels for individual plugins, use of variables in configuration files, and streaming processing without needing to modify record tags.
Report
Share
Related slideshows
What's new in v11 - Fluentd Casual Talks #3 #fluentdcasual
- 1. What s new in v11 Sadayuki Furuhashi fluentd.org
- 2. Non-stop restart v10 (old): supervisor process listen TCP socket worker process heartbeat
- 3. Non-stop restart v10 (old): supervisor process Downtime! heartbeat worker process restart initializing...
- 4. Non-stop restart v11: supervisor process listen TCP socket worker process heartbeat
- 5. Non-stop restart Zero downtime! v11: supervisor process listen TCP socket heartbeat worker process restart initializing...
- 7. Multiprocess worker process <worker> <worker> ... </worker> <worker> ･ in_tail ･ in_forward ･ out_groupcounter ･ out_VERY_UNSTABLE ･ out_webhdfs </worker> </worker> Separate important logging from experimental plugins
- 8. Error stream v10 (old): <source> <match **> {“event”: 1, ...} chunk event OK {“event”: 2, ...} ERROR! {“event”: 3, ...} ? {“event”: 4, ...} chunk ERROR! {“event”: 5, ...} ? {“event”: 6, ...} ? Error handling depends on plugin (throw away! etc.)
- 9. Error stream v10 (old): <source> 特殊ラベル <match **> {“event”: 1, ...} event chunk OK {“event”: 2, ...} ERROR! {“event”: 3, ...} OK {“event”: 4, ...} chunk ERROR! {“event”: 5, ...} OK {“event”: 6, ...} OK <label @ERROR> <match **> ... OK OK Fluentd core handles error events
- 10. Plugin version management v11: $ cat /etc/td-agent/Gemfile gem “fluent-plugin-s3”, “~> 0.11.3” gem “fluent-plugin-webhdfs”, “~> 0.2.2” gem “fluentd”, “~> 0.11.2” $
- 11. Log level per plugin v11: <source> <match **> type forward type mongo log_level error log_level debug </source> </match>
- 12. Variables in a config file v11: <source> type forward port “#{ENV[‘FLUENTD_LISTEN_PORT’]}” </source>
- 13. Variables in a config file v11?： <server> @set port 9800 @include “/etc/fluentd/server.conf” </server> <server> @set port 9801 @include “/etc/fluentd/server.conf” </server>
- 14. Streaming processing without tag hacking v10 (old): <match raw.**> type flowcounter add_tag_prefix counted </match> <match counted.**> type growthforecast </match> v11: <match **> type flowcounter <match **> type growthforecast </match> </match>
- 15. v11 Reliability & Non-stop restart Performance Multiprocess Error stream Management Plugin version management Log level per plugins Configuration Variables in a config file Streaming processing without tag hacking