SlideShare a Scribd company logo

Plugins by tagomoris #fluentdcasual

•
•
SATOSHI TAGOMORI
SATOSHI TAGOMORI

The document summarizes updates to Tagomoris' Fluentd cluster and new Fluentd plugins. It states that the cluster has grown from 89 processes on 12 nodes to 92,000 messages per second and 184Mbps, and the number of plugins has increased from 10 to 19, including new plugins for routing, monitoring, filtering, and outputting data. It promotes developing plugins to expand Fluentd's capabilities for real-time data processing.

1 of 30
Download to read offline
Plugins by tagomoris Fluentd Casual Talks LT 2012/05/18 @tagomoris (NHN Japan Corp.)
@tagomoris NHN Japan Corp. Web Service Business Division Development Department 2 Development Platforms
Appendix diff of 2012/02 and now • Our Fluentd cluster status
Fluentd Cluster status updates (1) 89 Fluentd processes 12 nodes 89 Fluentd processes 12 nodes
Fluentd Cluster status updates (2) ruby 1.9.2 + glibc malloc scribeline ruby 1.9.3 + jemalloc fluent-agent-lite
Fluentd Cluster status updates (3) from 127 servers 146 log streams from 205 servers 246 log streams
Fluentd Cluster status updates (4) 70,000 msgs/sec 120Mbps (at peak time) 92,000 msgs/sec 184Mbps
Fluentd Cluster status updates (5) 650GB/day (non-compressed bytes on HDFS) 995GB/day
Fluentd Cluster status updates (6) Before: 10 plugins in_forward, out_forward, in_scribe, out_scribe out_copy, out_roundrobin, out_exec_filter out_hoop out_flowcounter, out_growthforecast After: +9 plugins out_route, in_gc_stat, in_object_space out_mongo, out_sampling_filter out_amplifier_filter, out_datacounter out_file_alternative, out_forest
Plugins! All your plugin are belong to us.
gem search -r fluent-plugin 41 plugins fluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1) fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1) fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1) fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1) fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1) fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
gem search -r fluent-plugin 13/41 plugins fluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1) fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1) fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1) fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1) fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1) fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
file-alternative (1/13) out_file time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... } out_file_alternative time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] {"key1":"value1", "key2":"value2",... } tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] tag [TAB] value1 [TAB] value2 ... time,tag,value1,value2 ...
hoop (2/13) Hoop Fluentd Server HDFS output format compatible with out_file_alternative
mysql (3/13) alpha! Fluentd MySQL JSON into single column or values into columns
ikachan (4/13) alpha! Fluentd ikachan IRC Server Server
growthforecast (5/13) growthforecast by @kazeburo
flowcounter (6/13) messages count_keys * flowcounter unit minute {"count":300, "bytes":3660, "count_rate":5, "bytes_rate":61}
datacounter (7/13) messages unit minute datacounter aggregate all count_key referer pattern1 google google.com {"google_count":120, "google_rate":2, "google_percentage":20, "unmatched_count":480, ...}
alpha! numeric-monitor (8/13) messages unit minute aggregate all numeric_monitor monitor_key duration percentiles 90,95 {"min":100131,"max":483120, "avg":181343, "percentile_90":283201, "percentile_95":341210}
notifier (9/13) alpha! 10 Notify msg 12 suppressed 10 notifier in specified interval... warn 10 crit 30 15 13 Notify msg
sampling-filter (10/13) msg msg msg msg sampling msg msg filter msg interval 3 msg msg msg
amplifier-filter (11/13) 10 30 12 36 10 amplifier 30 filter 15 ratio 3 45 13 39
beautiful real-time monitoring world Streaming data Sampling Counting Amplifying Generatig Notification Output (Graph, IRC, ...)
parser (12/13) alpha! {"log":"2012/05/18:16:21 DRINK1 1"} format /^(?<time>S+) (?<item>S+) (?<num>d+)$/ numeric_monitor time_format %Y/%m/%d:%H:%M:%S key_name log reserve_data yes log_based_time message {"log":"2012/05/18:16:21 DRINK1 1", "item":"DRINK1", "num":"1"}
<match input.A> forest (13/13) type foo .... <match input.*> conf_a xxxx.A.yyyy type forest option_x 30 subtype foo .... remove_prefix input </match> <template> <match input.B> .... type foo conf_a xxxx.__TAG__.yyyy .... .... conf_a xxxx.B.yyyy </template> option_x 30 <case {C,D}> .... option_x 100 </match> </case> <match input.C> <case *> type foo option_x 30 .... </case> conf_a xxxx.C.yyyy </match> option_x 100 .... </match>
NO PLUGINS, NO FLUENTD Enjoy Fluentd!
One more thing
fluent-plugin-webhdfs Coming Soon!
Thanks!

More Related Content

Plugins by tagomoris #fluentdcasual

  • 1. Plugins by tagomoris Fluentd Casual Talks LT 2012/05/18 @tagomoris (NHN Japan Corp.)
  • 2. @tagomoris NHN Japan Corp. Web Service Business Division Development Department 2 Development Platforms
  • 3. Appendix diff of 2012/02 and now • Our Fluentd cluster status
  • 4. Fluentd Cluster status updates (1) 89 Fluentd processes 12 nodes 89 Fluentd processes 12 nodes
  • 5. Fluentd Cluster status updates (2) ruby 1.9.2 + glibc malloc scribeline ruby 1.9.3 + jemalloc fluent-agent-lite
  • 6. Fluentd Cluster status updates (3) from 127 servers 146 log streams from 205 servers 246 log streams
  • 7. Fluentd Cluster status updates (4) 70,000 msgs/sec 120Mbps (at peak time) 92,000 msgs/sec 184Mbps
  • 8. Fluentd Cluster status updates (5) 650GB/day (non-compressed bytes on HDFS) 995GB/day
  • 9. Fluentd Cluster status updates (6) Before: 10 plugins in_forward, out_forward, in_scribe, out_scribe out_copy, out_roundrobin, out_exec_filter out_hoop out_flowcounter, out_growthforecast After: +9 plugins out_route, in_gc_stat, in_object_space out_mongo, out_sampling_filter out_amplifier_filter, out_datacounter out_file_alternative, out_forest
  • 10. Plugins! All your plugin are belong to us.
  • 11. gem search -r fluent-plugin 41 plugins fluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1) fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1) fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1) fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1) fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1) fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
  • 12. gem search -r fluent-plugin 13/41 plugins fluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1) fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1) fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1) fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1) fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1) fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
  • 13. file-alternative (1/13) out_file time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... } out_file_alternative time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] {"key1":"value1", "key2":"value2",... } tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] tag [TAB] value1 [TAB] value2 ... time,tag,value1,value2 ...
  • 14. hoop (2/13) Hoop Fluentd Server HDFS output format compatible with out_file_alternative
  • 15. mysql (3/13) alpha! Fluentd MySQL JSON into single column or values into columns
  • 16. ikachan (4/13) alpha! Fluentd ikachan IRC Server Server
  • 17. growthforecast (5/13) growthforecast by @kazeburo
  • 18. flowcounter (6/13) messages count_keys * flowcounter unit minute {"count":300, "bytes":3660, "count_rate":5, "bytes_rate":61}
  • 19. datacounter (7/13) messages unit minute datacounter aggregate all count_key referer pattern1 google google.com {"google_count":120, "google_rate":2, "google_percentage":20, "unmatched_count":480, ...}
  • 20. alpha! numeric-monitor (8/13) messages unit minute aggregate all numeric_monitor monitor_key duration percentiles 90,95 {"min":100131,"max":483120, "avg":181343, "percentile_90":283201, "percentile_95":341210}
  • 21. notifier (9/13) alpha! 10 Notify msg 12 suppressed 10 notifier in specified interval... warn 10 crit 30 15 13 Notify msg
  • 22. sampling-filter (10/13) msg msg msg msg sampling msg msg filter msg interval 3 msg msg msg
  • 23. amplifier-filter (11/13) 10 30 12 36 10 amplifier 30 filter 15 ratio 3 45 13 39
  • 24. beautiful real-time monitoring world Streaming data Sampling Counting Amplifying Generatig Notification Output (Graph, IRC, ...)
  • 25. parser (12/13) alpha! {"log":"2012/05/18:16:21 DRINK1 1"} format /^(?<time>S+) (?<item>S+) (?<num>d+)$/ numeric_monitor time_format %Y/%m/%d:%H:%M:%S key_name log reserve_data yes log_based_time message {"log":"2012/05/18:16:21 DRINK1 1", "item":"DRINK1", "num":"1"}
  • 26. <match input.A> forest (13/13) type foo .... <match input.*> conf_a xxxx.A.yyyy type forest option_x 30 subtype foo .... remove_prefix input </match> <template> <match input.B> .... type foo conf_a xxxx.__TAG__.yyyy .... .... conf_a xxxx.B.yyyy </template> option_x 30 <case {C,D}> .... option_x 100 </match> </case> <match input.C> <case *> type foo option_x 30 .... </case> conf_a xxxx.C.yyyy </match> option_x 100 .... </match>
  • 27. NO PLUGINS, NO FLUENTD Enjoy Fluentd!
  • 29. fluent-plugin-webhdfs Coming Soon!