Fluentd intro at Tajo seoul meetup
•
21 likes•1,115 views
This document summarizes Fluentd, an open source data collector. It discusses Fluentd's pluggable architecture and 300+ community contributed plugins. It also provides instructions on installing and running Fluentd, and includes an example configuration for collecting Apache logs and writing them to HDFS using the webhdfs output plugin. Key details are Fluentd's built-in reliability, low resource usage, and ability to process high data volumes.
Report
Share
Fluentd intro at Tajo seoul meetup
- 1. open source data collector for unified logging layer
- 4. 2000+ data driven companies!
- 5. Pluggable Architecture 300+ community contributed plugins
- 7. Installation & Run $ gem install fluentd $ fluentd --setup ./fluent $ fluentd -c ./fluent/fluent.conf $ curl -L http://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh $ /etc/init.d/td-agent start $ /etc/init.d/td-agent stop $ /etc/init.d/td-agent restart $ /etc/init.d/td-agent status td-agent
- 8. Configuration <source> type tail format apache2 tag hdfs.apache.access path /var/log/apache2/access.log </source> <match hdfs.*.*> type webhdfs host namenode.your.cluster.local port 50070 path /log/%Y%m%d_%H/access.log.${hostname} flush_interval 10s field_separator TAB # or 'SPACE', ‘COMMA' output_data_type attr:path,status,referrer,agent </match>
- 9. Fluentd + HDFS 20,000 msgs/sec 1.5TB/ day http://www.slideshare.net/tagomoris/fluentd-and-webhdfs (NHN Japan) $ fluent-gem install fluent-plugin-webhdfs
- 10. • C + Ruby • 30~40M RAM • 20,000 events / Sec / Core • built-in monitoring • 300+ Plugins • Ruby / Java / Go / D / Perl / Python / PHP / node.js • Internet of Things • Active community