This document discusses Fluentd, an open source log collector. It provides a pluggable architecture that allows data to be collected, filtered, and forwarded to various outputs. Fluentd uses JSON format for log messages and MessagePack internally. It is reliable, scalable, and extensible through plugins. Common use cases include log aggregation, monitoring, and analytics across multiple servers and applications.
10. > Open sourced log collector written in Ruby
> Reliable, scalable and easy to extend
> Using rubygems ecosystem for plugins
!
!
In short
It’s like syslogd, but
uses JSON for log messages
12. > default second unit
> from data source or
adding parsed time
Event structure(log message)
✓ Time
> for message routing
✓ Tag
> JSON format
> MessagePack
internally
> non-unstructured
✓ Record
15. Configuration and operation
> No central / master node
> HTTP include helps configuration sharing
> Operation depends on your environment
> Use your deamon management
> Use Chef in Treasure Data
> Apache like syntax and Ruby DSL
16. # receive events via HTTP
<source>
type http
port 8888
</source>
!
# read logs from a file
<source>
type tail
path /var/log/httpd.log
format apache
tag apache.access
</source>
!
# save access logs to MongoDB
<match apache.access>
type mongo
database apache
collection log
</match>
# save alerts to a file
<match alert.**>
type file
path /var/log/fluent/alerts
</match>
!
# forward other logs to servers
<match **>
type forward
<server>
host 192.168.0.11
weight 20
</server>
<server>
host 192.168.0.12
weight 60
</server>
</match>
!
include http://example.com/conf
17. Reliability (core + plugin)
> Buffering
> Use file buffer for persistent data
> buffer chunk has ID for idempotent
> Retrying
> Error handling
> transaction, failover, etc on forward plugin
> secondary for backup
29. td-agent
> Open sourced distribution package of fluentd
> ETL part of Treasure Data
> deb, rpm, dmg (since td-agent 2.0)
> Including useful components
> ruby, jemalloc, fluentd
> 3rd party gems: td, mongo, webhdfs, etc…
> http://packages.treasure-data.com/
30. v1
> New features without breaking compatibility
> Filter, Label and better error handling
> Serverengine based: multi-process, signal, etc.
> New configuration and DSL format
> JRuby and Windows support
> github issue: Plan for v1 release #251
37. > Fluentd is a widely-used log collector
> There are many use cases
> Many contributors and plugins
> Keep it simple
> Easy to integrate your environment
Conclusion