Introducing log analysis to your organization
- 33. Buffer types
Disk || memory || combined hybrid approach
On source || centralized
App
Buffer
App
Buffer
file or local log shipper
easy scaling – fewer moving parts
often with the use of lightweight shipper
App
App
Kafka / Redis / Logstash / etc…
one place for all changes
extra features made easy (like TTL)
ES
ES
- 54. Elasticsearch – Indices
Index – logical place for data
Index – can be compared to database in DB
Index – built out of one or more shards
Shard – can be spread among multiple nodes
- 77. Hot – cold architecture
logs_2017.11.22
ES hot ES cold ES cold
-Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold
curl -XPUT localhost:9200/logs_2017.11.22 -d '{
"settings" : {
"index.routing.allocation.exclude.tag" : "cold",
"index.routing.allocation.include.tag" : "hot"
}
}'