Elk devops
•
4 likes•2,904 views
La gestione dei log è da sempre un argomento complesso e nel tempo si sono cercate varie soluzioni più o meno complesse, spesso difficili da integrare nel proprio stack applicativo. Daremo un’ overview generale dei principali sistemi di aggregazione evoluta dei log in realtime (Fluentd, Greylog, eccetera) e illustreremo del motivo ci ha spinto a scegliere ELK per risolvere un’esigenza del nostro cliente; ovvero di consultare i log in modo piu comprensibile da persone non tecniche. Lo stack ELK (Elasticsearch Logstash Kibana) permette agli sviluppatori di consultare i log in fase di debug / produzione senza avvalersi dello staff sistemistico. Dimostreremo come abbiamo eseguito il deployment dello stack ELK e lo abbiamo implementato per interpretare e strutturare i log applicativi di Magento.
Report
Share
Elk devops
- 1. ELK, a real case study Alessandro Mazzoli Sysadmin@Ideato am@ideato.it Paolo Tonin Sysadmin@Ideato pt@ideato.it
- 2. What is ELK stack? Elasticsearch Logstash Kibana http://en.wikipedia.org/wiki/Elk
- 3. Logging problems (especially if you are a small company)
- 7. Log As a Service?!?! Pretty expensive
- 8. Log analysis is hard for non technical people
- 9. cat access.log | cut -d' ' -f1 | sort | uniq
- 11. People don’t pay attention to boring things. - Brain Rules by John Medina
- 14. Why ELK, a case study Web Tier Apache Nginx Proxy MySQL DB Proxy FE Database BE NFS shared filesystem
- 15. We need to scale Web Tier
- 16. Why ELK, a case study Nginx proxy LB Apache web1 MySQL DB Memcached, Logstash Web TierProxy FE Database BE Apache web2 NFS shared filesystem
- 17. Nginx proxy LB Apache web1 MySQL DB Memcached, Logstash Web TierProxy FE Database BE Apache web2 NFS shared filesystem Why ELK, a case study local log local log
- 18. ELK architecture Server 1 Log File Logstash Shipper Redis Application Logstash Indexer Elasticsearch
- 20. Redis Logstash Indexer Redis Redis Redis ElasticsearchElasticsearch Server 2 Logstash Shipper Server 1 Logstash Shipper
- 21. Logstash configuration input { ! } filter { ! } output { ! } Where log come from?
- 22. Logstash configuration input { ! } filter { ! } output { ! } How we threat them ? Where log come from?
- 23. Logstash configuration input { ! } filter { ! } output { ! } Where will be stored? How we threat them ? Where log come from?
- 24. Example configuration input { file { path => "/var/log/messages" type => "syslog" } file { path => "/var/log/apache/access.log" type => “apache-access" } }
- 25. Example configuration filter { if [type] =~ "access" { mutate { replace => { "type" => “apache-access” } } grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] } } }
- 26. Example configuration output { elasticsearch { host => localhost } }
- 27. “How can I collect and consult my application Magento log?”
- 28. Custom filter
- 29. Grok It’s is currently the best way in logstash to parse crappy unstructured log data into something structured and queryable - http://logstash.net/docs/1.4.2/filters/grok
- 30. https://github.com/ideatosrl/logstash-magento-filter input { file { path => “/var/www/magento/var/log/*.log" type => "magento" codec => plain { charset => "ISO-8859-1" } } } Logstash Shipper
- 31. https://github.com/ideatosrl/logstash-magento-filter filter { if [type] == "magento" { grok { match => { "message" => "% {TIMESTAMP_ISO8601:timestamp} % {DATA:syslog_program}"} add_field => [ "received_at", "% {@timestamp}" ] } } } Logstash Shipper
- 32. https://github.com/ideatosrl/logstash-magento-filter output { redis { host => “REDIS_IP" data_type => "list" key => "logstash" } } Logstash Shipper
- 33. https://github.com/ideatosrl/logstash-magento-filter input { redis { host => "REDIS_IP" type => "redis-input" data_type => “list" key => "logstash" } } output { elasticsearch { host => “ES_PUBLIC_IP” protocol => "http" manage_template => false index => "logstash-%{+YYYY.MM.dd}" } } Logstash Indexer
- 34. Before… 2015-03-30T15:25:34.867Z SoapFault exception: [soap:Client] Server was unable to read request. ---> There is an error in XML document (2, 439). ---> Input string was not in a correct format. in /var/www/magento/vendor/ connect20/MailUp/app/code/local/MailUp/ MailUpSync/Model/Observer.php:158
- 35. { "_index": "logstash-2015.03.30", "_type": "magento", "_id": "AUxrSbc5UWx9I25Cgios", "_score": null, "_source": { "message": "SoapFault exception: [soap:Client] Server was unable to read request. ---> There is an error in XML document (2, 435). ---> Input string was not in a correct format. in /var/www/magento/vendor/connect20/MailUp/app/ code/local/MailUp/MailUpSync/Model/Observer.php:158", "@version": "1", "@timestamp": "2015-03-30T15:25:34.867Z", "type": "magento", "host": "web2", "path": "/var/www/magento/var/log/exception.log", }, }
- 36. And display it!
- 37. “Should I use ELK?”
- 38. PRO • Easy to install ! • Opensource ! • Not only web logs! ! • Many output filters; S3, Google Big Query, MongoDb etc…
- 39. ConclusionsCONS • Many moving parts, each part has their problems and issues ! • Quickly evolution, prepare yourself to upgrade frequently ! • Out-of-the-box configurations are not suitable for medium/large deployment
- 41. Questions?!?