SlideShare a Scribd company logo

Logging & Metrics with Docker

•Download as PPTX, PDF•
•
Stefan Zier
Stefan Zier

In this deck, I quickly summarize how people have dealt with logging in Docker historically and then describe a comprehensive approach to logging and monitoring in Docker, based on research and customer interviews. While Docker adds a welcome layer of abstraction to the deployment of applications, it also challenges assumptions on how those applications should be managed. I discuss a comprehensive approach for collecting logs and metrics into a centralized platform and dissect the latest additions to Docker itself (log drivers, stats).

Related slideshows

Fluentd and PHP
Fluentd and PHPFluentd and PHP
Fluentd and PHP
 
Docker practical solutions
Docker practical solutionsDocker practical solutions
Docker practical solutions
 
Getting instantly up and running with Docker and Symfony
Getting instantly up and running with Docker and SymfonyGetting instantly up and running with Docker and Symfony
Getting instantly up and running with Docker and Symfony
 
1 of 35
Logging & Metrics With Docker A Comprehensive Monitoring Solution Stefan Zier June 13th, 2015
whoami Infrastructure, Backend Dev/Architect Chief Architect, Sumo Logic, since 2010 Server & Infrastructure, ArcSight (HP), 2001-2010
Mandatory Slide Showing Shipping Containers
Docker – What’s making debugging hard? One more layer of abstraction Container per app = File system per process File systems short lived, transient Resource schedulers = no container affinity to host
What Our Customers Are Telling Us We have one process per container We like to log to stdout We have multiple processes per container We run the Sumo Logic collector on the Docker host We are looking into using Beanstalk with Docker We are using Amazon ECS Everyone here loves Docker We are logging straight from the application We are using /dev/log for Syslog We want immutable infrastructure
Goal Get logs from our containerized applications to a centralized logging platform.
How do apps emit logs Append to a file Use syslog() Use log4j, log4net, slf4, etc. printf() to stdout
Getting logs out of the container - Files Use VOLUME to mount a host directory Collect files from the host Collect files from another container sharing the VOLUME Need to manage disk space, i.e. rotate logs App (where supported) Host Yet another container with logrotate
Logging & Metrics with Docker
docker run -v /tmp/clogs:/tmp/clogs -d --name="sumo-logic-collector" sumologic/collector:latest-file [Access ID] [Access key]
Getting logs out - Syslog VOLUME /dev/log from host and use host syslogd Run a syslogd inside the container Emit TCP/UDP Write to a file using VOLUME Emit syslog TCP/UDP directly from the app
Logging & Metrics with Docker
docker run -d -p 514:514 -p 514:514/udp --name="sumo-logic-collector” sumologic/collector:latest-syslog [Access ID] [Access key]
Getting logs out – Logging frameworks Sumo Logic blog on official collector images http://www.sumologic.com/blog/company/an-official-docker-image-for- the-sumo-logic-collector https://github.com/SumoLogic/sumologic-collector-docker Rainer Gerhards on Rsyslog’s file input module http://www.slideshare.net/rainergerhards1/using-wildcards-with- rsyslogs-file-monitor-imfile OWASP Log Injection https://www.owasp.org/index.php/Log_injection
Getting logs out – Logging frameworks Directly to network destinations HTTP/HTTPS Also support files, stdout, etc.
Logging & Metrics with Docker
Getting logs out – Logging frameworks Various application stacks http://help.papertrailapp.com/ Log4J https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SyslogAppender.html Apache Web Server http://httpd.apache.org/docs/trunk/mod/mod_syslog.html https://raymii.org/s/snippets/Apache_access_and_error_log_to_syslog.html Nginx http://nginx.org/en/docs/syslog.html Postgres http://www.postgresql.org/docs/9.1/static/runtime-config-logging.html Sumo Logic blog on official syslog collector image http://www.sumologic.com/blog/company/an-official-docker-image-for-the-sumo-logic-collector https://github.com/SumoLogic/sumologic-collector-docker
Getting logs out – stdout Simply printf() Logging framework to console Symlink to /dev/stdout or /dev/stderr Configure paths to /dev/stdout or /dev/stderr RUN ln -sf /dev/stdout /var/log/nginx/access.log RUN ln -sf /dev/stderr /var/log/nginx/error.log
Docker Logging Drivers
What Docker provides Captures stdout/stderr Feeds it to logging drivers docker logs command Returns the entire log every time Works with json-file driver only Can tail logs docker logs –tf –-tail 0 [ID]
Docker Logging Drivers Configured on docker run stdout and stderr dispatched to drivers json-file (default pre 1.6) syslog journald No stats, no events json-file driver Output unbounded, can fill up the host disk Requires logrotate on the Docker host https://github.com/docker/docker/issues/7333
Stats
Docker Stats Per-container cgroups metrics (like docker stats) Memory CPU Block I/O CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O collector 2.23% 232.6 MiB/2 GiB 11.36% 191.9 KiB/636.3 KiB
Requirements How would we want it to work?
What information do we want to collect? Timestamp Log message Docker host info Container ID Image ID Process ID
How should it work? Use docker logging infrastructure Minimal moving parts Containerized - don’t touch the host Complete – pick up all available data Automatically discover new containers
Docker API The solution maybe?
Docker API Docker daemon has a REST API TCP or unix socket Streaming APIs Docker Events (container lifecycle updates) Container Stats (CPU, memory used, …) App Logs (container stdout/stderr)
Collecting via Docker API Discover new containers via events Start streaming their logs and stats When they go away, stop Do all of this via the API Send all of it to centralized log management
Collecting via Docker API, continued Single component to do it Zero footprint on the host Follows Docker standard way of logging
One more thing…
Introducing: Sumo Logic Docker Source
Sumo Logic Docker Source Active development Early access expected later this year
Demo Time
fin. Questions? @stefanzier

More Related Content

Logging & Metrics with Docker

  • 1. Logging & Metrics With Docker A Comprehensive Monitoring Solution Stefan Zier June 13th, 2015
  • 2. whoami Infrastructure, Backend Dev/Architect Chief Architect, Sumo Logic, since 2010 Server & Infrastructure, ArcSight (HP), 2001-2010
  • 3. Mandatory Slide Showing Shipping Containers
  • 4. Docker – What’s making debugging hard? One more layer of abstraction Container per app = File system per process File systems short lived, transient Resource schedulers = no container affinity to host
  • 5. What Our Customers Are Telling Us We have one process per container We like to log to stdout We have multiple processes per container We run the Sumo Logic collector on the Docker host We are looking into using Beanstalk with Docker We are using Amazon ECS Everyone here loves Docker We are logging straight from the application We are using /dev/log for Syslog We want immutable infrastructure
  • 6. Goal Get logs from our containerized applications to a centralized logging platform.
  • 7. How do apps emit logs Append to a file Use syslog() Use log4j, log4net, slf4, etc. printf() to stdout
  • 8. Getting logs out of the container - Files Use VOLUME to mount a host directory Collect files from the host Collect files from another container sharing the VOLUME Need to manage disk space, i.e. rotate logs App (where supported) Host Yet another container with logrotate
  • 10. docker run -v /tmp/clogs:/tmp/clogs -d --name="sumo-logic-collector" sumologic/collector:latest-file [Access ID] [Access key]
  • 11. Getting logs out - Syslog VOLUME /dev/log from host and use host syslogd Run a syslogd inside the container Emit TCP/UDP Write to a file using VOLUME Emit syslog TCP/UDP directly from the app
  • 13. docker run -d -p 514:514 -p 514:514/udp --name="sumo-logic-collector” sumologic/collector:latest-syslog [Access ID] [Access key]
  • 14. Getting logs out – Logging frameworks Sumo Logic blog on official collector images http://www.sumologic.com/blog/company/an-official-docker-image-for- the-sumo-logic-collector https://github.com/SumoLogic/sumologic-collector-docker Rainer Gerhards on Rsyslog’s file input module http://www.slideshare.net/rainergerhards1/using-wildcards-with- rsyslogs-file-monitor-imfile OWASP Log Injection https://www.owasp.org/index.php/Log_injection
  • 15. Getting logs out – Logging frameworks Directly to network destinations HTTP/HTTPS Also support files, stdout, etc.
  • 17. Getting logs out – Logging frameworks Various application stacks http://help.papertrailapp.com/ Log4J https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SyslogAppender.html Apache Web Server http://httpd.apache.org/docs/trunk/mod/mod_syslog.html https://raymii.org/s/snippets/Apache_access_and_error_log_to_syslog.html Nginx http://nginx.org/en/docs/syslog.html Postgres http://www.postgresql.org/docs/9.1/static/runtime-config-logging.html Sumo Logic blog on official syslog collector image http://www.sumologic.com/blog/company/an-official-docker-image-for-the-sumo-logic-collector https://github.com/SumoLogic/sumologic-collector-docker
  • 18. Getting logs out – stdout Simply printf() Logging framework to console Symlink to /dev/stdout or /dev/stderr Configure paths to /dev/stdout or /dev/stderr RUN ln -sf /dev/stdout /var/log/nginx/access.log RUN ln -sf /dev/stderr /var/log/nginx/error.log
  • 20. What Docker provides Captures stdout/stderr Feeds it to logging drivers docker logs command Returns the entire log every time Works with json-file driver only Can tail logs docker logs –tf –-tail 0 [ID]
  • 21. Docker Logging Drivers Configured on docker run stdout and stderr dispatched to drivers json-file (default pre 1.6) syslog journald No stats, no events json-file driver Output unbounded, can fill up the host disk Requires logrotate on the Docker host https://github.com/docker/docker/issues/7333
  • 22. Stats
  • 23. Docker Stats Per-container cgroups metrics (like docker stats) Memory CPU Block I/O CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O collector 2.23% 232.6 MiB/2 GiB 11.36% 191.9 KiB/636.3 KiB
  • 24. Requirements How would we want it to work?
  • 25. What information do we want to collect? Timestamp Log message Docker host info Container ID Image ID Process ID
  • 26. How should it work? Use docker logging infrastructure Minimal moving parts Containerized - don’t touch the host Complete – pick up all available data Automatically discover new containers
  • 28. Docker API Docker daemon has a REST API TCP or unix socket Streaming APIs Docker Events (container lifecycle updates) Container Stats (CPU, memory used, …) App Logs (container stdout/stderr)
  • 29. Collecting via Docker API Discover new containers via events Start streaming their logs and stats When they go away, stop Do all of this via the API Send all of it to centralized log management
  • 30. Collecting via Docker API, continued Single component to do it Zero footprint on the host Follows Docker standard way of logging
  • 33. Sumo Logic Docker Source Active development Early access expected later this year

Editor's Notes

  1. Introduced in Docker 1.6
  2. Introduced in Docker 1.6
  3. Introduced in Docker 1.6
  4. Introduced in Docker 1.6
  5. Introduced in Docker 1.6