Docker logs is missing log entires which are showing in docker attach

Question

I am working on a software running inside a docker container where the produced logs are mandatory in order to evaluate it later on.

My problem is that the command docker logs is missing some log entries that are definetly logged by my software. I can verify the fact that its logging the messege by attaching to the container with docker attach.

Basically docker attach shows all entires in real time but docker logs is only logging some of them without any recognisable pattern. All logs are written to stdout by the software.

Edit: I realised this behavior for other docker containers as well.

Answer 1

I found the issue. Our docker daemon is configured to use journald as its default logging driver. Because we have different containers logging quiet alot we're hitting the default journald rate limit and therefore some of the logs are getting lost in the void.

There are two solutions to the problem:

1. Solution

Increase the rate limit of journald. For this use the /etc/systemd/journald.conf configuration file and add/modify the following lines:

RateLimitInterval=30s
RateLimitBurst=1000

Afterwards just restart journald: systemctl restart systemd-journald

For more information about this start looking: here or on manpages

2. Solution

You can specify a different logging driver for a given container with the parameter --log-driver json-file

For more informations about docker logging drivers

Additional Infos:

A useful link to identify common docker logging gotchas.