Finding which specific Python process was killed by Linux OOM killer

Question

I'm trying to figure out which specific Python process or executable was killed by the Linux OOM killer.

In /var/log/messages I get this:

Aug 18 03:19:11 169 kernel: [ 7747]     0  7748  3226957  2875051    5692        0             0 python

(notice it just say name = "python")

and this:

Aug 18 03:19:11 169 kernel: Killed process 7748 (python) total-vm:12907828kB, anon-rss:11500204kB, file-rss:0kB

(again notice it just says the process was "python")

Ideally a log file for the process would have the PID somewhere. But suppose the logs wrapped (or suppose the process doesn't log the PID anywhere).

Does Linux provide a way to figure out the full command that was executed for the process? It would be nice to configure Linux's OOM killer to display the full name in the process table, such as:

/usr/bin/python /usr/lib/python2.7/site-packages/foo.pyc

Or maybe at the time of the OOM error Linux stores off some of the process details somewhere? i.e. copy the processes from /proc to X? (wishful thinking)

NOTE:

This question is very similar to this question: https://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer

But it fell short of what I'm trying to figure out.

Answer 1

Linux doesn't support querying such information by the time OOM Killer is about to kill a process or already killed the process.

You can monitor the system constantly and log every process with timestamp and PID and the full command line if you want. See https://stackoverflow.com/a/8255487/334451 for an example implementation.

Because you cannot know beforehand which process will be killed in the future, you must store information from all processes and keep that information until the process has stopped for reason other than OOM killer killing it.