How to avoid high latency near OOM situation?

Question

Minimal test case when Linux system does not have swap (or run sudo swapoff -a before testing). Run following bash one-liner as normal user:

while true; do date; nice -20 stress --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)k --vm-keep -m 1 --timeout 10s; sleep 5s; done

and run following bash one-liner with high priority root shell (e.g. sudo nice -n -19 bash):

while true; do NS=$(date '+%N' | sed 's/^0*//'); let "S=998000000 - $NS"; S=$(( S > 0 ? S : 0)); LC_ALL=C sleep "0.$S"; date --iso=ns; done

The high priority process is supposed to run date every second as accurately as possible. However, even if this process is running with priority -19, the background process running on priority 20 is able to cause major delays. It seems that there's no limit for the latency induced by the low priority background process because higher delays can be activated by increasing the stress --timeout value.

Is there a way to limit maximum latency and automatically kill the stress if needed to accomplish that? Increasing /proc/sys/vm/user_reserve_kbytes or /proc/sys/vm/admin_reserve_kbytes or /proc/sys/vm/min_free_kbytes does not seem to help.

Answer 1

Please consider trying* the kernel patch from this question, as it seems to do the job(avoid high latency near oom) for me so far(even using your code from the question to test it) and I'm also avoiding a ton of disk thrashing(for example when I compile firefox which usually caused the OS to freeze due to running out of memory).
The patch avoids evicting Active(file) pages, thus keeping (at least) the executable code pages in RAM so that context switches don't cause kswapd0(?) to re-read them(which would cause lots of disk reading and a frozen OS).

* or even suggesting a better way?

Answer 2

There are a few tools designed to avoid this particular issue, listed with increasing complexity/configurability:

• earlyoom, probably good enough for desktop/laptop computers
• nohang, a more configurable solution
• Facebook's solution oomd for their own servers.