Measuring cumulative peak memory usage of multiple parallel processes

Question

Imagine I run a bunch of processes in parallel using xargs and want to measure their cumulative peak memory usage. How can I do that in Linux? GNU time only measures peak RSS of a single process I run. In other words, if each process consumes M GB of memory and I run N of them, I want to see N*M GBs as the result, not M which is given by GNU time.

Answer 1

Are you interested in sum of peak memory usage of each process, or peak of the sum of memory usage of the processes?

For example, to get peak resident memory usage of top 5 biggest spenders in your system, you can do something like this:

$ printf "PID\tPeak RSS usage (kB)\n"; grep VmHWM: /proc/*/status | sort -rnk2 | head -n 5 | perl -npe 's#^/proc/(\d+)/\S+\s+(\d+) kB$#$1\t$2#'
PID Peak RSS usage (kB)
19799   8515140
28243   930620
1357    911612
19833   766020
17926   734952

That will list 5 processes that have had biggest RAM usage in their process history. However, it's pretty common that those processes did not peak at the very same moment so the peak of sum of those processes is probably much lower than sum of peak of all processes.

The only sure method do compute peak of sum of process memory usage over multiple processes is to run that collection of processes in a separate memory cgroup. In theory you just run the program with systemd-run --pipe --user --pty ... and then monitor the peak memory usage of that memory cgroup. However, in my experience systemd-run doesn't correctly implement transient user mode groups so you would have to create separate system level cgroup to contain the group you want measure and then start your process tree in that group. If you run recent enough distribution with more recent systemd-run than I've tried, it might work without any hacks. See https://stackoverflow.com/a/61916151/334451 for details. You can inspect results via memory cgroup mount which is usually mounted somewhere below /sys/fs/cgroup. Possible locations include subdirectories memory or unified. Try mount | grep cgroup for details. Your result may be memory.max_usage_in_bytes but I think that's committed usage, not actual resident RAM usage - you didn't specify which one you're looking for. See memory.stat inside the memory cgroup for additional statistics.

If you use transient memory cgroup you probably want to run e.g. long sleep command as the last process to make sure you have enough time to read the peak memory usage before the cgroup is deleted after your process tree is complete.