Oracle Linux 7 - Which processes are using so much buff/cache memory

Question

I hope someone can guide me in the right direction.

I recently updated some software on a server, and a few days later, this software crashed because no memory could be allocated. Initially, I thought there was a bug in the new release until I checked other Oracle Linux 7 servers (without the updated software) and observed the following:

[root@server yabbath]# free -mh
              total        used        free      shared  buff/cache   available
Mem:            62G        2.2G         42G         24M         18G         59G
Swap:           63G          0B         63G
[root@server2 yabbath]# free -mh

              total        used        free      shared  buff/cache   available
Mem:            62G         11G        4.1G         34M         47G         50G
Swap:           31G          0B         31G

Unfortunately, monitoring doesn't subtract the buff/cache value, so everything seemed fine. After implementing this, I got the following graph:

smem output is the following

smem -r
      PID User     Command                         Swap      USS      PSS      RSS
     1485 mongod   /usr/bin/mongod -f /etc/mon        0  1155092  1155941  1160916
    20146 root     /usr/sbin/nsrexecd                 0    38884    38987    41232
     1365 zabbix   /usr/sbin/zabbix_agent2 -c         0    30956    31471    35648
     1835 zabbix   /usr/sbin/zabbix-agent2-plu        0    21796    21836    23304
     1366 root     /usr/bin/python2 -Es /usr/s        0    13060    14959    22176
      682 root     /usr/lib/systemd/systemd-jo        0     4988    14089    34972
     1046 polkitd  /usr/lib/polkit-1/polkitd -        0     8184     9462    14724
    10225 root     python /bin/smem -r                0     7592     8479    11488
     1363 redis    /usr/local/bin/redis-server        0     7132     7848    10776
     1371 root     /usr/sbin/rsyslogd -n              0     1876     7605    23312
     1369 redis    /usr/local/bin/redis-sentin        0     5992     6705     9624
     1105 root     /usr/sbin/NetworkManager --        0     5164     6689    13908
        1 root     /usr/lib/systemd/systemd --        0     4852     5658    10416
     1040 root     /usr/bin/VGAuthService -s          0     3596     4198     9340
      717 root     /usr/lib/systemd/systemd-ud        0     3240     3998     8468

Can someone tell me how to get information about which process is using how much of this buff/cache memory?

Thanks in advance and regards,

yabberth

Edit:

This is just on of many servers. I also have machines where the buff/cache memory is using 47GB of a total of 64GB.

Answer 1

Buffering/cache memory are per instance and it is used from all processes. And for the moment you should not be concern as you have 11GB for cache with total memory of 64GB.

You can see what is inside using these techniques (from this answer):

You can see which blocks are currently in your cache by using fincore. Here is an example from the project page:

 # fincore --pages=false --summarize --only-cached *  

stats for CLUSTER_LOG_2010_05_21.MYI: file size=93840384 , total pages=22910 , cached pages=1 , cached size=4096, cached perc=0.004365 
stats for CLUSTER_LOG_2010_05_22.MYI: file size=417792 , total pages=102 , cached pages=1 , cached size=4096, cached perc=0.980392 
stats for CLUSTER_LOG_2010_05_23.MYI: file size=826368 , total pages=201 , cached pages=1 , cached size=4096, cached perc=0.497512 
stats for CLUSTER_LOG_2010_05_24.MYI: file size=192512 , total pages=47 , cached pages=1 , cached size=4096, cached perc=2.127660 
stats for CLUSTER_LOG_2010_06_03.MYI: file size=345088 , total pages=84 , cached pages=43 , cached size=176128, cached perc=51.190476 

As for how to clear them, from man 5 proc:

/proc/sys/vm/drop_caches (since Linux 2.6.16)

Writing to this file causes the kernel to drop clean caches, dentries, and inodes from memory, causing that memory to become free.

This can be useful for memory management testing and performing reproducible filesystem benchmarks. Because writing to this file causes the benefits of caching to be lost, it can degrade overall system performance.

Answer 2

If you look at the output of your free -mh, you will see that available+buf/cache are bigger than total:

[root@server yabbath]# free -mh
              total        used        free      shared  buff/cache   available
Mem:            62G        2.2G         42G         24M         18G         59G
Swap:           63G          0B         63G

How is that? Easy: Only RAM that would be unused else is put to service as buffer/cache memory to speed up IO operations. If a process needs more RAM than what is currently free, memory from the buff/cache pool will be released (possibly after writing back its content) and then given to the requesting process.

The important metric for you is available: This is, what your payload can actually work with.