Too many open file descriptors RHEL 7.4

Question

I have migrated to RHEL 7.4 from 6.9 and same application which used to run fine on 6.9 is running into "Too many open files" issue.

ulimits are set to same values across both environments. upon verifying with "lsof" command, I noticed that the same set of file descriptors are being opened by the process under different "tasks". For e.g. a process which required only 4 file descriptors in RHEL 6.9 uses around 350+ in RHEL 7.4

Due to this any process which opens multiple files runs into the "Too many open files" error within few minutes.

Can anyone guide me down the right path for debugging/fixing this problem?

Answer 1

First, you may want to check the current maximum number of file descriptors and ensure it's a reasonable value. This question answers in detail how to do so.

In short, you can check the max global FDs available in /proc/sys/fs/file-max, or the max FDs available to the shell with bash's ulimit command.

Once you've verified this, you will probably be interested in what processes are eating up all your file descriptors. This answer details a shell script you can use to enumerate how many FDs are consumed by the top 10 running processes. The script from the question is copied here:

find -maxdepth 1 -type d -name '[0-9]*' \
     -exec bash -c "ls {}/fd/ | wc -l | tr '\n' ' '" \; \
     -printf "fds (PID = %P), command: " \
     -exec bash -c "tr '\0' ' ' < {}/cmdline" \; \
     -exec echo \; | sort -rn | head

This should help you locate what service on your machine is eating up your FDs. I hope this helps, please let me know if you have any questions.