How can determine how my partition is full even when it appears it is not?

Question

I have run "df -kh" on my FreeBSD box and I see that /var is near 100% full. It was over but I have trimmed what I can but there is less then 500MB total on the partition according to running "du -ksh *" while in /var. The /var partition itself is 6.8G and it now says it only has 18M free. It just does not add up.

I run the following command I do not find any culprits.

 find . -type f -size +2000 -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

I even drill down folder by folder using "du -ksh *" to identify the folders with lots of disk usage but I cannot find how it all adds up. I would like to compress, delete or move files in order to free up more space since the other partitions have space, but between df and du I cannot seem to find what is using all the disk space.

What else should I do to find the files and folders taking up all the space?

Answer 1

I'd run the following, so that you have a nice snapshot of all files to work with:

du -kx /var | sort -n | tee /var/tmp/du_results.out

... and then look at the last few hundred lines.

This will catch the size of all directories (to catch large numbers of small files), as well as catching ordinary large files. Note that I'm not using -h (human-readable), so that the results are easily sorted numerically. Note also that I'm not using /var/*, which would pick up anything mounted in /var as well.

The possible culprits include, in rough order of likelihood:

• Stale unlinked files. Try what @not simon suggested (lsof /var). /var/tmp is sometimes an obvious place to find these. If you don't have lsof installed, you can use fstat -f /var | sort -k 8 -n, which doesn't show file names, but will at least show sizes (smallest to largest), inodes, and which process has the file open.

• Filesystem corruption. Runfsck from single-user mode to check /var.

• Old files hidden under mount points. If there are any mounts under /var, unmount them and then run your search/diagnostic again. The safest way to do this is probably to boot in single-user mode, and just mount /var without mounting anything under it.

Differences in file size reporting can be due to the presence of sparse or compressed files. You can use the -A option to du to work with apparent file size instead of disk usage, which might explain some discrepancies. Put it in and then leave it out, and compare the results. In my roughly 3.6Gb /var partition, results with -A were about 100Mb smaller.

Answer 2

Have you considered that you may have an unlinked file that is still open? Try looking over the output of lsof /var to see if there are any unusually large files listed.

Answer 3

I had a similar problem where a disk went away. The mount point still existed, but for some reason files were being written to the directory on the partition containing the mount point. This may not apply to your operating system, but booting a rescue system and perusing the partitions from there may reveal more.