Odd File system disk consumption

Question

I know similar has been asked as I searched and tried to figure as it relates to my system. running Ubuntu 18 and other than some install for PostgreSQL, some recent security updates, etc... my file system drive has ballooned and is seeming to continue to shrink. Some additional searching regarding trying to figure what is taking room at /dev/sde1/ as shown in image below (highlighted) indicates maybe something to do with journaling or something which may have been triggered with getting a mail server up. I am not sure the best and safest way to approach this. This mount is my boot drive that is being affected.

Images to what I am seeing:

from cmd:

from webmin:

Thoughts on a sensible approach to this?

Answer 1

Well, if something is occupying a lot of space, the sensible approach is:

1. Find out what is occupying a lot of space.

   A convenient tool is ncdu, specifically ncdu -x /. It will show you what directories in the root filesystem are using the most, and lets you navigate inside them (basically, similar to WinDirStat or SpaceSniffer).

   It's also doable using manual repeated du -xhs <dir>, but a bit tedious. (In both cases, the "x" option tells it to avoid digging into virtual filesystems like /proc.)

2. Remove the thing that's occupying a lot of space.

   Don't immediately delete the files though – figure out what lets them grow in the first place, and whether they need to be deleted using a special command. For example, database log/journal files must be flushed through the DB itself. System logs in /var/log are safe to delete but they have a configurable retention period anyway – check why that isn't applied.

3. If the used space doesn't add up – e.g. ncdu shows only 50 GB worth of files – it's possible that a large file that has already been deleted (unlinked) is still held open by some program. Restarting the program or rebooting the system would solve this.

   sudo lsof -n | grep "(deleted)" would show all deleted but still open files, alongside the process IDs. (Ignore those under /tmp or /dev/shm – that's normal and they're on a separate filesystem anyway.)

   If you were using LVM/Btrfs/ZFS, you should also check whether you have any old snapshots or lost subvolumes – those aren't visible to the filesystem, but still occupy space.