What should happen if I delete files in a BtrFS snapshot?

Question

I have an openSUSE computer that started with BtrFS early (like Leap 42.2). At one moment in the past the /tmp subvolume had become full (one big file) and I was not able to recover space until reboot (rm had triggered a No space left on device). Then everything looked sound for at least one year.

But recently (at Leap 15.1 in the meantime) BtrFS got full again, and I wondered what to do: I had many snapshots like this:

# ls -l /.snapshots/
total 4
drwxr-xr-x 1 root root  32 Dec 18  2015 1
drwxr-xr-x 1 root root  32 May 14 09:45 1820
drwxr-xr-x 1 root root  66 May 14 09:46 1821

...

drwxr-xr-x 1 root root  32 Aug  8 08:08 1926
drwxr-xr-x 1 root root  38 Aug  8 08:09 1927
drwxr-xr-x 1 root root  38 Aug  8 08:12 1928

After having checked all the blocks checksums sucessfully (no problems) I started a "balance" hoping for some free space to appear. But balance never seemed to finish, so I tried to abort it. Having waited at least 15 minutes for balance to abort, I rebooted the computer to try something else. At that time the filesystem was 99% full.

I thought I'd clean up the oldest snapshot (1) using rm -rf /.snapshots/1. Unfortunately after finishing, essential programs from /usr had vanished and my system had become unbootable!

So my question is: Is that expected behavior, or did I do something wrong? If I did something wrong, what is the correct procedure to remove old snapshots?

Answer 1

It seems the issues seen after removing /.snapshots/1 is not so much a feature of BtrFS by itself, but a (mis?)feature of SUSE Linux:

I can't remember what the root filesystem was, but on a comparable SLES 15.0 system I noticed that snapshot 1 was mounted as root filesystem (for whatever reason):

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=16329060k,nr_inodes=4082265,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=24506344k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
#...
/dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14427)
#...
/dev/sda2 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/.snapshots)
/dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/opt)
/dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local)
/dev/sda2 on /var type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/var)
#...
/dev/sda2 on /root type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/root)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3267512k,mode=700)

So subvol=/@/.snapshots/1/snapshot) seems to be the root cause.