rsync permissions errors at destination even though root, possibly due to sticky bit?

Question

I'm using rsync to backup local files to a usb disk. That has been working fine for years, but now rsync is reporting filesystem permission errors, even though run as root on a rw filesystem. This is the command that I run and part of the log outputs:

$ sudo rsync --verbose --archive --delete --partial --inplace /var/ /mnt/backup/var
[...]
log/postgresql/postgresql-12-main.log
log/postgresql/postgresql-12-main.log.1
log/postgresql/postgresql-12-main.log.10.gz
log/postgresql/postgresql-12-main.log.2.gz
log/postgresql/postgresql-12-main.log.3.gz
log/postgresql/postgresql-12-main.log.4.gz
log/postgresql/postgresql-12-main.log.5.gz
log/postgresql/postgresql-12-main.log.6.gz
log/postgresql/postgresql-12-main.log.7.gz
log/postgresql/postgresql-12-main.log.8.gz
log/postgresql/postgresql-12-main.log.9.gz
spool/anacron/cron.daily
tmp/
tmp/systemd-private-3bbae922a969453097f3c87f9cfcc52d-fwupd.service-iiu3gi/
tmp/systemd-private-3bbae922a969453097f3c87f9cfcc52d-fwupd.service-iiu3gi/tmp/
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.1" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.10.gz" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.2.gz" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.3.gz" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.4.gz" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.5.gz" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.6.gz" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.7.gz" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.8.gz" failed: Permission denied (13)
rsync: open "/mnt/backup/var/log/postgresql/postgresql-12-main.log.9.gz" failed: Permission denied (13)

sent 568,313,680 bytes  received 11,776 bytes  32,475,740.34 bytes/sec
total size is 9,562,808,631  speedup is 16.83
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]

As said, I'm running this as root and the destination filesystem is mounted rw. See these mount outputs:

/dev/mapper/ubuntu--vg-root on / type ext4 (rw,relatime,errors=remount-ro)
/dev/mapper/backup-backup on /mnt/backup type ext4 (rw,nosuid,nodev,noexec,relatime,user)

Notice that rsync only complains about file permissions in the destination directory. Also, I only observe these errors, if some of the files have been changed in the source directory. If the files are missing in the destination directory, rsync copies them just fine. If the files have not changed since previous run, rsync detects that successfully and omits them from copying. It does not emit errors or warnings in either of these situations.

I'm not experiencing much trouble with other files that the above rsync command processes. Everything looks just fine for these.

One thing that I notice though, is that the parent directory of the failing files has its sticky flag set:

$ sudo ls -la /mnt/backup/var/log/postgresql/
total 48
drwxrwxr-t  2 root     postgres 4096 Mar  6 00:00 .
drwxrwxr-x 17 root     syslog   4096 Mar  6 00:00 ..
-rw-r-----  1 postgres adm         0 Mar  6 00:00 postgresql-12-main.log
-rw-r-----  1 postgres adm      1873 Mar  6 00:00 postgresql-12-main.log.1
-rw-r-----  1 postgres adm       437 Dec 12 09:37 postgresql-12-main.log.10.gz
-rw-r-----  1 postgres adm       701 Feb 27 00:00 postgresql-12-main.log.2.gz
-rw-r-----  1 postgres adm       436 Feb 20 00:00 postgresql-12-main.log.3.gz
-rw-r-----  1 postgres adm       435 Feb 13 00:00 postgresql-12-main.log.4.gz
-rw-r-----  1 postgres adm       535 Feb  6 00:00 postgresql-12-main.log.5.gz
-rw-r-----  1 postgres adm       720 Jan 30 00:00 postgresql-12-main.log.6.gz
-rw-r-----  1 postgres adm       453 Jan 23 00:00 postgresql-12-main.log.7.gz
-rw-r-----  1 postgres adm       453 Jan 13 10:49 postgresql-12-main.log.8.gz
-rw-r-----  1 postgres adm       441 Jan  5 22:00 postgresql-12-main.log.9.gz

But so do some other directories that I throw at rsync and those do not cause any errors. Needless to say that I can use other CLI tools to manipulate the above files without any trouble as long as I do it as root.

I get the feeling that I must be missing something really obvious. Can anyone tell me what is going on here? And maybe propose some rsync flags or mount options that could fix this?

Finally, here is some version info:

rsync  version 3.1.3  protocol version 31
Ubuntu 20.04.4 LTS

Update: Outputs of sysctl fs:

$ sudo sysctl fs | grep fs.protect
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1

Answer 1

In addition to its usual meaning, the 'sticky' flag causes Linux to apply additional protection as a precaution against many security issues in the past, which were caused by privileged services being careless about the shared /tmp directory (such as following symlinks planted by a user and being tricked into overwriting files in /etc).

• If the fs.protected_regular sysctl is non-zero, then only the file's owner is permitted to open existing files for "create + write" (O_CREAT), unless those files are owned by the same uid as the directory itself.

  If fs.protected_regular is set to 2, this also applies to directories that are just g+w, not only o+w.

• (Similarly, if the fs.protected_symlinks sysctl is non-zero, then symlinks within a sticky directory can only be followed by the link's owner. For example, if user A symlinks /etc/passwd to /tmp/passwd, then only user A may attempt to open the /tmp/passwd symlink; everyone else including root is not allowed.)

Answer 2

I found out that my problem can also be solved by removing the --inplace option from my rsync command. That will cause rsync create a new file and delete the old one.

That will kinda outsmart the sticky bit functionality described in user1686's answer (see comments there). Without --inplace, rsync will never actually open the existing file for write, so fs.protected_regular=2 does not cause any trouble.

Not sure, if that works for all combinations of permissions on directories with the sticky bit and the files therein. But it works for me so far.

That said, I opted for user1886's answer instead, so I'll leave that as the accepted answer.

Answer 3

Maybe something like

find /mnt/backup/var type d perm /1000 print0 > sticky_dirs
xargs -0 --arg-file=sticky_dirs chmod -t
your rsync command here
xargs -0 --arg-file=sticky_dirs chmod +t

to turn off all the sticky bits, and then restore them.