I would open a support ticket with synology. You have excellent details and perhaps they have a patch or better setup procedure for your specific version of macOS and storage software.
- specifically ask them if you could be using a sparse bundle for storage and if they can automate compacting of that bundle
As for your existing backup date - since space is tight, I would manually clean up some old backups using command line tools. Check for sparsebundle files which optimize network traffic and also need to be compacted periodically if space gets tight.
As you delete a few old backups, you can watch how much space is cleared and then attempt a backup after each cleanup.
Lastly, if you think you have enough space and don’t agree with the estimate for needed space, exclude some large folders and/or system files temporarily from backup and attempt a more limited backup. Eventually if you exclude enough “space” you can often get a small backup to run. After successful backups, the automatic pruning is more vigorous in my experience. After a few hours of clean runs, reverse the exclusions and see if your intended coverage now works with your storage.
Don’t overlook using an inexpensive direct attached disk as well to get one clean backup to a new destination that will let you know the Mac data is good and the problem is only with the synology side or existing data.
