Here's my use-case: I have roughly a dozen machines loosely clustered together, each with a few hundred GB partition that's currently not used. I would like to make a scratch partition so that users can dump data locally rather than stressing the single NFS server for heavy writes, if they don't need to.
The catch that makes this a little bit difficult is that it's quite awkward to tie the storage to a single machine. I would like to present to the user a file system that writes locally, but reads (and deletes!) globally.
Something like a normal unionfs/aufs/overlayfs (combined with NFS) I think would work, expect that to actually delete something a user would have to hunt down the specific local mount where it was created and delete it there: deleting something on the master unionfs wouldn't actually free up space. Additionally, the overlayfs docs (at least, I expect the others wouldn't like it either) say that changing the underlying fileystem while it's in use is a bad idea that will produce undefined behaviour.
The other option I've looked into a little bit is a full clustered filesystem (ceph, lustre, etc), but that both seems like a lot of overhead and I don't think I actually want redundant distributed storage. It is actually preferable if writes by a machine get handled by the disk in that machine and don't go off to cause problems elsewhere.
Is there a way of accomplishing this sort of thing in a practical (and relatively easy) manner?
