1

My problem is quite specific:

I need to run an application that has 500 GB but I have two HDs: a fast 200 GB and a very very slow 2TB.

The 2TB HD is external USB and was mounted over the network (NFS) on a very slow computer. It's speed is just 1 MB/s. The 200 GB HD has a speed of 200 MB/s.

Filesystem                             Size  Used Avail Use% Mounted on
--------------------------------------------------------------------------
/dev/sda2                              219G  7.1G  201G   4% /
192.168.1.10:/mnt/remote               1.8T  456G  1.3T  27% /mnt/remote-hd

I want to "mount" a "virtual HD" like:

/mnt/local_plus_remote

Which is the transparent blend of these two HDs.

There is an additional problem:

This slow HD, because it is over the network, is not reliable, sometimes it can fall and be out for a few hours.

I know the app often only uses ~10GB and the rest is like a "archive" that is rarely consulted. Is there any tool that makes this union intelligently? Leaving the least accessed files on the slower HD and the most accessed files on the fastest part? (More or less the swap does?)

I believe that such a tool does not exist, but it doesn't hurt to ask...

Improve this question
4
  • Maybe this tool? github.com/trapexit/mergerfs Possible duplicated: serverfault.com/questions/191299/…
    – Felipe
    Commented Jun 19, 2022 at 21:34
  • Do you have to use NFS to mount the drive? You may have more options if you can expose it as a block device rather then an NFS mount. I'd not recommend it (its all round easier and better to just get another 500 gig hard drive as local storage...) but if you can expose it as a block device you can use caching techniques like lvmcache or bcache or dm-cache.
    – davidgo
    Commented Jun 20, 2022 at 11:16
  • What do you mean by "expose it as a block device"? What should I do?
    – Felipe
    Commented Jun 21, 2022 at 6:40
  • 1
    A block device is skmething that appeara to the OS as a single disk or partition. iSCSI is one means of doing this, but there are others. The idea being that Linux will handle a block device like a hard drive (rather then a Network File System) and will allow you to "build up" your storage without the constraints of filesystems and file access controls. Its harder to do but faster and more powerful thrn file level solutions.
    – davidgo
    Commented Jun 21, 2022 at 10:56

1 Answer 1

Reset to default
2

At the end, I used the tool mergerfs. This tool has a lot of interesting options, but it doesn't have any "smart" or "automatic" solution. You have to study your case and setup according.

In my case, the final solution was:

sudo mergerfs -o allow_other,use_ino,category.action=ff,category.create=ff,category.search=ff,cache.open=30,cache.statfs=30,cache.attr=30,cache.entry=5,cache.negative_entry=5,cache.files=auto-full,cache.writeback=true,cache.symlinks=true,cache.readdir=true,dropcacheonclose=true /home/doritos/doritos_blocks:/mnt/fandangos-remote-hd/doritos/blocks=RO /mnt/doritos+fandangos_blocks

The "RO" option says it is ready-only. So, I just copied all big files that I know won't be changed to the slow and big disk and marked it as "read only". The other new files is created in the fast and small disk.

Of course you should adapt it to your specific case.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .