How to create a linux container which inherits the hosts root file system, possibly with zfs?

Question

What

Hopefully the question in the title makes sense. I am wanting to create a low overhead (therefore no virtual machine) copy of the host operating system, with it running in a linux container.

I am thinking of somehow using zfs to create a snapshot of the hosts rootfs and then feeding that into lxc somehow. That way any changes in the container are constrained to the container via ZFS's copy-on-write functionality, and any future changes to the hosts rootfs do not propagate to the container.

Are there any potential issues I have to take account of, such as recursive directories wreaking havoc or anything else?

Why

The reasoning for this is so I can quickly use the host as a template and play with, for example, installing new build tools that pollute the rootfs without worrying about that effecting the host, and not have to waste lots of drive space copying the hosts rootfs. Also, I spent a decent amount of time setting up the host the way I like and don't want to have to spend a few days again creating a template of the process (even though I should, writing it all down is a good idea, automating it is even better). Since this would be in a linux container, I can have multiple instances running at once.

Answer 1

I drafted a manual procedure that can accomplish this.

Prerequisites

These procedures make the following assumptions:

• Your shell is /bin/bash.
• You are root.
• You have deployed the operating system on ZFS pool rpool and also want the LXD container on the same ZFS pool (rpool).
• Your host rootfs is installed to the rpool/ROOT/os ZFS dataset.
• You took a snapshot of your host rootfs and called it rpool/ROOT/os@20180516T091126CDT.
• You're running the Snappy package of LXD.
• You have an lxc storage named rpool using the zfs driver on the source rpool/lxd.
• You want to create an unprivileged container called demo.

You'll have to tweak the procedure for every deviation from the above requirements.

Instructions

1. Create an LXC container using an image similar to your host operating system:

    root@node51 [~]# lxc launch images:ubuntu/18.04 demo -s rpool
    Creating demo
    Starting demo
   

2. Stop the container:

    root@node51 [~]# lxc stop demo
   

3. Mount the LXC storage volume so that we can fetch some metadata from it:

    root@node51 [~]# zfs mount rpool/lxd/containers/demo
   

4. Copy the metadata out somewhere (like /tmp/demo/):

    root@node51 [~]# rsync -avHXShPs --exclude rootfs/ /var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo/ /tmp/demo/
    sending incremental file list
    created directory /tmp/demo
    ./
    backup.yaml
              2.05K 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=4/6)
    metadata.yaml
                529 100%  516.60kB/s    0:00:00 (xfr#2, to-chk=3/6)
    templates/
    templates/hostname.tpl
                 21 100%   20.51kB/s    0:00:00 (xfr#3, to-chk=1/6)
    templates/hosts.tpl
                140 100%  136.72kB/s    0:00:00 (xfr#4, to-chk=0/6)
   
    sent 3.12K bytes  received 135 bytes  6.50K bytes/sec
    total size is 2.74K  speedup is 0.84
   

5. Delete the ZFS dataset that LXC created:

    root@node51 [~]# zfs destroy rpool/lxd/containers/demo
   

6. Clone the ZFS dataset to the same name that LXC would expect:

    root@node51 [~]# zfs clone rpool/ROOT/os@20180516T091126CDT rpool/lxd/containers/demo
   

7. Set the mountpoint to the original mountpoint:

    root@node51 [~]# zfs set mountpoint=/var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo rpool/lxd/containers/demo
   

8. Create a rootfs directory for the new container data:

    root@node51 [~]# mkdir -v /var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo/rootfs/
    mkdir: created directory '/var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo/rootfs/'
   

9. Extend your shell's globbing functionality to ensure that the upcoming mv takes all the file system data:

    root@node51 [~]# shopt -s extglob ; shopt -s dotglob
   

10. Shorten the upcoming commands a bit by cding into the container's dataset:

    root@node51 [~]# cd /var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo/
    root@node51 [/var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo]#
    

11. Move all the container's data into the rootfs/ folder:

    root@node51 [/var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo]# mv !(rootfs) rootfs/
    

12. Create some folders that are needed for the container to boot:

    root@node51 [/var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo]# mkdir rootfs/{dev,sys,proc}
    

13. Move the metadata backed up from earlier into the container's dataset:

    root@node51 [/var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo]# mv /tmp/demo/* .
    

14. Remove the empty temporary directory from the metadata backup:

    root@node51 [/var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo]# rm -rfv /tmp/demo
    removed directory '/tmp/demo'
    

15. Return to your previous directory so that you can unmount the container's dataset:

    root@node51 [/var/snap/lxd/common/lxd/storage-pools/rpool/containers/demo]# cd -
    /root
    

16. Unmount the container's dataset so that LXC can take it over:

    root@node51 [~]# zfs umount rpool/lxd/containers/demo
    

17. To tell LXC to convert the container's files to unprivileged on the next start, run

    lxc config edit demo
    

    and change the line that reads

    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    

    to

    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":0,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":0,"Nsid":0,"Maprange":1000000000}]'
    

18. Start the container.
    This will take a while because every file in the container's rootfs is being converted to unprivileged. There is no progress indicator.

    root@node51 [~]# lxc start demo
    

19. Enter the container:

    root@node51 [~]# lxc exec demo -- bash
    

    From here, you can configure networking, your systemd startup sequence, and/or other things you need to get this LXC container clone of your host up and running.