Encryption at rest for VM storage in a hypervisor done properly/elegantly

Question

Given a very simple infrastructure - a single machine running a hypervisor(Proxmox if it matters), I want to achieve a good data protection against the hardware being stolen.

Let's say there are various VM's running on the hypervisor - most of those don't need to be encrypted, but for the NAS/personal cloud ones I want the data to be safe if the hardware gets stolen.

I would like to get some ideas on the best practices to handle such scenarios. I've thought about it a bit and came up with several approaches from which I have hard time choosing which is optimal and if there are any problems/risks that I'm overlooking.

Full disk encryption of the host

This is out of the question, because the hardware is often left unattended, so it has to be able to recover in a reboot on itself and some of the VM's(CCTV, web servers, etc) need to start without intervention

Encrypting the disk/partition containing the VM volumes on the host

An easy approach would be encrypting a disk that would hold the sensitive VM volumes.

What I like in this approach is I can group all qcow2 volumes I need into a storage bucket that's locked on startup and I manually unlock it via SSH or a custom API.

What I don't like is this has to live on the Proxmox host and I'd like to keep it as clean as possible. Also a downside is that I can't boot a VM without some of its vm disks available(I hoped it would behave just like trying to boot a real computer with a harddisk removed). This would basically mean I can't have the NAS boot without the sensitive disk attached.

There is a way to programmatically detach the disk from the vm, and attach it on unlocking, but again I would like to keep the Proxmox side as clean as possible.

A third potential drawback are backups, since the actual qcows and FS's inside are not encrypted, so I have to take additional care of encrypting the backup itself.

Delegate all encryption to the VM guests

The cool thing here is each VM can handle the encryption on its own - like a VM that I want fully encrypted, I can turn on full disk encryption on the system drive and need to connect via VNC/console to unlock it on boot, OR, I can have a single partition/vm disk encrypted on the guest side(for example the NAS storage drives I need to be safe).

On the downside I need to unlock each VM separately, also this brings the responsibility to maintain/backup the luks headers for each partition encrypted, and I'm concerned if so many levels of abstraction of the storage could become fragile - in this case it would be the FS of the host, a qcow2 storage on top of it, then inside - a virtual disk, then on top of it - luks/dm-crypt, then the fs and partitions themselves.

The huge upside is this would be completely irrelevant for the host so backups will have the guest fs's encrypted, no modifications of the host, and most flexibility on the VM's. I think sounds the most convenient and "compatible" way of doing it, treating the VM's as standalone machines. My only worry here is this many levels of abstraction - am I paranoid?

Encrypted hdd passthrough

I'm currently using this approach where the sensitive storage is a fully encrypted separate disk, that's passed-through into the NAS system. Although this feel safest, as I don't have additional layers of abstraction, this causes me some pains in terms of automatic backups and Proxmox not being particularly friendly to passed-through disks - it doesn't count them as standard storage, doesn't expose any UI for maintaining them, etc.

The good side is, it's a completely standard data drive that I can just unplug, put into another PC and decrypt/read.

---

So, from an elegance standpoint I think I'm leaning towards option 3 - treating each VM as a standalone PC that bears the responsibility for its encryption - whether it's the whole disk encryption for the VM, or encrypting a single partition/virtual disk inside.

I have several concerns which I have no idea how valid or sensible are:

• Is there any "safe limit" to the size of a qcow2 file? Like, is there any downside in having a storage virtual disk in a qcow2 format with a size of, say, 1TB?
• Is there any real additional in having these additional abstraction levels in storage in terms of hidden data corruption, etc? I suppose the ordeal of data recovery of such a filesystem would be hellish, so reliable backups become even more paramount?
• Is there a "standard" way of handling storage dependencies in services running under Linux? Meaning, let's say I let the NAS boot, but its backup storage isn't unlocked. A separate VM that depends on this storage via NFS won't be able to mount it, so if whatever service uses the 2nd VM, it would start writing in the mount point. I surely can script my way out of this, but I don't want to rediscover the wheel if good practices are already laid out.

Any ideas or advice are very welcome.

Answer 1

Full disk encryption of the host

I think either you don't understand Full disk encryption, or you haven't had enough experience with this.

There's a number of ways of doing this in a perfectly servicable way - one option is just to use SEDs (self-encrypting disks), it's totally transparent to the disk controller and above (OS, hypervisor, VMs). Another is to get the controller to manage the encryption - HPe controllers are particularly good at this - you set a key on the controller and it encrypts/decrypts onto any regular disks - again this is transparent to the OS etc.

Either of these are secure, performant and requires no input from the OS, and allow for unattended reboots.

Answer 2

2.29. Encryption of Disk Images

Oracle VM VirtualBox enables you to transparently encrypt the data stored in hard disk images for the guest. It does not depend on a specific image format to be used. Images which have the data encrypted are not portable between Oracle VM VirtualBox and other virtualization software.

Oracle VM VirtualBox uses the AES algorithm in XTS mode and supports 128-bit or 256-bit data encryption keys (DEK). The DEK is stored encrypted in the medium properties and is decrypted during VM startup by entering a password which was chosen when the image was encrypted.

Since the DEK is stored as part of the VM configuration file, it is important that it is kept safe. Losing the DEK means that the data stored in the disk images is lost irrecoverably. Having complete and up to date backups of all data related to the VM is the responsibility of the user. 2.29.1. Limitations of Disk Encryption

There are some limitations the user needs to be aware of when using this feature:

This feature is part of the Oracle VM VirtualBox Extension Pack, which needs to be installed. Otherwise disk encryption is unavailable.

Since encryption works only on the stored user data, it is currently not possible to check for metadata integrity of the disk image. Attackers might destroy data by removing or changing blocks of data in the image or change metadata items such as the disk size.

Exporting appliances which contain encrypted disk images is not possible because the OVF specification does not support this. All images are therefore decrypted during export.

The DEK is kept in memory while the VM is running to be able to decrypt data read and encrypt data written by the guest. While this should be obvious the user needs to be aware of this because an attacker might be able to extract the key on a compromised host and decrypt the data.

When encrypting or decrypting the images, the password is passed in clear text using the Oracle VM VirtualBox API. This needs to be kept in mind, especially when using third party API clients which make use of the webservice where the password might be transmitted over the network. The use of HTTPS is mandatory in such a case.

Encrypting images with differencing images is only possible if there are no snapshots or a linear chain of snapshots. This limitation may be addressed in a future Oracle VM VirtualBox version. 

2.29.2. Encrypting Disk Images

Encrypting disk images can be done either using the GUI or VBoxManage. While the GUI is easier to use, it works on a per VM basis and encrypts all disk images attached to the specific VM. With VBoxManage one can encrypt individual images, including all differencing images. To encrypt an unencrypted medium with VBoxManage, use:

VBoxManage encryptmedium uuid|filename
--newpassword filename|- --cipher cipher-ID --newpasswordid "ID

To supply the encryption password point VBoxManage to the file where the password is stored or specify - to let VBoxManage ask you for the password on the command line.

The cipher parameter specifies the cipher to use for encryption and can be either AES-XTS128-PLAIN64 or AES-XTS256-PLAIN64. The specified password identifier can be freely chosen by the user and is used for correct identification when supplying multiple passwords during VM startup.

If the user uses the same password when encrypting multiple images and also the same password identifier, the user needs to supply the password only once during VM startup. 2.29.3. Starting a VM with Encrypted Images

When a VM is started using the GUI, a dialog will open where the user needs to enter all passwords for all encrypted images attached to the VM. If another frontend like VBoxHeadless is used, the VM will be paused as soon as the guest tries to access an encrypted disk. The user needs to provide the passwords through VBoxManage using the following command:

VBoxManage controlvm uuid|vmname addencpassword ID password [--removeonsuspend yes|no]

ID must be the same as the password identifier supplied when encrypting the images. password is the password used when encrypting the images. Optionally, you can specify --removeonsuspend yes|no to specify whether to remove the password from VM memory when the VM is suspended. Before the VM can be resumed, the user needs to supply the passwords again. This is useful when a VM is suspended by a host suspend event and the user does not want the password to remain in memory. 2.29.4. Decrypting Encrypted Images

In some circumstances it might be required to decrypt previously encrypted images. This can be done in the GUI for a complete VM or using VBoxManage with the following command:

VBoxManage encryptmedium uuid|filename --oldpassword file|-

The only required parameter is the password the image was encrypted with. The options are the same as for encrypting images.