QEMU Nested virtual machines

Question

I am trying to build a test environment using qemu and virt-manager on Debian 12, to host a Windows 11 guest running WSL2.

I have managed to install Windows 11 (using swtpm and ovmf to meet the requirements for TPM and UEFI), but when I try to install WSL, the installation itself appears to succeed but when the guest reboots to finish the installation process it boots to the Windows 11 automatic repair screen from which it never recovers.

I assumed that the problem had something to do with the CPU emulation not supporting nested virtualisation, so to check the guest could support nested virtualisation I did a fresh install of Debian 12 on the guest itself, and then installed qemu and virt-manager on the guest and before creating a nested guest machine successfully.

The system requirements for WSL seem a little vague (the FAQ just says nested virtualisation is needed), but obviously something isn't quite right as even though I can nest QEMU guests WSL2 wont work!

What do I need to do either to the template below, or on the Windows 11 guest to get WSL2 to work - or is my physical host's processor missing a key feature?

Edit 15/11 : It isn't a hardware problem I can install WSL2 on a Windows 11 guest using VMware Player 17.

Thank you.

Debian 12 installs the following versions of the software.

gir1.2-libvirt-glib-1.0:amd64        4.0.0-2
ipxe-qemu                            1.0.0+git-20190125.36a4c85-5.1
libvirt-clients                      9.0.0-4
libvirt-daemon                       9.0.0-4
libvirt-daemon-config-network        9.0.0-4
libvirt-daemon-config-nwfilter       9.0.0-4
libvirt-daemon-driver-qemu           9.0.0-4
libvirt-daemon-system                9.0.0-4
libvirt-daemon-system-systemd        9.0.0-4
libvirt-glib-1.0-0:amd64             4.0.0-2
libvirt0:amd64                       9.0.0-4
ovmf                                 2022.11-6
python3-libvirt                      9.0.0-1
qemu-system-common                   1:7.2+dfsg-7+deb12u2
qemu-system-data                     1:7.2+dfsg-7+deb12u2
qemu-system-x86                      1:7.2+dfsg-7+deb12u2
qemu-utils                           1:7.2+dfsg-7+deb12u2
swtpm                                0.7.1-1.3
swtpm-libs:amd64                     0.7.1-1.3
swtpm-tools                          0.7.1-1.3
virt-manager                         1:4.1.0-2
virtinst                             1:4.1.0-2

My guest machine was defined using the following XML template.

<domain type='kvm'>
  <name>guest</name>
  <uuid>68171bfe-29f3-4e19-93b6-00b8d9501469</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/11"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-7.2'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'/>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/.../disk1.qcow2'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:fe:66:8e'/>
      <source bridge='br0'/>
      <model type='e1000e'/>
      <link state='up'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'/>
    </tpm>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <audio id='1' type='spice'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Cat /proc/cpuinfo gives the following details (first core only) on the physical host;

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 142
model name  : Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz
stepping    : 12
microcode   : 0xf8
cpu MHz     : 400.000
cache size  : 8192 KB
physical id : 0
siblings    : 8
core id     : 0
cpu cores   : 4
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust sgx bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
vmx flags   : vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple shadow_vmcs pml ept_mode_based_exec
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs taa itlb_multihit srbds mmio_stale_data retbleed eibrs_pbrsb gds
bogomips    : 4599.93
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

and the following results on the guest.

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 142
model name  : Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz
stepping    : 12
microcode   : 0xf8
cpu MHz     : 2303.996
cache size  : 16384 KB
physical id : 0
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust sgx bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves arat umip md_clear arch_capabilities
vmx flags   : vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest shadow_vmcs pml
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs taa srbds mmio_stale_data retbleed eibrs_pbrsb gds
bogomips    : 4607.99
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

# cat /sys/module/kvm_intel/parameters/nested
Y
#

Enabling ignore_msrs as suggested below makes no difference even after reinstalling Windws 11.

# cat /sys/module/kvm/parameters/ignore_msrs
Y
#

QEMU seems to think everything is OK (only a warning and I know secure boot is enabled as Windows 11 worked until WSL was installed).

# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)

Repeated a fresh install of Windows 11 and it works fine until I install WSL then it just goes into a boot/repair loop....

If there is a BSOD I'm not seeing it.

Answer 1

It appears that less really is more in this case.

Disabling the SGX CPU feature in the physical host's BIOS allowed me to install WSL2 on Windows 11 running as a QEMU guest. No changes to the QEMU or guest configuration were required.

I suspect it may have something to do with the EPC memory sections discussed here (or possibly the lack of them) but right now I simply don't know enough about this subject to be able to say.

Hope this helps someone!

Answer 2

I believe it really got to the SYSTEM THREAD EXCEPTION NOT HANDLED BSOD repeatedly, which would eventually lead it to automatic repair.

It appears to be a long-time issue which can be worked around by setting ignore_msrs of the kvm module to 1.

To do so, create /etc/modprobe.d/kvm.conf and have the following line in the file:

options kvm ignore_msrs=1

Then run the following with sudo / as root:

modprobe -r kvm_intel
modprobe kvm_intel

(Yes, kvm_intel, not kvm. Make sure you include the -r, so that kvm would be unloaded as well. You can just reboot instead as well if you prefer.)

You can check that /sys/module/kvm/parameters/ignore_msrs has Y to confirm that the configuration is in effect.

P.S. This assumes that your distro have both kvm and kvm_intel built as module. If it doesn't, have kvm.ignore_msrs=1 in your kernel command line instead.

Answer 3

After installing Docker Desktop (which used WSL by default) in a Windows 11 guest on a Linux host, the Windows virtual machine was not booting but the issue was fixed with the comment from AnthonyK (QEMU Nested virtual machines):

disabling CPU Passthrough and then clicking on Clear CPU configuration in the Model dropdown in Virt-manager resolved the issue.

However, Docker Desktop and WSL were still not working but I found a solution (https://stackoverflow.com/questions/62274613/run-wsl2-in-windows-10-guest-vm-running-on-a-linux-host/70321015#70321015 based on Kvm nested Virtualbox windows guest) by searching "virt-manager wsl Passthrough" on Google.

I had to adapt the solution by adding the last line in the XML settings of virt-manager due to a bug (https://gitlab.com/libvirt/libvirt/-/issues/608):

<cpu mode="custom" match="exact" check="partial">
    <model fallback="allow">Skylake-Client-noTSX-IBRS</model>
    <feature policy="disable" name="hypervisor"/>
    <feature policy="require" name="vmx"/>
    <feature policy='disable' name='mpx'/>
</cpu>

Note: I had a temporary boot error that was fixed by using all the CPU cores in virt-manager configuration but this does not seem necessary anymore. There was also a boot error when I tried to change the RAM from 24576 MiB to 22528 MiB. I saw that Docker Desktop is working by running the "hello world" program, which may take a while to install.