Get Illegal Instruction error when booting Linux in VirtualBox, works fine when booted directly

Question

I have a computer on which I am dual booting Windows 7 and Gentoo Linux (both 64-bit). I want to be able to load up my Linux installation in a VM while I am booted into Windows. I have installed VirtualBox and followed the instructions for creating a raw disk VMDK. When I start the VM, Linux starts booting, but then fails with the following error when unlocking my root partition:

truecrypt[441] trap invalid opcode ip:373615538e0 sp:3dd0e0dfb60 error:0 in libpixman-1.so.0[373614d6000+8d000]

Everything works fine when I boot into Linux directly.

What could cause an illegal instruction to be hit in libpixman only when booting in VirtualBox?

Update:

As a troubleshooting step, I recompiled pixman without "-march", and no longer get an illegal instruction error in that library. (The boot fails in the same spot with the same error in a different library, however.)

How can I determine the specific opcode that isn't working in VirtualBox so I can disable it in my CFLAGS without having to disable all CPU-specific optimizations?

I am still confused as to why there would be any user-mode instruction that would fail to work in a VM. Is this a known limitation? My CPU is an Intel Core i7 3720QM, and I have hardware virtualization support enabled.

Answer 1

It turned out the problem was that "-march=native" resulted in the compiled system binaries containing SSE4 and AVX instructions, which are not supported by the version of VirtualBox I was using. Explicitly disabling those in my compiler flags and recompiling allowed the system to boot properly under VirtualBox.