Rys Sommefeldt

FreeBSD 12.0 Spectre mitigation for Linux guests under bhyve

The current stable release of FreeBSD, 12.0-RELEASE, has an issue where the kernel and bhyve, the FreeBSD hypervisor used for running virtual machines, don’t present the host processor(s) to the virtual machine in a way that lets Linux guests that try to mitigate against the Spectre security vulnerability work properly, only on AMD systems.

Spectre mitigation happens by way of checking for specific MSRs and then configuring the host or guest kernel appropriately, depending on what the MSR interaction returns. The issue is that the FreeBSD host kernel is exposing a number of Spectre-specific MSRs to guests unfiltered via bhyve, only on AMD systems, whereas the common kernel and hypervisor tactic is to filter those MSRs from guests.

The issue trivially manifested on my AMD Ryzen Threadripper 2990WX system in the form of guest kernel hangs. It didn’t matter how many virtual CPUs (vCPUs) I presented to the guests, although I did figure out that more than 1 vCPU triggered the bug faster due to the way the Linux kernel brings the CPU up after boot.

Some Linux distributions would let me get through their installer system before breaking, others wouldn’t even get that far due to the installer requiring a full-featured modern kernel. I eventually tracked it down to the MSR exposure: thankfully bhyve lets you ignore “bad” (really just unimplemented) MSRs interaction with the -w switch.

If you use vm-bhyve as your bhyve front-end on FreeBSD, as I do, you expose -w via vm-bhyve’s ignore_bad_msr="yes" config option. Experimenting with the -w switch is what led me to figure it out.

I also figured out that certain Linux distributions have kernels that support spectre_v2_user=off as a boot parameter, to avoid the Spectre v2 mitigation code in the kernel from running at all. That has the same effect as properly handling the MSR in the hypervisor, if running Linux as a guest.

I reported the behaviour as a bug on the FreeBSD mailing list, and the kernel developers took a look. After verifying the Linux kernel behaviour, a small patch was developed to help fix it at a basic level by masking the MSRs in the same way as the FreeBSD kernel does on Intel systems — thankfully AMD and Intel both implement the same MSR control for Spectre, described in the programming manuals that AMD produced for Spectre and Meltdown for programmers in the aftermath of the bugs being exposed.

Another small patch prints more of the CPU configuration based on process flags, to help identify what’s going on. I tested both patches against the 12.0-RELEASE kernel on my system and now Linux guests boot correctly without any special handling in bhyve, or using the mitigation boot parameter on the guest kernel. All of that happened in just 4 days!

There’s more for the FreeBSD kernel developers to do in order to make a full production-ready fix, mostly around VMM sysctls, but at least there’s something to test now for anyone trying to make progress.

I imagine the final fix will be present in a future stable FreeBSD GENERIC kernel on AMD platforms, ideally in the next stable release of the 12 branch of the operating system. I also learned how to get and build a FreeBSD kernel of my own from the base sources, which was fun and good to know!