Modern versions of Windows support GPU paravirtualization in Hyper-V with normal consumer graphics cards. This is used e.g. for graphics acceleration in Windows Sandbox, as well as WSLg. In some cases, it may be useful to create a normal VM with GPU acceleration using this feature, but this is not officially supported. People already figured out how to do it with Windows guests though, so why not do the same with Linux? It should be easy given that WSLg is open source and reasonably well documented, right?
Well... not quite. I managed to get it to run... but not well.
- Verify driver support
Run Get-VMHostPartitionableGpu in PowerShell. You should see your graphics card listed, if you get nothing, update your graphics drivers and try again.
- Create a new VM in Hyper-V Manager.
Make sure to:
- Use Generation 2
- DISABLE dynamic memory (it interferes with vGPU on Windows so it probably won't work on Linux either, I didn't check this yet though)
- DISABLE automatic snapshots (they are not supported with vGPU and will only cause problems)
- DISABLE secure boot (we'll need custom kernel drivers, and I never tried to make this work with secure boot)
- Don't forget to add more CPU cores because the stupid wizard still adds only one vCPU...
- Add GPU-PV adapter
From PowerShell running as administrator:
Set-VM -VMName <vmname> -GuestControlledCacheTypes $true -LowMemoryMappedIoSpace 1GB -HighMemoryMappedIoSpace 32GB
Add-VMGpuPartitionAdapter -VMName <vmname>-
Install Ubuntu 21.04 in the VM like you would usually
-
Build the
dxgkrnldriver
Until Microsoft upstreams this driver to the mainline Linux kernel, you will have to build it manually. Use the following script I made to get the driver from the WSL2-Linux-Kernel tree, patch it for out-of-tree build and add it to DKMS:
#!/bin/bash -e
BRANCH=linux-msft-wsl-5.10.y
if [ "$EUID" -ne 0 ]; then
echo "Swithing to root..."
exec sudo $0 "$@"
fi
apt-get install -y git dkms
git clone -b $BRANCH --depth=1 https://github.com/microsoft/WSL2-Linux-Kernel
cd WSL2-Linux-Kernel
VERSION=$(git rev-parse --short HEAD)
cp -r drivers/hv/dxgkrnl /usr/src/dxgkrnl-$VERSION
mkdir -p /usr/src/dxgkrnl-$VERSION/inc/{uapi/misc,linux}
cp include/uapi/misc/d3dkmthk.h /usr/src/dxgkrnl-$VERSION/inc/uapi/misc/d3dkmthk.h
cp include/linux/hyperv.h /usr/src/dxgkrnl-$VERSION/inc/linux/hyperv_dxgkrnl.h
sed -i 's/\$(CONFIG_DXGKRNL)/m/' /usr/src/dxgkrnl-$VERSION/Makefile
sed -i 's#linux/hyperv.h#linux/hyperv_dxgkrnl.h#' /usr/src/dxgkrnl-$VERSION/dxgmodule.c
echo "EXTRA_CFLAGS=-I\$(PWD)/inc" >> /usr/src/dxgkrnl-$VERSION/Makefile
cat > /usr/src/dxgkrnl-$VERSION/dkms.conf <<EOF
PACKAGE_NAME="dxgkrnl"
PACKAGE_VERSION="$VERSION"
BUILT_MODULE_NAME="dxgkrnl"
DEST_MODULE_LOCATION="/kernel/drivers/hv/dxgkrnl/"
AUTOINSTALL="yes"
EOF
dkms add dxgkrnl/$VERSION
dkms build dxgkrnl/$VERSION
dkms install dxgkrnl/$VERSION- Copy GPU drivers from your host system
Now you will also need to copy some files from the host machine: the closed-source D3D12 implementation provided by Microsoft, as well as Linux parts of the graphics driver provided by your GPU vendor. If you ever tried to run GPU-PV with a Windows guest, this part should look familiar. Figuring out how to transfer the files into the VM is left as an exercise to the reader, I'll just assume that your Windows host volume is available at /mnt for simplicity:
mkdir -p /usr/lib/wsl/{lib,drivers}
cp -r /mnt/Windows/system32/lxss/lib/* /usr/lib/wsl/lib/
cp -r /mnt/Windows/system32/DriverStore/FileRepository/nv_dispi.inf_amd64_* /usr/lib/wsl/drivers/ # this may be different for different GPU vendors, refer to tutorials for Windows guests if needed
chmod -R 0555 /usr/lib/wslNote: You will need to repeat this step every time you update Windows or your graphics drivers
- Set up the system to be able to load libraries from
/usr/lib/wsl/lib/:
echo "/usr/lib/wsl/lib" > /etc/ld.so.conf.d/ld.wsl.conf
ldconfig # (if you get 'libcuda.so.1 is not a symbolic link', just ignore it)- Workaround a bug in the D3D12 implementation (it assumes that the
/usr/lib/wsl/lib/mount is case-insensitive... just Windows things...)
ln -s /usr/lib/wsl/lib/libd3d12core.so /usr/lib/wsl/lib/libD3D12Core.so- Reboot the VM
If you've done everything correctly, glxinfo | grep "OpenGL renderer string" should display D3D12 (Your GPU Name). If it does not, here are some useful commands for debugging:
sudo lspci -v # should list the vGPU and the dxgkrnl driver
ls -l /dev/dxg # should exist if the dxgkrnl
/usr/lib/wsl/lib/nvidia-smi # should be able to not fail :P- The thing is UNSTABLE. Just running
glxgearscrashes GNOME, spectacularly. I'd recommend switching to a simple window manager like i3 for testing. - GPU acceleration doesn't seem to be picked up everywhere, sometimes it falls back to software rendering with llvmpipe for no apparent reason
- While when it works you can clearly see that the GPU is working from the FPS counter... I didn't figure out a good way to get these frames from the VM at a reasonable rate yet! The Hyper-V virtual display is slow, and even if you get enhanced session to work it's just RDP under the hood which is not really designed for high FPS output either. On Windows, you can simply use something like Parsec to connect to the VM, but all streaming solutions I know of don't work on a Linux host at all.
Hello. I can get this working passing through a 3070 to an Ubuntu 22.04 VM but if I also try to pass through a PCI device (Coral TPU) using DDA, the video card is very unstable and causes FFMPEG to segfault. Either GPU-PV alone or DDA alone work fine but don't play well together on the VM. It was really cool to be able to use GPU-PV in Hyper-V.
Anyway, I was wondering if you saw the D3D12/VAAPI capability that was added to WSL2 and if you had any thoughts about how to enable VAAPI for Hyper-V VMs.
Thanks.