ArseniyShestakov/gpu-hotplug.md

## gpu-hotplug.md

      
    Raw
  

              gpu-hotplug.md
            
          
    Check it's working

DRI_PRIME=1 glxinfo | grep OpenGL
Unbind AMD GPU from Radeon driver

echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id
Use it in QEMU

qemu-system-x86_64 

-name "win81_amd" 


-bios /usr/share/seabios/bios.bin 

-enable-kvm 


-usb 

-device usb-kbd 

-device usb-mouse 


-vga none 

-device qxl 


-m 12G 


-no-hpet 

-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic 

-smp 4,sockets=1,cores=2,threads=2 


-soundhw ac97 


-M q35 

-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio 


-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 

-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on 


-net none 


-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 

-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0 


-monitor telnet:127.0.0.1:1234,server,nowait
Now you need to remove PCI device

echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove
echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove
Rescan will attach it back to Radeon

echo 1 > /sys/bus/pci/rescan
Check it via PRIME once again

DRI_PRIME=1 glxinfo | grep OpenGL

  
## qemu_shared
Introduction:
 This is my configuration for VMs and I'm change it from time to time.

 At moment I'm decide to use following configuration that work best for hardware I have:
  - By default I have "i915.enable_hd_vgaarb=0" because most of time I don't need Nvidia GPU.
  - When I only need VM with AMD HD6950 I boot with option unchanged to keep Intel DRI working on host
  - Once I need to run both VMs simultaneously I just change it to "i915.enable_hd_vgaarb=1" at boot time.
  - Have two sets of options for AMD VM because w/o Nvidia one I have more RAM and don't to use taskset


Hardware:
 - ASRock Z87 Extreme6
 - i7 4771 and 32GB RAM
 - 60GB SSD (for /), 2x 250GB HDDs in RAID (for /home), 1TB HDD with NTFS partition for Windows content
 - AMD HD 6950 1GB and Nvidia GTX 260


Software:
 - Ubuntu 14.04
 - Linux Kernel 3.19 with ACS and i915 VGAarbiter patches
   - Preemption Model set to Low Latency Desktop
   - Timer frequency set to 1000HZ
 - QEMU 2.0.0~rc1+dfsg-0ubuntu3 from repositories
   - In past I used QEMU 2.2.1 from ppa:syseleven-platform/virtualization, but then I decide to use older Nvidia driver instead.
     Also with 2.2.1 I had weird disconnect issue in "Path of Exile" game while changing areas.


System configuration:
 - /etc/default/grub:
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.enable_hd_vgaarb=1 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"
 - /etc/modprobe.d/blacklist-gpu.conf:
    blacklist radeon
    blacklist nouveau
 - /etc/qemu/bridge.conf:
    allow br0
 - /etc/sysctl.d/40-hugepage.conf:
    vm.nr_hugepages = 8200
 - /etc/fstab:
    hugetlbfs /dev/hugepages hugetlbfs mode=1770,gid=78 0 0


Both VMs using same "backing files" that not modified in runtime:
- /win81-base.qcow2 [13GB only!], compressed
    This QCOW2 image contain installed and configured Windows 8.1
- /dev/disk/by-uuid/0AD949C94647B394
    It's NTFS partition on 1TB HDD with games and some software installed.
Those files are base for QCOW2 images that used by virtual machines:
qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-amd.qcow2
qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-nvidia.qcow2
qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-amd.qcow2
qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-nvidia.qcow2
So I basically store VM-specific changes to my system drive on SSD and changes for content stored on HDD RAID.


Kernel build:
apt-get install kernel-package
apt-get build-dep linux-image-$(uname -r)
apt-get build-dep linux-image-3.19.0-23-generic
apt-get source linux-image-$(uname -r)
apt-get source linux-image-3.19.0-23-generic
cd linux-*
# Now just moment to hate Ubuntu packaging, I do not use VBOX!
rm -rf ubuntu/vbox
# Comment out virtualbox crap
nano ubuntu/Makefile
# Apply patches (my patch files are above kernel directory)
# Personally I just used patch version published by someone on google drive
https://gist.github.com/ArseniyShestakov/768a426e79e2b901f9eb
patch -p1 < ../override_for_missing_acs_capabilities.patch
patch -p1 < ../i915_317.patch
# Copy your current kernel config
cp -vi /boot/config-`uname -r` .config
# Change some options for better performance
# Processor Type and Features > Preemption Model > Preemptible Kernel (Low Latency Desktop)
#  (CONFIG_PREEMPT=y)
# Processor Type and Features > Timer Frequency > 1000 Hz
# (CONFIG_HZ_1000=y CONFIG_HZ=1000)
make menuconfig
# Now build kernel
fakeroot make-kpkg -j 8 --initrd kernel-image kernel-headers --append-to-version=-acs-igdarb-lowpremption-timer1000hz
# Install built packages
dpkg -i ../*acs-igdarb-lowpremption-timer1000*.deb


All useful information may be found there:
https://bbs.archlinux.org/viewtopic.php?id=162768
https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=0
http://vfio.blogspot.com


My personal expirience of why I use some of options:
https://www.reddit.com/r/linux_gaming/comments/3dr62f/set_up_gpu_passthrough_without_using_the_cli/ct8gr34
https://www.reddit.com/r/linux_gaming/comments/2z2y7h/gpu_passthrough_or_how_to_play_any_game_at_near/cpfh76x

## qemu_vm_amd
# This system use real network card (00:19.0) as I have two in my mobo.


#################################################
# Exclusive configuration when no other VM running (more RAM, all cores)
#################################################
qemu-system-x86_64 \
-name "win81_amd_exclusive" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 10G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 8,sockets=1,cores=4,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net none \
\
-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0


#################################################
# Shared configuration when Nvidia VM is used too:
#################################################
taskset -c 0,2,4,6 qemu-system-x86_64 \
-name "win81_amd" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 8G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 4,sockets=1,cores=2,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net none \
\
-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0

## qemu_vm_nvidia
# Driver on guest is 335.23, it's last driver that no break KVM and Hyper-V
# Anyway this doesn't make much sense because I only run D3D9 games on that card and those not sensetive to old drivers

# Optional removal of existing data in Nvidia VM
# This VM used for testing so I drop all changes on ever reboot
rm /win81-writable-nvidia.qcow2
rm /home/gameshdd-writable-nvidia.qcow2
cp /win81-writable-nvidia.qcow2.backup /win81-writable-nvidia.qcow2
cp /home/gameshdd-writable-nvidia.qcow2.backup /home/gameshdd-writable-nvidia.qcow2

taskset -c 1,3,5,7 qemu-system-x86_64 \
-name "win81_nvidia" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 7G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 4,sockets=1,cores=2,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-nvidia.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-nvidia.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net nic,model=virtio \
-net bridge,br=br0 \
\
-serial stdio \
-nographic

# Nvidia reset. Boot to reset GPU between usages to avoid host freeze.
# Also make it set fans to minimum so it's not so loud when VM not used.
timeout 10 qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 512 \
-cpu host \
-rtc base=localtime \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin \
-boot menu=off,strict=on -vga none -nographic \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on


# Everything else only needed if i915 VGA arbitration not used
# I don't really use it because
##########################################
# kvm=off required to bypass Nvidia driver virtualization limitations
# Hyper-V hv_ options can't be used for same reason

# The way I got consistent VM startup:
 - First boot up "win81_nvidia_reset" configuration with livecd that use Nouveau.
 - Once monitor activated shutdown livecd or just kill QEMU.
   Display will remain with frozen image on it (or shutdown black screen, but active).
 - Now boot "win81_nvidia" configuration.
 - In "Device Manager" disable and enable device.
   Now monitor must lost connection and go idle / switch to different active port.
 - Disable and enable device once again. It's should be working now.

qemu-system-x86_64 \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
\
-m 1G \
\
-cpu host \
-smp 1,sockets=1,cores=1,threads=1 \
\
-name "win81_nvidia_reset" \
-M q35 \
\
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,x-vga=on \
\
-drive file=/opt/kubuntu-15.04-desktop-amd64.iso,id=isocd \
-device ide-cd,bus=ide.1,drive=isocd

## vfio-setup.sh
#!/bin/sh
service lightdm stop

modprobe vfio-pci
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:00:19.0" > /sys/bus/pci/devices/0000:00:19.0/driver/unbind
echo "0000:00:19.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind
echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/remove_id

service lightdm star

## xorg.conf.d
/etc/X11/xorg.conf.d/10-intel.conf:
Section "Device"
    Identifier "Intel Graphics"
    Driver "intel"
    Option "DRI" "3"
EndSection

/etc/X11/xorg.conf.d/20-radeon.conf:
Section "Device"
    Identifier "Radeon"
    Driver "radeon"
    Option "DRI3" "1"
EndSection
	Introduction:
	This is my configuration for VMs and I'm change it from time to time.

	At moment I'm decide to use following configuration that work best for hardware I have:
	- By default I have "i915.enable_hd_vgaarb=0" because most of time I don't need Nvidia GPU.
	- When I only need VM with AMD HD6950 I boot with option unchanged to keep Intel DRI working on host
	- Once I need to run both VMs simultaneously I just change it to "i915.enable_hd_vgaarb=1" at boot time.
	- Have two sets of options for AMD VM because w/o Nvidia one I have more RAM and don't to use taskset



	Hardware:
	- ASRock Z87 Extreme6
	- i7 4771 and 32GB RAM
	- 60GB SSD (for /), 2x 250GB HDDs in RAID (for /home), 1TB HDD with NTFS partition for Windows content
	- AMD HD 6950 1GB and Nvidia GTX 260



	Software:
	- Ubuntu 14.04
	- Linux Kernel 3.19 with ACS and i915 VGAarbiter patches
	- Preemption Model set to Low Latency Desktop
	- Timer frequency set to 1000HZ
	- QEMU 2.0.0~rc1+dfsg-0ubuntu3 from repositories
	- In past I used QEMU 2.2.1 from ppa:syseleven-platform/virtualization, but then I decide to use older Nvidia driver instead.
	Also with 2.2.1 I had weird disconnect issue in "Path of Exile" game while changing areas.



	System configuration:
	- /etc/default/grub:
	GRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.enable_hd_vgaarb=1 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"
	- /etc/modprobe.d/blacklist-gpu.conf:
	blacklist radeon
	blacklist nouveau
	- /etc/qemu/bridge.conf:
	allow br0
	- /etc/sysctl.d/40-hugepage.conf:
	vm.nr_hugepages = 8200
	- /etc/fstab:
	hugetlbfs /dev/hugepages hugetlbfs mode=1770,gid=78 0 0



	Both VMs using same "backing files" that not modified in runtime:
	- /win81-base.qcow2 [13GB only!], compressed
	This QCOW2 image contain installed and configured Windows 8.1
	- /dev/disk/by-uuid/0AD949C94647B394
	It's NTFS partition on 1TB HDD with games and some software installed.
	Those files are base for QCOW2 images that used by virtual machines:
	qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-amd.qcow2
	qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-nvidia.qcow2
	qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-amd.qcow2
	qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-nvidia.qcow2
	So I basically store VM-specific changes to my system drive on SSD and changes for content stored on HDD RAID.



	Kernel build:
	apt-get install kernel-package
	apt-get build-dep linux-image-$(uname -r)
	apt-get build-dep linux-image-3.19.0-23-generic
	apt-get source linux-image-$(uname -r)
	apt-get source linux-image-3.19.0-23-generic
	cd linux-*
	# Now just moment to hate Ubuntu packaging, I do not use VBOX!
	rm -rf ubuntu/vbox
	# Comment out virtualbox crap
	nano ubuntu/Makefile
	# Apply patches (my patch files are above kernel directory)
	# Personally I just used patch version published by someone on google drive
	https://gist.github.com/ArseniyShestakov/768a426e79e2b901f9eb
	patch -p1 < ../override_for_missing_acs_capabilities.patch
	patch -p1 < ../i915_317.patch
	# Copy your current kernel config
	cp -vi /boot/config-`uname -r` .config
	# Change some options for better performance
	# Processor Type and Features > Preemption Model > Preemptible Kernel (Low Latency Desktop)
	# (CONFIG_PREEMPT=y)
	# Processor Type and Features > Timer Frequency > 1000 Hz
	# (CONFIG_HZ_1000=y CONFIG_HZ=1000)
	make menuconfig
	# Now build kernel
	fakeroot make-kpkg -j 8 --initrd kernel-image kernel-headers --append-to-version=-acs-igdarb-lowpremption-timer1000hz
	# Install built packages
	dpkg -i ../acs-igdarb-lowpremption-timer1000.deb


	All useful information may be found there:
	https://bbs.archlinux.org/viewtopic.php?id=162768
	https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=0
	http://vfio.blogspot.com



	My personal expirience of why I use some of options:
	https://www.reddit.com/r/linux_gaming/comments/3dr62f/set_up_gpu_passthrough_without_using_the_cli/ct8gr34
	https://www.reddit.com/r/linux_gaming/comments/2z2y7h/gpu_passthrough_or_how_to_play_any_game_at_near/cpfh76x
	# This system use real network card (00:19.0) as I have two in my mobo.


	#################################################
	# Exclusive configuration when no other VM running (more RAM, all cores)
	#################################################
	qemu-system-x86_64 \
	-name "win81_amd_exclusive" \
	\
	-bios /usr/share/seabios/bios.bin \
	-enable-kvm \
	\
	-vga none \
	-device qxl \
	\
	-usb \
	-device usb-mouse \
	\
	-m 10G \
	-mem-path /dev/hugepages \
	\
	-no-hpet \
	-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
	-smp 8,sockets=1,cores=4,threads=2 \
	\
	-soundhw hda \
	\
	-M q35 \
	\
	-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
	-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
	\
	-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
	-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
	\
	-net none \
	\
	-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
	-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0


	#################################################
	# Shared configuration when Nvidia VM is used too:
	#################################################
	taskset -c 0,2,4,6 qemu-system-x86_64 \
	-name "win81_amd" \
	\
	-bios /usr/share/seabios/bios.bin \
	-enable-kvm \
	\
	-vga none \
	-device qxl \
	\
	-usb \
	-device usb-mouse \
	\
	-m 8G \
	-mem-path /dev/hugepages \
	\
	-no-hpet \
	-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
	-smp 4,sockets=1,cores=2,threads=2 \
	\
	-soundhw hda \
	\
	-M q35 \
	\
	-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
	-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
	\
	-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
	-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
	\
	-net none \
	\
	-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
	-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0
	# Driver on guest is 335.23, it's last driver that no break KVM and Hyper-V
	# Anyway this doesn't make much sense because I only run D3D9 games on that card and those not sensetive to old drivers

	# Optional removal of existing data in Nvidia VM
	# This VM used for testing so I drop all changes on ever reboot
	rm /win81-writable-nvidia.qcow2
	rm /home/gameshdd-writable-nvidia.qcow2
	cp /win81-writable-nvidia.qcow2.backup /win81-writable-nvidia.qcow2
	cp /home/gameshdd-writable-nvidia.qcow2.backup /home/gameshdd-writable-nvidia.qcow2

	taskset -c 1,3,5,7 qemu-system-x86_64 \
	-name "win81_nvidia" \
	\
	-bios /usr/share/seabios/bios.bin \
	-enable-kvm \
	\
	-vga none \
	-device qxl \
	\
	-usb \
	-device usb-mouse \
	\
	-m 7G \
	-mem-path /dev/hugepages \
	\
	-no-hpet \
	-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
	-smp 4,sockets=1,cores=2,threads=2 \
	\
	-soundhw hda \
	\
	-M q35 \
	\
	-drive file=/win81-writable-nvidia.qcow2,id=disk1,format=qcow2,if=virtio \
	-drive file=/home/gameshdd-writable-nvidia.qcow2,format=qcow2,if=virtio \
	\
	-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
	-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
	\
	-net nic,model=virtio \
	-net bridge,br=br0 \
	\
	-serial stdio \
	-nographic

	# Nvidia reset. Boot to reset GPU between usages to avoid host freeze.
	# Also make it set fans to minimum so it's not so loud when VM not used.
	timeout 10 qemu-system-x86_64 \
	-enable-kvm \
	-M q35 \
	-m 512 \
	-cpu host \
	-rtc base=localtime \
	-smp 4,sockets=1,cores=4,threads=1 \
	-bios /usr/share/qemu/bios.bin \
	-boot menu=off,strict=on -vga none -nographic \
	-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
	-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on


	# Everything else only needed if i915 VGA arbitration not used
	# I don't really use it because
	##########################################
	# kvm=off required to bypass Nvidia driver virtualization limitations
	# Hyper-V hv_ options can't be used for same reason

	# The way I got consistent VM startup:
	- First boot up "win81_nvidia_reset" configuration with livecd that use Nouveau.
	- Once monitor activated shutdown livecd or just kill QEMU.
	Display will remain with frozen image on it (or shutdown black screen, but active).
	- Now boot "win81_nvidia" configuration.
	- In "Device Manager" disable and enable device.
	Now monitor must lost connection and go idle / switch to different active port.
	- Disable and enable device once again. It's should be working now.

	qemu-system-x86_64 \
	-boot menu=on \
	-bios /usr/share/seabios/bios.bin \
	-enable-kvm \
	\
	-vga none \
	\
	-m 1G \
	\
	-cpu host \
	-smp 1,sockets=1,cores=1,threads=1 \
	\
	-name "win81_nvidia_reset" \
	-M q35 \
	\
	-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
	-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,x-vga=on \
	\
	-drive file=/opt/kubuntu-15.04-desktop-amd64.iso,id=isocd \
	-device ide-cd,bus=ide.1,drive=isocd
	#!/bin/sh
	service lightdm stop

	modprobe vfio-pci
	echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id
	echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
	echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
	echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id

	echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id
	echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
	echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind
	echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id

	echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/new_id
	echo "0000:00:19.0" > /sys/bus/pci/devices/0000:00:19.0/driver/unbind
	echo "0000:00:19.0" > /sys/bus/pci/drivers/vfio-pci/bind
	echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/remove_id

	echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/new_id
	echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind
	echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
	echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/remove_id

	service lightdm star
	/etc/X11/xorg.conf.d/10-intel.conf:
	Section "Device"
	Identifier "Intel Graphics"
	Driver "intel"
	Option "DRI" "3"
	EndSection

	/etc/X11/xorg.conf.d/20-radeon.conf:
	Section "Device"
	Identifier "Radeon"
	Driver "radeon"
	Option "DRI3" "1"
	EndSection