VT-d GPU Passthrough

Posted by F1ak3r on May 9, 2016, 12:41 p.m.

What is the best single-PC setup for a Linux enthusiast who still likes to play Windows-only games and run other Windows-only software?

Not WINE, because despite working surprisingly well in many cases and being in general better than you might expect, it's often flaky and unstable, and is absolutely not guaranteed to run every program.

Not dual-booting, because then you have to constantly reboot and context-switch.

Not a Windows virtual machine, because those don't let you run anything graphically intensive.

Not a Linux virtual machine on a Windows host, because then you're not really using Linux (especially evident when Windows blue-screens and takes your poor VM along with it).

What you want is a way run Linux and Windows simultaneously, allowing each operating system access to the tools it needs to run your system: Windows should get the graphics card and everything else it needs for playing games, and Linux should get the rest for doing everything else.

Solution: set up a Linux host with a Windows virtual machine that uses modern virtualisation technology (VT-d & IOMMU) to directly access the graphics card.

I set this up over the last week or so, and it was actually a lot easier than I thought it would be. A year or two ago you needed to compile custom kernel images to get it right, but with the latest version of Ubuntu (16.04) running Linux kernel ~4.4, all I had to do was:

1. Make some minor config changes to assign my graphics card to the pci-stub driver1 on boot instead of the default radeon driver.

2. Set up a Windows virtual machine with Qemu-KVM and assign it my card (there was even a neat GUI for this, and only one point where you need to abandon it and dip into config files).

3. Install graphics card drivers on Windows VM.

I had to do a bit more futzing around to get keyboard and mouse sharing (Synergy), some futzing with that to get mouselook in FPS games to work (by default Synergy sends absolute mouse coordinates, but games want relative ones, so you end up with a madly spinning screen or a crosshair that simply refuses to move) and a little more mucking around to get my onboard graphics card to play nice so I could use it for the Linux host.

All-in-all, easier than getting graphics card drivers to work on Linux itself. /s

Under my setup, I have both monitors wired to both graphics cards. So I can start up my Windows VM, move my mouse up, switch output on the screens, and then feel exactly like I'm using Windows on a normal Windows PC. And if I don't want to actually use Windows, I can Steam's in-home streaming to literally play games on Linux.

Graphics settings in games are exactly the same as when I was running Windows, and performance seems the same too (though this may take a few weeks to fully assess).

Startup is obviously faster, and I can now spend most of my time using my favourite minimal tiling window manager.

Overall, this worked a lot better than I expected, and was almost entirely painless to set up. My thoughts could change in the coming weeks (during which I plan to do a full writeup of how I set things up on my other blog), but so far, so good. Really worth a shot if you think you'd like it and have compatible hardware (you need to be able to enable VT-d in your BIOS and will also require a reasonably recent graphics card). For reference, the three main resources I used were this Linux Mint forum thread, this Arch Wiki page and the five-part guide on the VFIO Tips and Tricks blog.

EDIT 2016/09/09: And here at long last is my comprehensive guide to setting this up, as promised months ago.

Comments

LAR Games 7 years, 11 months ago

I haven't really experimented much with Virtual Machines in a while. I'm surprised that the graphics card can actually be used in a VM! I remember looking into that a while ago and being pretty disappointed.

Mega 7 years, 11 months ago

This is a project I've been planning on attempting for a while, but keep pushing it back until I clear up a spare HDD to test it on before committing.

One of my concerns was with the Windows 10 License I have (Upgrade from Win 8 Pro). The license is supposed to be tied to the hardware fingerprint. So in the case of running it in a VM, will it report the same?

I'll keep an eye on your other blog for the full writeup, that might be the decider for me. I'm currently using a VM as well as my laptop (with an SSH server), but I'd love to have Linux as my main OS on my primary machine.

Jani_Nykanen 7 years, 11 months ago

Interesting, I might try that in (distant) future.

Before that… well, dual-booting isn't that bad!

F1ak3r 7 years, 11 months ago

Quote: Mega
This is a project I've been planning on attempting for a while, but keep pushing it back until I clear up a spare HDD to test it on before committing.
This was basically why I did it. Through various shenanigans, my PC has two SSDs: one small and one big. I cleared everything off the small one and used it to experiment with the setup non-committally before wiping my Windows installation from the big one.

Quote: Mega
One of my concerns was with the Windows 10 License I have (Upgrade from Win 8 Pro). The license is supposed to be tied to the hardware fingerprint. So in the case of running it in a VM, will it report the same?
Sadly not. Microsoft picked it up as two separate devices.

I've heard of people extracting the keys from their installations, though, but this sort of thing is the main reason I bit the bullet and bought Windows 10 rather than upgrading.

flashback 7 years, 11 months ago

I hear you can also use this trick in QubesOS if that's your cup of tea.

aeron 7 years, 11 months ago

Awesome write-up. I will probably refer back to this when I re-image my desktop. Right now my desktop has Win7 and a headless Arch VM (which I only access via PuTTY). I'd like to model my next image after my laptop which is Ubuntu with Win10 in a VM. The laptop has integrated graphics so I wasn't concerned with getting accelerated games to run in VirtualBox, but your experience with PCI pass-through in KVM is promising.

So you mention multiple graphics cards, is that required to get this pass-through setup working?

Also woot woot for i3! Praise be the window manager of the Gods of Linux.

flashback 7 years, 11 months ago

IIRC, you must have a separate GPU dedicated to just the VM for this to work.

F1ak3r 7 years, 11 months ago

Quote: flashback
I hear you can also use this trick in QubesOS if that's your cup of tea.
QubesOS is certainly on my radar. I know a few guys who are experimenting with it, and from what I've seen it works really nicely, bar a few Windows issues. Definitely something to try out at some stage.

Quote: aeron
So you mention multiple graphics cards, is that required to get this pass-through setup working?
Quote: flashback

IIRC, you must have a separate GPU dedicated to just the VM for this to work.
Yes. Your host must boot up without using the graphics card you intend to pass through, so you need another one to provide your display. Luckily this isn't too esoteric a setup – I used my onboard Intel graphics for this, though I've heard of people using a second Nvidia/AMD card.

mrpete 7 years, 11 months ago

I like the pass-through concept you've chosen to do(aside from having to have two cards and some switching). VirtualBox offers it but only with certain PCI configurations… My mini-ITX setup did not meet those requirements. I currently run an OLD XP(for compatibility scaling) image through my PuppyLinux setup. Surprisingly, most things work well with the plain vanilla setup as long as whatever I run doesn't dive too deep into DirectX. OpenGL is also another story as well(very bare). I like it because it works with my minimalist setup which suits me fine these days. Most things that I download from 64digits have worked flawlessly through the virtual machine. Most unity games work as well.

Anyways, good write-up.

mrpete 7 years, 11 months ago

I like the pass-through concept you've chosen to do(aside from having to have two cards and some switching). VirtualBox offers it but only with certain PCI configurations… My mini-ITX setup did not meet those requirements. I currently run an OLD XP(for compatibility scaling) image through my PuppyLinux setup. Surprisingly, most things work well with the plain vanilla setup as long as whatever I run doesn't dive to deep into DirectX. OpenGL is also another story as wel(very bare)l. I like it because it works with my minimalist setup which suits me fine these days. Most things that I download from 64digits have worked flawlessly through the virtual machine. Most unity games work as well.

Anyways, good write-up.