GPU Passthrough on KVM: An Advanced VFIO Implementation Guide
A GPU passthrough KVM setup assigns a physical PCIe graphics device directly to a virtual machine through VFIO. The guest loads the native vendor driver and controls the GPU with far less virtualization overhead than an emulated display. The hard part is not adding a PCI address. It is proving that firmware, PCIe topology, kernel driver ownership, DMA isolation, reset behavior, and guest firmware all agree on the assignment.
This guide assumes you already administer Linux virtualization hosts and understand KVM, QEMU, libvirt, initramfs, and PCI addressing. The workflow targets a dedicated host with console or out-of-band access. Do not perform the first reboot remotely without a recovery path, especially when the selected GPU is also the host boot display.
The safest design uses a GPU that the host never needs, an isolated IOMMU group, a Q35 machine type, OVMF UEFI, and explicit assignment of every required PCI function. Before changing boot parameters, record the current kernel command line, VM XML, GPU driver, PCI addresses, and rollback commands. That baseline turns a hardware-sensitive change into a controlled operation.
Understand VFIO GPU Passthrough Isolation
VFIO exposes a device to QEMU while the IOMMU limits its DMA access to memory mapped for that VM. The assignment boundary is not always one PCI function. It is the IOMMU group created by the platform topology.
| Layer | What must be correct | Typical failure |
|---|---|---|
| Firmware | VT-d or AMD-Vi enabled | No IOMMU groups appear |
| PCIe topology | GPU functions safely isolated | Group is not viable |
| Host kernel | Device owned by vfio-pci | Native driver claims the GPU |
| VM platform | Q35 and UEFI configured | BAR mapping or boot failure |
| Guest | Correct vendor driver installed | Device cannot initialize |
A multifunction GPU may expose graphics, audio, USB, and USB Type-C controller functions. Inspect the complete slot. Passing only the graphics function while leaving a related function attached to the host can block KVM PCI passthrough or create an unsafe ownership split.
Prerequisites
Hardware and Firmware
Use a CPU and motherboard that support Intel VT-d or AMD-Vi. Enable the relevant virtualization and IOMMU options in firmware. Above 4G Decoding is recommended for modern GPUs with large PCI BARs. Resizable BAR can remain enabled, but disable it during diagnosis if resource mapping is inconsistent.
Keep integrated graphics, a second GPU, serial access, or a remote management controller available for recovery.
Preflight Checklist
Confirm the GPU is not required for host management.
Shut down the VM before changing PCI ownership.
Record every function associated with the target GPU.
Verify that no host-critical controller shares its group.
Back up the VM XML and bootloader configuration.
Check whether installed GPUs share identical vendor and device IDs.
Confirm that a failed boot will not lock you out.
The final check matters because an ids= rule can bind every matching card to vfio-pci. With identical GPUs, use a BDF-specific binding method or a tested initramfs hook instead of a broad device-ID rule.
Validate PCI Topology
Identify the GPU and Its Functions
List display and audio devices, then inspect the complete slot:
lspci -nn | grep -Ei 'vga|3d|display|audio'
lspci -nnk -s 01:00
Replace 01:00 with the actual bus and slot. Record each full address, such as 0000:01:00.0, its vendor and device ID, and the active kernel driver.
Audit IOMMU Groups
After enabling IOMMU in firmware, inspect the groups:
for d in /sys/kernel/iommu_groups/*/devices/*; do
printf '%s ' "${d#*/iommu_groups/*}:"
lspci -nns "${d##*/}"
done
Several functions belonging to the same GPU can share a group if all are assigned together. A group containing a storage controller, management NIC, or unrelated host-critical device is not safe to pass.
Moving the card to another slot can change the upstream bridge and improve isolation. PCIe ACS override patches may split groups in software without changing the physical routing guarantees. Use them only when reduced isolation is an explicit risk decision.
GPU Passthrough KVM Setup: Step-by-Step
1. Enable IOMMU at Boot
For a GRUB-based Intel host, append:
intel_iommu=on iommu=pt
For AMD, use:
amd_iommu=on iommu=pt
Regenerate the bootloader configuration and reboot:
sudo update-grub
sudo reboot
Verify the active command line and kernel initialization:
cat /proc/cmdline
dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi'
2. Load VFIO and Bind the Device
Load the required modules through /etc/modules-load.d/vfio.conf:
vfio
vfio_pci
vfio_iommu_type1
A typical vfio-pci binding rule in /etc/modprobe.d/vfio.conf is:
options vfio-pci ids=10de:AAAA,10de:BBBB
Replace the placeholders with the GPU and audio IDs, rebuild the initramfs, and reboot:
sudo update-initramfs -u
sudo reboot
Confirm that every assigned function reports vfio-pci as the active driver:
lspci -nnk -s 01:00.0
lspci -nnk -s 01:00.1
If the native driver still wins, inspect initramfs contents, framebuffer drivers, and module load order.
3. Configure the VM Platform
Use Q35 for a PCIe-capable chipset and OVMF for UEFI boot. Set CPU mode to host-passthrough when live migration compatibility is not required. Keep a virtual display during initial deployment so guest boot and driver installation do not depend on physical GPU output.
For deterministic workloads, use fixed memory rather than aggressive ballooning. Save a working copy of the domain XML before attaching host devices.
4. Add the libvirt hostdev Entries
Add one managed entry per required function:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
</hostdev>
managed='yes' lets libvirt detach and reattach the device around the VM lifecycle. It does not fix weak isolation or guarantee a reliable GPU reset.
Validate before starting the guest:
virsh dumpxml gpu-vm > gpu-vm.before-passthrough.xml
virsh edit gpu-vm
virsh domxml-to-native qemu-argv gpu-vm
5. Install Guest Drivers
Boot the guest and install the native NVIDIA or AMD driver. On Linux, verify with lspci -nnk, kernel logs, and the vendor management utility. On Windows, inspect Device Manager and the vendor driver service.
Do not hide the KVM signature by default. Add KVM hiding or a custom Hyper-V vendor ID only when a specific driver failure is reproducible. Legacy workarounds can complicate debugging without solving a current problem.
Security and Hardening
Protect the Host Boundary
The IOMMU group is the minimum safe ownership unit. Never split a group between the host and an untrusted guest merely because QEMU starts. DMA-capable hardware can affect host integrity when isolation is incomplete.
Avoid passing the host management GPU, boot storage controller, or primary network interface. Restrict access to libvirt management sockets and VFIO device nodes. A user who can redefine arbitrary hostdev entries may be able to claim sensitive PCI devices.
Keep the Configuration Auditable
Document assigned BDF addresses, group membership, firmware settings, and binding rules. Firmware updates can reorder PCI enumeration, so revalidate the GPU passthrough KVM setup after BIOS changes, hardware moves, and major kernel upgrades.
Patch the guest firmware, operating system, and GPU driver. Direct hardware access does not replace normal guest hardening, network segmentation, or workload isolation.
Performance and Reliability Tips
NUMA Tuning and Locality
A technically correct VFIO GPU passthrough can still perform poorly when vCPUs and memory come from a remote NUMA node. Check the GPU's NUMA association and keep latency-sensitive vCPUs and memory local where practical.
Pin CPUs only after measuring contention, and reserve host capacity for QEMU I/O threads. Huge pages can reduce page table overhead, but they add allocation constraints.
Use VirtIO for storage and networking unless those devices also need direct assignment. Avoid host swap, confirm the PCIe link negotiates the expected width and speed, and keep guest memory fixed for latency-sensitive workloads.
Test Device Reset Behavior
Some GPUs reset cleanly after guest shutdown; others remain unusable until the host reboots. Test repeated start, shutdown, and restart cycles before declaring the GPU passthrough KVM setup production-ready. One successful cold boot is not a reliability test.
GPU Passthrough Troubleshooting
IOMMU Group Is Not Viable
Another group member is still attached to a host driver. Inspect every member and either assign all safe functions, move the GPU to another slot, or stop. Do not pass unrelated critical devices merely to satisfy QEMU.
vfio-pci Does Not Claim the GPU
Check whether a native driver or early framebuffer binds first. Confirm that the VFIO rule exists inside the generated initramfs. With identical GPUs, verify that the rule targets the intended BDF instead of every matching device ID.
Guest Boots to a Black Screen
Keep a virtual display until the vendor driver is installed. Verify OVMF, Q35, GPU power, monitor input, and whether physical output activates only after driver initialization. Some compute cards provide no display output.
BAR Allocation Fails
Enable Above 4G Decoding, use Q35, and review guest firmware resource allocation. Test with Resizable BAR disabled if behavior is inconsistent. Large BAR errors are platform mapping failures, not usually guest driver installation problems.
VM Fails After the First Shutdown
Suspect reset behavior. Test a full guest shutdown and repeated lifecycle operations. If the card cannot return to a clean state, use a different GPU, keep the VM running between maintenance windows, or plan controlled host reboots.
Validation and Rollback
Validate the completed GPU passthrough KVM setup with these steps:
Start and stop the VM several times.
Reboot the guest without rebooting the host.
Run a representative compute, rendering, or encoding workload.
Confirm host networking, storage, and management remain stable.
Review logs for IOMMU faults, AER errors, and VFIO reset failures.
Verify the guest still starts after a host reboot.
For rollback, remove the hostdev entries while the VM is off, disable the VFIO binding rule, rebuild the initramfs, restore the original kernel command line if needed, and reboot. Confirm that the normal host driver reclaims the GPU.
Conclusion
A reliable GPU passthrough KVM setup depends on isolation discipline more than XML syntax. Prove the PCIe topology first, identify every function, and treat the IOMMU group as the ownership boundary. Bind only the intended device to vfio-pci, then build the guest around Q35, OVMF, native drivers, and a recoverable virtual display.
For advanced deployments, operational testing determines whether the design is ready. Validate NUMA locality, fixed memory behavior, repeated GPU resets, kernel upgrades, and firmware changes. Avoid broad device-ID rules with identical cards, and avoid ACS override unless reduced isolation is an accepted tradeoff. Persistent failures become easier to diagnose when separated into four layers: firmware initialization, host driver ownership, IOMMU isolation, and guest driver startup. Keep rollback artifacts for each layer so the configuration remains maintainable rather than becoming a one-boot experiment.
Comments
Post a Comment