GPU Passthrough on KVM: A Production-Focused VFIO Guide
A GPU passthrough KVM setup assigns a physical PCIe graphics device directly to one virtual machine through VFIO. The guest uses the vendor driver and accesses the card with much less virtualization overhead than an emulated or paravirtualized display. This is useful for CUDA workloads, rendering, remote workstations, machine learning, and applications that require native driver capabilities.
The difficult part is not adding a PCI address to a VM definition. Reliable passthrough depends on firmware settings, IOMMU topology, early driver binding, guest firmware, reset behavior, and a rollback path. A configuration can boot successfully and still be operationally weak if the host occasionally claims the GPU, if another device shares its isolation group, or if the VM cannot recover after a guest reboot.
This guide targets Debian and Ubuntu hosts running KVM, QEMU, and libvirt. The same principles apply to other distributions, but bootloader, initramfs, and package commands may differ. The procedure assumes the GPU is dedicated to one guest. It does not cover mediated devices, SR-IOV GPU partitioning, or vendor-specific multi-instance technologies. Before changing the host, keep an out-of-band console available and verify that another GPU or integrated display can handle host access.
Architecture and Assignment Boundaries
VFIO exposes a physical device to userspace while relying on the IOMMU to restrict direct memory access. QEMU becomes the userspace device owner, and the guest loads the normal NVIDIA or AMD driver.
The assignment boundary is the IOMMU group, not merely the GPU function. A card may expose display, HDMI audio, USB, or controller functions. Evaluate the entire group because an unsafe sibling device can weaken isolation or block assignment.
| Component | Purpose | Failure symptom |
|---|---|---|
| IOMMU | Translates and isolates device DMA | Empty groups or unsafe assignment |
| vfio-pci | Owns the GPU on the host | Vendor driver claims the card |
| QEMU and libvirt | Attach PCI functions to the VM | VM start failure |
| OVMF | Supplies UEFI firmware to the guest | Black screen or firmware mismatch |
| Q35 | Provides a modern PCIe-oriented machine model | PCIe topology limitations |
| Guest driver | Operates the passed-through GPU | Driver failure or no compute device |
Prerequisites and Change Planning
A stable GPU passthrough KVM setup starts with hardware checks, not kernel parameters.
Use this checklist before modifying the host:
CPU and motherboard support Intel VT-d or AMD-Vi.
Firmware exposes IOMMU and hardware virtualization controls.
The GPU is visible with all related PCI functions.
A second display adapter, serial console, IPMI, or equivalent recovery path exists.
The guest can use UEFI firmware.
The selected GPU is not required for the host console.
Current VM definitions and bootloader configuration are backed up.
The maintenance window includes at least one host reboot.
Inspect the Platform
Confirm virtualization and enumerate display-related devices:
lscpu | grep -E 'Virtualization|Model name'
lspci -nnk | grep -A3 -Ei 'VGA|3D|Display|Audio'
Record each PCI address and vendor-device ID. Do not assume the audio function is the only companion function.
Enable Firmware Options
Enable VT-d on Intel systems or IOMMU on AMD systems. Hardware virtualization must also be enabled. Above 4G Decoding is often required for large PCI BARs. Treat Resizable BAR as a compatibility variable and keep it enabled only when the platform is stable.
Validate IOMMU Topology
Before binding anything, confirm that the kernel creates useful isolation groups.
Enable IOMMU at Boot
On a GRUB-based Intel host, add the following kernel parameters:
intel_iommu=on iommu=pt
For AMD:
amd_iommu=on iommu=pt
Update GRUB and the initramfs using the commands appropriate for the distribution, then reboot. After startup, verify initialization:
dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi'
find /sys/kernel/iommu_groups/ -type l
The iommu=pt option can reduce translation overhead for non-assigned devices, but it does not replace isolation.
Review Every Device in the GPU Group
Use a compact group report:
for d in /sys/kernel/iommu_groups/*/devices/*; do
g=${d#*/iommu_groups/}; g=${g%%/*}
printf 'Group %s: ' "$g"
lspci -nn -s "${d##*/}"
done
A GPU grouped with its own audio function is usually expected. A GPU grouped with a storage controller, management NIC, or unrelated endpoint is a deployment blocker. First try a different slot or firmware setting. ACS override patches can split groups logically, but they do not change the physical isolation characteristics of the motherboard, so they should not be treated as a production security fix.
Step-by-Step GPU Passthrough KVM Setup
The sequence below keeps the host changes narrow and verifies ownership before the VM starts.
1. Load VFIO Modules
Create a module list that loads VFIO early:
vfio
vfio_pci
vfio_iommu_type1
Verify which VFIO modules exist on the running kernel rather than forcing obsolete names.
2. Bind the Complete Device Set to vfio-pci
Identify all functions belonging to the card:
lspci -nn -s 41:00
Then configure their vendor-device IDs:
options vfio-pci ids=10de:2235,10de:1aef
Rebuild the initramfs and reboot. Avoid broad driver blacklists on multi-GPU hosts. ID-based binding is precise, although identical cards may require PCI address-based overrides.
Confirm ownership:
lspci -nnk -s 41:00.0
lspci -nnk -s 41:00.1
Each assigned function should report Kernel driver in use: vfio-pci.
3. Prepare the Guest Definition
Use OVMF with a Q35 machine type. CPU mode host-passthrough is appropriate when migration compatibility is not required. Add the GPU and every required companion function as managed PCI host devices:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/>
</source>
</hostdev>
Create another hostdev entry for each required companion function. With managed='yes', libvirt handles detachment and reattachment. Never assign one physical function to multiple active guests.
4. Boot With a Fallback Display
Keep a virtual video adapter and console during the first boot. Install the guest GPU driver, confirm the device is healthy, and only then decide whether the passed-through GPU should become the primary display. Removing it too early turns driver problems into blind troubleshooting.
5. Verify Inside the Guest
For Linux, inspect PCI ownership and driver status:
lspci -nnk | grep -A3 -Ei 'VGA|3D|Display'
nvidia-smi
For AMD compute workloads, verify that the expected kernel and userspace stack detect the card. For Windows, check Device Manager, driver events, and the vendor utility. A visible device is not enough. Run a workload that allocates GPU memory, then test shutdown and restart.
Security and Hardening
Direct assignment expands the guest's relationship with host hardware. The IOMMU limits DMA, but the management plane still requires protection.
Preserve Isolation
Do not pass through devices from a group that also contains a host-critical controller. Avoid ACS override in environments where guest trust boundaries matter. Keep firmware current, but stage updates because PCIe topology and reset behavior can change between BIOS releases.
Restrict VM Administration
Only trusted operators should be able to modify libvirt domain XML, attach host devices, or access QEMU monitor interfaces. Device assignment is privileged infrastructure control. Audit changes to VM definitions and keep a known-good copy outside the host.
Secure Boot requires planning on both sides. The host must load VFIO components, and the guest must trust its driver modules. Signed modules are preferable to permanently disabling verification.
Performance and Reliability Tuning
This GPU passthrough KVM setup removes much of the virtual graphics path, but NUMA locality, memory policy, CPU placement, and storage still shape performance.
Place guest vCPUs and memory near the NUMA node connected to the GPU when possible. Check the device locality under sysfs and compare it with numactl --hardware. Cross-socket memory traffic can reduce consistency in data-intensive workloads.
Use huge pages only after measurement. CPU pinning can improve consistency, but poor placement may overload sibling threads or starve QEMU emulator threads.
Test cold host boot, guest reboot, shutdown and restart, driver initialization, and workload recovery. If the GPU remains unusable until a host reboot, investigate reset support, firmware, logs, and missing companion functions.
Troubleshooting Common Failures
The VM Fails With a Device-Busy Error
The host driver still owns one function, or another VM has the device attached. Check every PCI function with lspci -nnk, stop conflicting guests, and confirm that libvirt uses managed host devices.
The Guest Boots to a Black Screen
Keep the virtual display enabled. Confirm OVMF, Q35, GPU firmware compatibility, and whether the output requires a monitor or dummy plug. A black display does not prove VFIO failed.
The Driver Loads but Compute Is Unavailable
Check guest logs, driver versions, permissions, and the compute stack. Test the vendor utility before debugging CUDA, ROCm, or an application framework.
The GPU Disappears After a VM Restart
This usually indicates a reset problem. Review shutdown logs, test a power cycle, and confirm whether a vendor-reset mechanism supports the exact device. Avoid automated rebind loops until recovery is understood.
IOMMU Groups Are Missing
Recheck firmware settings, boot parameters, and the active kernel command line:
cat /proc/cmdline
dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi'
If firmware does not expose remapping support, software configuration cannot create secure PCI passthrough.
Operational Rollback
Plan rollback before deployment. Stop the guest, remove its hostdev entries, delete the vfio-pci ID configuration, restore the original initramfs and boot parameters, reboot, and confirm the vendor driver owns the GPU.
If the host uses the card for local display, keep a separate console available during rollback. Do not attempt to rebind a production display GPU remotely without a tested recovery path. Preserve logs from both successful and failed boots because early driver binding problems may not remain visible after later service restarts.
Conclusion
A production-grade GPU passthrough KVM setup is an isolation and lifecycle problem as much as a virtualization task. The essential path is straightforward: enable the IOMMU, validate groups, bind every required PCI function to vfio-pci, attach the devices through managed libvirt hostdev entries, and verify the guest with a real workload.
The quality of the result depends on what happens outside the happy path. Keep a fallback display, avoid unsafe group splitting, test reset behavior, document rollback, and account for NUMA locality. These controls matter more than adding extra flags copied from unrelated hardware.
Once the configuration survives host reboots, repeated guest restarts, driver initialization, and sustained workload allocation, it is ready for controlled production use. Until then, treat a successful first boot as an installation milestone, not proof of reliability.
Comments
Post a Comment