GPU Passthrough on KVM with VFIO: Advanced Setup Guide

GPU passthrough on KVM gives a virtual machine direct ownership of a physical PCIe graphics device instead of presenting an emulated adapter. The guest loads the native vendor driver, controls the GPU memory, and submits work with near-native device access. This model suits CUDA or ROCm workloads, rendering, workstation virtualization, and applications that require a real display controller.

The hard part is not adding a PCI address to a VM definition. A reliable deployment depends on firmware configuration, interrupt remapping, correct IOMMU grouping, early driver binding, guest firmware, PCIe topology, and predictable device reset behavior. A GPU may also expose separate audio, USB, or serial functions that belong to the same physical card. Passing only the display function can produce a VM that starts once, fails after shutdown, or leaves part of the device attached to the host.

This guide assumes a Linux host running KVM, QEMU, libvirt, VFIO, and an OVMF-based guest. Commands use Debian or Ubuntu conventions where required, but the validation method applies to other distributions. Perform the work through SSH, a BMC console, or a second host GPU. Binding the active console GPU to VFIO can immediately remove local display access.

Prerequisites and Topology Planning

Hardware and Firmware

The platform must support CPU virtualization and IOMMU. Intel firmware normally exposes VT-d, while AMD systems expose AMD-Vi or IOMMU. Enable Above 4G Decoding when available, especially for GPUs with large BARs. If Resizable BAR causes allocation failures, disable it during troubleshooting.

A second display adapter or BMC is strongly recommended. The passed device becomes unavailable to the host while the guest owns it.

Recommended VM Baseline

Use current kernel, QEMU, libvirt, and OVMF packages. Configure the guest with Q35 and UEFI firmware.

Component	Recommended choice	Reason
Host console	BMC or separate GPU	Preserves recovery access
Guest firmware	OVMF UEFI	Better modern GPU compatibility
Machine type	Q35	Native PCIe topology
CPU model	host-passthrough	Exposes host CPU features
Disk and network	VirtIO	Reduces emulation overhead
PCI management	libvirt managed hostdev	Automates detach and reattach

Understand IOMMU Isolation

The Group Is the Assignment Boundary

VFIO relies on the IOMMU group created by the platform. Devices in one group may share routing or reset relationships, so the entire group must be considered the minimum secure assignment unit.

List the relevant functions:

lspci -nn | grep -Ei 'vga|3d|display|audio|usb'

Inspect the groups:

for g in /sys/kernel/iommu_groups/*; do
  echo "Group ${g##*/}"
  for d in "$g"/devices/*; do lspci -nn -s "${d##*/}"; done
done

A GPU often includes HDMI audio. Some cards also expose USB or USB-C controllers. Pass every required function that belongs to the card and shares its reset domain.

Avoid False Isolation

Moving the GPU to another slot can change the upstream PCIe path and improve grouping. Firmware updates may also help. An ACS override patch can split devices in software without creating equivalent hardware isolation. It may be acceptable in a lab, but it is unsuitable as a security boundary for untrusted tenants.

Step-by-Step GPU Passthrough on KVM

1. Enable and Verify IOMMU

Add the correct boot parameter: intel_iommu=on for Intel or amd_iommu=on for AMD. iommu=pt can reduce overhead for devices that are not assigned to guests.

After updating the bootloader and rebooting, verify initialization:

dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi|interrupt remapping'

Do not continue until IOMMU and interrupt remapping appear active.

2. Record PCI Addresses and Drivers

Use full domain addresses:

lspci -Dnnk -s 0000:65:00.0
lspci -Dnnk -s 0000:65:00.1

Record the vendor and device IDs, current driver, IOMMU group, and physical slot. This matters when several identical GPUs are installed.

3. Bind the Device to vfio-pci

Load the required modules early:

printf '%s\n' vfio vfio_pci vfio_iommu_type1 \
  | sudo tee /etc/modules-load.d/vfio.conf

For unique device IDs:

echo 'options vfio-pci ids=10de:2330,10de:1aef' \
  | sudo tee /etc/modprobe.d/vfio-pci.conf

Replace the example IDs. If two identical GPUs use the same IDs and only one should be assigned, ID-based binding is too broad. Use per-device driver_override through an initramfs hook or an early systemd unit.

Rebuild the initramfs, reboot, and confirm that every target function reports Kernel driver in use: vfio-pci.

4. Prepare the Guest

Create the VM with OVMF, Q35, and VirtIO disk and network devices. Keep a virtual display during operating system installation and vendor driver setup.

Use host-passthrough when maximum CPU feature exposure matters more than migration portability. It tightly couples the VM to the host CPU, microcode, and QEMU environment.

5. Add the PCI Functions

Edit the domain with virsh edit VM_NAME and add managed host devices:

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x65' slot='0x00' function='0x0'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x65' slot='0x00' function='0x1'/>
  </source>
</hostdev>

Managed mode lets libvirt detach the device before startup and reattach it after shutdown. A physical function cannot be used by the host or two active guests simultaneously.

6. Start and Inspect Logs

Start the domain, then inspect kernel and libvirt messages:

virsh start VM_NAME
journalctl -k -b | grep -Ei 'vfio|iommu|dmar'

Resolve group viability, permission, or BAR mapping errors before installing guest drivers.

Guest Driver Installation and Validation

Linux Guests

Install the vendor driver supported by the guest release. For NVIDIA, validate both PCI detection and driver operation:

lspci -nnk | grep -A3 -Ei 'vga|3d'
nvidia-smi

For AMD compute workloads, use the supported ROCm stack for the selected GPU and distribution. Successful PCI enumeration is not enough. Run a small memory allocation and compute test.

Windows Guests

Install the official vendor package after Windows detects the physical adapter. Keep a secondary virtual display until remote access works. Hiding KVM or changing the Hyper-V vendor ID should be treated as a compatibility fallback, not a default setting.

Security and Hardening

Protect the DMA Boundary

GPU passthrough on KVM gives the guest control of a DMA-capable device. The IOMMU restricts its memory access, but security still depends on correct grouping, interrupt remapping, firmware behavior, and a trustworthy device.

Avoid these shortcuts:

• Do not grant broad access to all /dev/vfio/* nodes.

• Do not run QEMU as root to bypass permissions.

• Do not pass unrelated controllers from the same group without review.

• Do not use ACS override for untrusted workloads.

• Do not disable SELinux or AppArmor globally.

Keep libvirt confinement enabled and restrict who can edit domain XML. Arbitrary hostdev access can expose sensitive PCI hardware.

Preserve Rollback Access

Keep SSH, serial, or BMC access available. Save the previous bootloader, initramfs, and modprobe configuration. The rollback path should remove VFIO binding, restore the vendor driver, rebuild the initramfs, and reboot.

Performance and Reliability Tuning

Align NUMA Locality

Find the GPU NUMA node:

cat /sys/bus/pci/devices/0000:65:00.0/numa_node

Pin vCPUs and allocate guest memory from the same NUMA node when possible. Cross-socket memory traffic can reduce throughput for data-heavy GPU workloads. Avoid aggressive CPU and memory overcommit.

Huge pages may reduce translation overhead, but they do not replace correct NUMA placement. Measure them with the real workload.

Test Reset Behavior

Test cold boot, guest reboot, shutdown, and a second start without rebooting the host. Some GPUs do not expose a reliable function-level reset and remain unusable until the host restarts.

Physical assignment also limits mobility. Disk snapshots do not capture GPU state. Standard live migration is generally unavailable unless the device, VFIO migration support, QEMU, libvirt, and destination platform are all compatible.

Troubleshooting Common Failures

The Group Is Not Viable

Inspect every function in the IOMMU group. All active devices must be detached from host drivers before VFIO can use the group. Do not ignore an audio, USB, storage, or network function sharing the boundary.

The Vendor Driver Still Owns the GPU

If lspci -nnk shows nouveau, nvidia, or amdgpu, the vendor driver loaded before vfio-pci. Check module order, initramfs contents, and early framebuffer drivers. Blacklisting alone may not be sufficient for the boot display GPU.

The Guest Has a Black Screen

Confirm OVMF, Q35, ROM BAR behavior, cable or dummy-plug requirements, and guest primary-display settings. Keep a virtual console until the physical adapter works inside the guest.

The First Start Works but the Second Fails

Inspect kernel logs for reset errors. A device that cannot return to a clean state may require a host reboot. Treat this as a hardware lifecycle limitation, not merely a guest driver problem.

Performance Is Unexpectedly Low

Check PCIe link width and speed with lspci -vv, guest power settings, CPU contention, NUMA placement, storage throughput, and memory pinning limits. A correctly assigned GPU can remain underused when data loading or remote memory is the bottleneck.

Production Checklist

Before declaring the deployment ready, confirm:

• IOMMU and interrupt remapping remain active after kernel updates.

• Every required GPU function uses vfio-pci.

• The IOMMU group contains no unintended active device.

• The VM uses OVMF and Q35.

• Libvirt manages detach and reattach.

• A second host console or BMC is reachable.

• Guest drivers survive reboot and updates.

• Cold start, warm restart, shutdown, and second start succeed.

• CPU and memory locality match the GPU NUMA node.

• Monitoring covers host VFIO errors and guest driver health.

• Recovery procedures do not assume GPU state is captured.

• Maintenance plans account for limited migration options.

Conclusion

A reliable GPU passthrough on KVM deployment is primarily an isolation and lifecycle problem. The VM definition is straightforward, but production stability depends on what happens before startup and after shutdown. Validate the IOMMU boundary, bind all related PCI functions early, preserve independent host access, and test repeated start and reset cycles.

For compute-heavy guests, pair the GPU with local NUMA memory, sufficient CPU capacity, and storage fast enough to keep the accelerator supplied. Do not weaken isolation merely to make an inconvenient PCIe topology appear clean. When secure grouping is unavailable, change the slot, platform, or assignment model.

Document the PCI addresses, firmware settings, kernel parameters, driver ownership, and rollback procedure for every host. This turns upgrades and incident recovery into controlled operations. A passthrough VM should be managed as a hardware-attached service with stricter maintenance constraints than an ordinary portable virtual machine.

Sources

• https://www.educba.com/scalable-server-infrastructure/