Proxmox GPU Passthrough
1.Let's Check PCI Devices
Proxmox We can log in via "SHELL" from the Web Console or via SSH from our computer with the help of a terminal emulator. Later ; We write the lspci -v command and, as you can see below, we display the information of our NVIDIA graphics card. # NOTE: Initially, it might not show the kernel driver vfio-pci, please check the kernel driver. 01:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ZOTAC International (MCO) Ltd. TU116 [GeForce GTX 1650 SUPER] Flags: bus master, fast devsel, latency 0, IRQ 11, IOMMU group 1 Memory at de000000 (32-bit, non-prefetchable) [size=16M] Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at d0000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [258] L1 PM Substates Capabilities: [128] Power Budgeting Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 Capabilities: [900] Secondary PCI Express Capabilities: [bb0] Physical Resizable BAR Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau 01:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1) Subsystem: ZOTAC International (MCO) Ltd. TU116 High Definition Audio Controller Flags: bus master, fast devsel, latency 0, IRQ 10, IOMMU group 1 Memory at df080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel
2. Let's Check Detailed NVIDIA Device Information
lspci | grep -i nvidia Using the command above, we wanted it to bring only information about our NVIDIA product. 01:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1) 01:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1) 01:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1) 01:00.3 Serial bus controller: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1) Then, to learn the ID information of these devices; lspci -nn | grep -i nvidia 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:2187] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1) 01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1) 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)
3.GRUB Configuration Update
We need to add the necessary parameters to the GRUB configuration of our Proxmox server. ; vim /etc/default/grub --> We open the Grub file with the vi editor. You can also use the "nano" editor instead of the "vim" editor. ... GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init" # Alternative configuration # GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream initcall_blacklist=sysfb_init nofb video=vesafb:off,simplefb:off,efifb:off rd.driver.blacklist=snd_hda_intel,nvidia,nvidiafb,nouveau module.blacklist=snd_hda_intel,nvidia ,nvidiafb ,nouveau vfio-pci.ids=10de:2187,10de:1aeb" ... After making the above updates, we save and exit. Afterwards; update-grub --> With this command, we notify the system of the changes we have made in the GRUB configuration file.
4.Loading VFIO Modules
We are adding vfio modules to be loaded during boot. vim /etc/modules --> we open the configuration file here with the editor. vfio vfio_iommu_type1 vfio_pci vfio_virqfd vfio_nvidia We write the above module updates, save and exit.
5.Blacklist Module Configuration
We blacklist the default NVIDIA drivers to ensure that vfio-pci is used. vim /etc/modprobe.d/blacklist.conf --> we open the configuration file here with the editor. blacklist nouveau blacklist nvidia blacklist nvidiafb blacklist snd_hda_intel We write the above module updates, save and exit.
6. VFIO Configuration
We add the GPU and audio device IDs to the vfio configuration. vim /etc/modprobe.d/vfio.conf --> we open the configuration file here with the editor. options vfio-pci ids=10de:2187,10de:1aeb disable_vga=1 disable_idle_d3=1 initcall_blacklist=sysfb_init We write the above module updates, save and exit.
7. Initramfs and Reboot Process
We update the initramfs and restart the Proxmox server to apply the changes. update-initramfs -u -k all reboot
8. Checking VFIO Driver
lsmod | We use the grep vfio --> command. vfio_pci 16384 0 vfio_pci_core 86016 1 vfio_pci irqbypass 12288 2 vfio_pci_core,kvm vfio_iommu_type1 49152 0 vfio 69632 3 vfio_pci_core,vfio_iommu_type1,vfio_pci iommufd 98304 1 vfio # Run lspci -v again to confirm both video and audio components are bound to vfio-pci lspci -v
9. IOMMU Groups Check
We create a script to verify IOMMU groups. vim iommu_group.sh --> with this command, we create a file using the vim editor and open the editor. We add the following codes. #!/bin/bash shopt -s nullglob for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do echo "IOMMU Group ${g##*/}:" for d in $g/devices/*; do echo -e "\t$(lspci -nns ${d##*/})" done; done; chmod +x iommu_group.sh ./iommu_group.sh ... 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:2187] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1) 01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1) 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1) ...
10. Creating a Windows VM
We are using Windows 11 Pro. Make sure to mount the VirtIO driver ISO to add the missing drivers . Everything else during installation will proceed as usual. To add GPU PASSTHROUGH, add a PCI device and select your GPU with All Functions, ROM-Bar and PCI-Express options checked.
11. Final WE LAUNCH WINDOWS!
We install NVIDIA GeForce drivers.
