vGPU experience and profiles

Community Support
2

I managed to figure out vGPU setup and profile creation. Here’s the information needed for a successful vGPU launch and profile selection.

First, you need to refer to the NVIDIA vGPU support guide by ON: NVIDIA vGPU support.

For Enabling Virtual Functions, I recommend creating a systemd service.

Create the service:

nano /etc/systemd/system/nvidia-sriov.service

Add the following:

[Unit]
Description=Enable NVIDIA SR-IOV
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/lib/nvidia/sriov-manage -e 00:41:0000.0
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

NOTE: 00:41:0000.0 may differ in your case. Check the ON guide carefully.

Save and exit /etc/systemd/system/nvidia-sriov.service, then run:

systemctl daemon-reload
systemctl enable nvidia-sriov
reboot

This ensures that SR-IOV and mdev remain active after a server reboot.

When creating a VM in ON or adding a PCI device, the UUID of the mdev device will be required. These need to be created in advance.

You can list all UUIDs using the following command:

/var/tmp/one/im/kvm-probes.d/host/system/pci.rb

Example output:

PCI = [
 TYPE = "10de:27b8:0302" ,
 VENDOR = "10de" ,
 VENDOR_NAME = "NVIDIA Corporation" ,
 DEVICE = "27b8" ,
 CLASS = "0302" ,
 CLASS_NAME = "3D controller" ,
 ADDRESS = "0000:01:04:3" ,
 SHORT_ADDRESS = "01:04.3" ,
 DOMAIN = "0000" ,
 BUS = "01" ,
 SLOT = "04" ,
 FUNCTION = "3" ,
 NUMA_NODE = "0" ,
 DEVICE_NAME = "NVIDIA Corporation AD104GL [L4]" ,
 UUID = "811ba00f-b95c-5d0d-a870-99b48162add9" ,
 PROFILES = "nvidia-908,nvidia-909,nvidia-910,nvidia-911,nvidia-912,nvidia-913,nvidia-914,nvidia-915,nvidia-916,nvidia-917,nvidia-918,nvidia-919,nvidia-920,nvidia-921,nvidia-922,nvidia-923,nvidia-924,nvidia-925"
]

Using this information, you need to create an mdev and make it auto-enabled so it persists after reboot:

echo 811ba00f-b95c-5d0d-a870-99b48162add9 > /sys/class/mdev_bus/0000:01:04:3/mdev_supported_types/nvidia-919/create

Then, you can create a VM and assign the vGPU to it using Sunstone with SHORT_ADDRESS = "01:04.3".

The VM should boot successfully with the chosen profile.

nvidia-919 is the selected profile.
You can check the profile details like this:

cat /sys/class/mdev_bus/0000:01:04.3/mdev_supported_types/nvidia-919/description

Example:

num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12

The only issue I haven’t solved yet is whether it’s possible to have multiple different profiles on a single GPU.
In my case, after creating one 2048M profile, I can create up to 11 more profiles of the same type, but other profiles get locked after creating the first one.

1 Like