# vGPU experience and profiles

**URL:** <https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332>\
**Category:** Product Support\
**Created:** [December 4, 2024, 1:57pm UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332 "2024-12-04T13:57:07Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mayhem](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@Mayhem](https://forum.opennebula.io/u/Mayhem)\
**Post date:** [December 4, 2024, 1:57pm UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/1 "2024-12-04T13:57:07Z")

</div>

Hello everyone!

Has anyone had experience working with vGPU in OpenNebula?  
This is an entirely new experience for me, and so far, I’ve only learned how to add a GPU to a VM created in Nebula using passthrough.

[https://docs.opennebula.io/6.10/open\_cluster\_deployment/kvm\_node/pci\_passthrough.html](https://docs.opennebula.io/6.10/open_cluster_deployment/kvm_node/pci_passthrough.html)

Everything works fine, but even when I follow the instructions here: [NVIDIA vGPU support — OpenNebula 6.10.1 documentation](https://docs.opennebula.io/6.10/open_cluster_deployment/kvm_node/vgpu.html), nothing new appears. Only a single GPU is displayed everywhere. I don’t understand what to do or where to get the profiles, how to add and display them in OpenNebula.

For testing, I have an NVIDIA L4 graphics card and have installed the NVIDIA Host Drivers.

Could anyone share their experience on how to create, display, and use vGPU profiles in OpenNebula?

---

<div class="post-metadata">

**Author:** ![Mayhem](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@Mayhem](https://forum.opennebula.io/u/Mayhem)\
**Post date:** [December 6, 2024, 7:21am UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/2 "2024-12-06T07:21:28Z")

</div>

I managed to figure out vGPU setup and profile creation. Here’s the information needed for a successful vGPU launch and profile selection.

First, you need to refer to the NVIDIA vGPU support guide by ON: [NVIDIA vGPU support](https://docs.opennebula.io/6.10/open_cluster_deployment/kvm_node/vgpu.html).

For **Enabling Virtual Functions** , I recommend creating a systemd service.

Create the service:

```auto
nano /etc/systemd/system/nvidia-sriov.service

```

Add the following:

```auto
[Unit]
Description=Enable NVIDIA SR-IOV
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/lib/nvidia/sriov-manage -e 00:41:0000.0
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

```

**NOTE** : `00:41:0000.0` may differ in your case. Check the ON guide carefully.

Save and exit `/etc/systemd/system/nvidia-sriov.service`, then run:

```auto
systemctl daemon-reload
systemctl enable nvidia-sriov
reboot

```

This ensures that SR-IOV and mdev remain active after a server reboot.

When creating a VM in ON or adding a PCI device, the UUID of the mdev device will be required. These need to be created **in advance**.

You can list all UUIDs using the following command:

```auto
/var/tmp/one/im/kvm-probes.d/host/system/pci.rb

```

**Example output** :

```auto
PCI = [
 TYPE = "10de:27b8:0302" ,
 VENDOR = "10de" ,
 VENDOR_NAME = "NVIDIA Corporation" ,
 DEVICE = "27b8" ,
 CLASS = "0302" ,
 CLASS_NAME = "3D controller" ,
 ADDRESS = "0000:01:04:3" ,
 SHORT_ADDRESS = "01:04.3" ,
 DOMAIN = "0000" ,
 BUS = "01" ,
 SLOT = "04" ,
 FUNCTION = "3" ,
 NUMA_NODE = "0" ,
 DEVICE_NAME = "NVIDIA Corporation AD104GL [L4]" ,
 UUID = "811ba00f-b95c-5d0d-a870-99b48162add9" ,
 PROFILES = "nvidia-908,nvidia-909,nvidia-910,nvidia-911,nvidia-912,nvidia-913,nvidia-914,nvidia-915,nvidia-916,nvidia-917,nvidia-918,nvidia-919,nvidia-920,nvidia-921,nvidia-922,nvidia-923,nvidia-924,nvidia-925"
]

```

Using this information, you need to create an mdev and make it auto-enabled so it persists after reboot:

```auto
echo 811ba00f-b95c-5d0d-a870-99b48162add9 > /sys/class/mdev_bus/0000:01:04:3/mdev_supported_types/nvidia-919/create

```

Then, you can create a VM and assign the vGPU to it using Sunstone with `SHORT_ADDRESS = "01:04.3"`.

The VM should boot successfully with the chosen profile.

**nvidia-919** is the selected profile.  
You can check the profile details like this:

```auto
cat /sys/class/mdev_bus/0000:01:04.3/mdev_supported_types/nvidia-919/description

```

Example:

```auto
num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12

```

The only issue I haven’t solved yet is whether it’s possible to have multiple different profiles on a single GPU.  
In my case, after creating one 2048M profile, I can create up to 11 more profiles of the same type, but other profiles get locked after creating the first one.

---

<div class="post-metadata">

**Author:** ![Mayhem](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@Mayhem](https://forum.opennebula.io/u/Mayhem)\
**Post date:** [December 6, 2024, 10:16am UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/3 "2024-12-06T10:16:34Z")

</div>

I also want to add to the post that you can view the list of mdev devices using the command:

```auto
mdevctl list

```

Example output:

```auto
811ba00f-b95c-5d0d-a870-99b48162add9 0000:01:00.6 nvidia-919 manual

```

To avoid manual recreating the mdev device after a reboot, execute the following commands:

```auto
mdevctl define --uuid 811ba00f-b95c-5d0d-a870-99b48162add9
mdevctl set-auto --uuid 811ba00f-b95c-5d0d-a870-99b48162add9

```

---

<div class="post-metadata">

**Author:** ![amosteiro](https://avatars.discourse-cdn.com/v4/letter/a/439d5e/32.png) [@amosteiro](https://forum.opennebula.io/u/amosteiro)\
**Post date:** [December 9, 2024, 5:45pm UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/4 "2024-12-09T17:45:59Z")

</div>

Hi, @Mayhem.

It is possible to have [different vGPU profiles](https://docs.nvidia.com/vgpu/17.0/grid-vgpu-release-notes-generic-linux-kvm/index.html#different-vgpus-on-same-gpu-support) in a single GPU by enabling mixed-size mode for the GPU, according to NVIDIA’s documentation for vGPU v17. This feature does not seem to be available in v16, according to [docs](https://docs.nvidia.com/vgpu/16.0/grid-vgpu-release-notes-generic-linux-kvm/index.html#different-vgpus-on-same-gpu-support), so depending on the version you’re using you should be able to use several profiles with some limitations.

Please, take a look the NVIDIA documentation on how to [configure mixed-size mode in KVM](https://docs.nvidia.com/vgpu/17.0/grid-vgpu-user-guide/index.html#putting-gpu-into-mixed-size-mode-linux-with-kvm) and the [valid configurations](https://docs.nvidia.com/vgpu/17.0/grid-vgpu-user-guide/index.html#homogeneous-grid-vgpus).

---

<div class="post-metadata">

**Author:** ![Mayhem](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@Mayhem](https://forum.opennebula.io/u/Mayhem)\
**Post date:** [December 9, 2024, 6:04pm UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/5 "2024-12-09T18:04:18Z")

</div>

Thank you so much! I’ll try it soon as well.  
Could you please tell me if it is possible to deploy a VM from a template and how to specify a PCI device in the template so that mdev is created by OpenNebula oneadmin? I have to first manually create an mdev device on the host node and only then can I create a VM and not get an error with “mediated device”

---

<div class="post-metadata">

**Author:** ![amosteiro](https://avatars.discourse-cdn.com/v4/letter/a/439d5e/32.png) [@amosteiro](https://forum.opennebula.io/u/amosteiro)\
**Post date:** [December 10, 2024, 8:01am UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/6 "2024-12-10T08:01:52Z")

</div>

Yes, you can edit the VM template to deploy a VM using the vGPU. You can add the PCI device to the template following the [Usage](https://docs.opennebula.io/6.10/open_cluster_deployment/kvm_node/pci_passthrough.html#usage) section of the PCI Passthrough documentation. For this, you have to add the PCI section to the VM template (onetemplate update will open a text editor), after checking the parameters with onehost show. Add the PCI section at the end of your VM template as shown in the docs:

PCI = [  
VENDOR = “8086”,  
DEVICE = “0a0c”,  
CLASS = “0403” ]

---

<div class="post-metadata">

**Author:** ![Mayhem](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@Mayhem](https://forum.opennebula.io/u/Mayhem)\
**Post date:** [December 11, 2024, 5:56am UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/7 "2024-12-11T05:56:46Z")

</div>

Thank you so much!

---

<div class="post-metadata">

**Author:** ![a2k](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@a2k](https://forum.opennebula.io/u/a2k)\
**Post date:** [December 12, 2024, 8:12pm UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/8 "2024-12-12T20:12:21Z")

</div>

@Mayhem i am also trying to test this, i have NVidia driver working on the KVM host, however when i try to create a VM and add a PCI device nothing is displayed in the drop down. I also followed your steps above one thing that didn’t return an output was listing all the UUIDs, it returned with nothing displayed.

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/opennebula/original/2X/2/2469c44bc1c39870b8ec2336d8c85c0e9ff39b6c.png)

---

<div class="post-metadata">

**Author:** ![Mayhem](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@Mayhem](https://forum.opennebula.io/u/Mayhem)\
**Post date:** [December 13, 2024, 9:49am UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/9 "2024-12-13T09:49:39Z")

</div>

Hello, @a2k

When VM is created and Guest Driver installed on the Host Node you should check that nvidia-smi works on VM.

When VM with driver running you may see it with nvidia-smi vgpu on the host node

Have you already added all mdevs? Guest Drivers been installed?

---

<div class="post-metadata">

**Author:** ![amosteiro](https://avatars.discourse-cdn.com/v4/letter/a/439d5e/32.png) [@amosteiro](https://forum.opennebula.io/u/amosteiro)\
**Post date:** [December 13, 2024, 1:37pm UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/10 "2024-12-13T13:37:47Z")

</div>

The fact that nothing is displayed in the drop down menu is a [bug](https://github.com/OpenNebula/one/issues/6293) that will be fixed in next releases of OpenNebula.

---

<div class="post-metadata">

**Author:** ![a2k](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@a2k](https://forum.opennebula.io/u/a2k)\
**Post date:** [December 13, 2024, 3:04pm UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/11 "2024-12-13T15:04:10Z")

</div>

> [@Mayhem](#):
>
> `/var/tmp/one/im/kvm-probes.d/host/system/pci.rb`

no when i ran this command it returned no output /var/tmp/one/im/kvm-probes.d/host/system/pci.rb

---

<div class="post-metadata">

**Author:** ![Daniel\_Ruiz\_Molina](https://avatars.discourse-cdn.com/v4/letter/d/f08c70/32.png) [@Daniel\_Ruiz\_Molina](https://forum.opennebula.io/u/Daniel_Ruiz_Molina)\
**Post date:** [January 7, 2026, 1:43pm UTC](https://forum.opennebula.io/t/vgpu-experience-and-profiles/13332/12 "2026-01-07T13:43:35Z")

</div>

Me the same… `/var/tmp/one/im/kvm-probes.d/host/system/pci.rb` doesn’t return anything…