# PCI Passthrough VM error

**URL:** <https://forum.opennebula.io/t/pci-passthrough-vm-error/6853>\
**Category:** Installation & Configuration\
**Created:** [February 1, 2019, 9:56am UTC](https://forum.opennebula.io/t/pci-passthrough-vm-error/6853 "2019-02-01T09:56:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![plopes](https://avatars.discourse-cdn.com/v4/letter/p/ce73a5/32.png) [@plopes](https://forum.opennebula.io/u/plopes)\
**Post date:** [February 1, 2019, 9:56am UTC](https://forum.opennebula.io/t/pci-passthrough-vm-error/6853/1 "2019-02-01T09:56:20Z")

</div>

Hello,  
I was trying to implement PCI Passthrough on a host. The VM problem is:

Thu Jan 31 16:10:07 2019 [Z0][VM][I]: New state is ACTIVE  
Thu Jan 31 16:10:07 2019 [Z0][VM][I]: New LCM state is PROLOG  
Thu Jan 31 16:10:23 2019 [Z0][VM][I]: New LCM state is BOOT  
Thu Jan 31 16:10:27 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/46/deployment.0  
Thu Jan 31 16:10:29 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm\_context.  
Thu Jan 31 16:10:30 2019 [Z0][VMM][I]: ExitCode: 0  
Thu Jan 31 16:10:30 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.  
Thu Jan 31 16:10:32 2019 [Z0][VMM][I]: Command execution fail: cat \<\< EOT | /var/tmp/one/vmm/kvm/deploy ‘/var/lib/one//datastores/0/46/deployment.0’ ‘opennebula’ 46 opennebula  
Thu Jan 31 16:10:32 2019 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/0/46/deployment.0  
Thu Jan 31 16:10:32 2019 [Z0][VMM][I]: error: internal error: process exited while connecting to monitor: 2019-01-31T16:10:32.030824Z qemu-system-x86\_64: warning: host doesn’t support requested feature: CPUID.80000001H:ECX.svm [bit 2]  
Thu Jan 31 16:10:32 2019 [Z0][VMM][I]: 2019-01-31T16:10:32.060795Z qemu-system-x86\_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.1,addr=0x1: vfio error: 0000:02:00.0: group 43 is not viable  
Thu Jan 31 16:10:32 2019 [Z0][VMM][I]: Please ensure all devices within the iommu\_group are bound to their vfio bus driver.  
Thu Jan 31 16:10:32 2019 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/0/46/deployment.0  
Thu Jan 31 16:10:32 2019 [Z0][VMM][I]: ExitCode: 255  
Thu Jan 31 16:10:32 2019 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.  
Thu Jan 31 16:10:32 2019 [Z0][VMM][E]: Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/0/46/deployment.0  
Thu Jan 31 16:10:32 2019 [Z0][VM][I]: New LCM state is BOOT\_FAILURE

Has anyone ever had or knows what might be the problem?  
Hope that someone can help me here.

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [February 4, 2019, 9:13am UTC](https://forum.opennebula.io/t/pci-passthrough-vm-error/6853/2 "2019-02-04T09:13:43Z")

</div>

> [@plopes](#):
>
> Please ensure all devices within the iommu\_group are bound to their vfio bus driver.

This probably means a limitation on your hypervisors, more details here:

> **[IOMMU Groups, inside and out](http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html)**
>
> Information on using VFIO for assigning PCI/VGA/GPU devices to Linux and Windows guests with QEMU/KVM

---

<div class="post-metadata">

**Author:** ![plopes](https://avatars.discourse-cdn.com/v4/letter/p/ce73a5/32.png) [@plopes](https://forum.opennebula.io/u/plopes)\
**Post date:** [February 6, 2019, 10:35am UTC](https://forum.opennebula.io/t/pci-passthrough-vm-error/6853/3 "2019-02-06T10:35:45Z")

</div>

Good Morning  
Thanks Ruben for the tip. The problem was not assigning all iommu\_groups devices to the Guest VM.  
In spite of everything has now appeared another problem, in VM we can see the graphics card, we installed the corresponding drivers in our case nvidia-410, but although the card is visible the drivers can not get access to the graphics card in our case is an nvidia geforce gtx 1080 ti.  
Has anyone had this problem?

$ lspci -v  
01:01.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])  
Subsystem: ASUSTeK Computer Inc. GP102 [GeForce GTX 1080 Ti]  
Physical Slot: 1  
Flags: bus master, fast devsel, latency 0, IRQ 11  
Memory at fc000000 (32-bit, non-prefetchable) [size=16M]  
Memory at d0000000 (64-bit, prefetchable) [size=256M]  
Memory at e0000000 (64-bit, prefetchable) [size=32M]  
I/O ports at c000 [size=128]  
Expansion ROM at fd000000 [disabled] [size=128K]  
Capabilities: \<access denied\>  
Kernel driver in use: nvidia  
Kernel modules: nvidiafb, nouveau, nvidia\_drm, nvidia

01:02.0 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)  
Subsystem: ASUSTeK Computer Inc. GP102 HDMI Audio Controller  
Physical Slot: 2  
Flags: bus master, fast devsel, latency 0, IRQ 10  
Memory at fd020000 (32-bit, non-prefetchable) [size=16K]  
Capabilities: \<access denied\>  
Kernel driver in use: snd\_hda\_intel  
Kernel modules: snd\_hda\_intel

$ nvidia-smi  
Unable to determine the device handle for GPU 0000:01:01.0: Unknown Error

---

<div class="post-metadata">

**Author:** ![knawnd](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/knawnd/32/6119_2.png) [@knawnd](https://forum.opennebula.io/u/knawnd)\
**Post date:** [February 8, 2019, 5:51am UTC](https://forum.opennebula.io/t/pci-passthrough-vm-error/6853/4 "2019-02-08T05:51:40Z")

</div>

@plopes, I was able to pass through GPU-card successfully into KVM VM but on libvirt/qemu level.

Steps from my notes are posted below.

**Passing through PCI-devices (GPU)**

# References:

> **[GPU Virtualization with KVM / QEMU – Cale Rogers – Medium](https://medium.com/@calerogers/gpu-virtualization-with-kvm-qemu-63ca98a6a172)**
>
> This is part two of our building a deep learning machine series. You can find the other posts here:

  
[https://www.server-world.info/en/note?os=CentOS\_7&p=kvm&f=10](https://www.server-world.info/en/note?os=CentOS_7&p=kvm&f=10)  
[https://wiki.archlinux.org/index.php/PCI\_passthrough\_via\_OVMF](https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF)

# VT-d and IOMMU

Ensure that AMD-Vi/Intel VT-d is supported by the CPU and enabled in the BIOS settings.  
Enable IOMMU:  
`grubby --update-kernel=ALL --args='intel_iommu=on iommu=pt'`

Reboot.

`dmesg | grep -e DMAR -e IOMMU`

Ensuring that the groups are valid

```
cat list_iommu_groups.sh
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

```

`bash list_iommu_groups.sh|grep -i nvidia`

# VFIO

Extract PCI vendor-device ID pair:

```
lspci -nn |grep -i nvidia|grep -e "\[.*\]"
(should be something like [10de:15f8])

cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:15f8

```

Re-generate initramfs image  
`dracut -f`

Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices:  
`dmesg | grep -i vfio`

It is not necessary for all devices (or even expected device) from vfio.conf to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operational in guest VM:

```
lspci -nnk -d 10de:15f8
06:00.0 3D controller [0302]: NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] [10de:15f8] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:118f]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

```

# Update QEMU

```
/usr/libexec/qemu-kvm -version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-141.el7_4.6), Copyright (c) 2003-2008 Fabrice Bellard

```

`yum install centos-release-qemu-ev`

`sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo`

`yum --enablerepo=centos-qemu-ev install qemu-kvm-ev`

`systemctl restart libvirtd`

```
/usr/libexec/qemu-kvm -version
QEMU emulator version 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.7.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

```

# Setting up an OVMF-based guest VM

`yum install OVMF`

```
rpm -ql OVMF|grep -i fd
/usr/share/OVMF/OVMF_CODE.secboot.fd
/usr/share/OVMF/OVMF_VARS.fd

```

Update /etc/libvirt/qemu.conf

```
diff /etc/libvirt/qemu.conf{,.orig}
731,733d730
< nvram = [
< "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd"
< ]

```

`systemctl restart libvirtd.service`

Creating GPU-enabled KVM VM

```
virt-install \
--name centos7 \
--ram 8192 \
--disk path=/var/kvm/images/centos7.img,size=30 \
--vcpus 4 \
--os-type linux \
--os-variant rhel7 \
--network bridge=br0 \
--graphics none \
--console pty,target_type=serial \
--location 'http://ftp.iij.ad.jp/pub/linux/centos/7/os/x86_64/' \
--extra-args 'console=ttyS0,115200n8 serial' \
--host-device 03:00.0 \
--features kvm_hidden=on \
--machine q35

```

Not sure if kvm\_hidden=on is mandatory but if it is and in case it’s missing in already existing VM one can add that feature:

`virsh shutdown <vm-name>`

`virsh edit <vm-name>`

Add to a file between tags

```
<kvm>
  <hidden state='on'/>
</kvm>

```

# CUDA and NVidia driver installation

Reference:  
[https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pre-installation-actions](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pre-installation-actions)

Disable nouveau driver:

```
grubby --info=ALL
grubby --update-kernel=ALL --args="nouveau.modeset=0 rd.driver.blacklist=nouveau"
dracut -f

```

or (and?)

```
vi /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0

```

Install required packages:

```
yum install gcc kernel-devel-$(uname -r) kernel-headers-$(uname -r)
yum install epel-release

```

`yum install http://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-10.0.130-1.x86_64.rpm`

`yum clean all`

`yum install cuda`

No need to reboot.

`lsmod|grep -i nvidia`

`nvidia-smi`

---

<div class="post-metadata">

**Author:** ![plopes](https://avatars.discourse-cdn.com/v4/letter/p/ce73a5/32.png) [@plopes](https://forum.opennebula.io/u/plopes)\
**Post date:** [February 20, 2019, 10:39am UTC](https://forum.opennebula.io/t/pci-passthrough-vm-error/6853/5 "2019-02-20T10:39:08Z")

</div>

Good Morning,  
Thanks for the information.  
In this momente there is only one problem is that the host (Ubuntu 4.18) has two graphics cards NVIDIA and onboard (aspeed) and regardless of the settings made always detects the two but uses NVIDIA and not onboard, later giving error in the allocation to the VM because it is in use. We have experienced many configuration, there is some particularity that is escaping us?

Hope that someone can help me here.