Nic pci passthrough acts differently on ONe 5.12

Community Support General
1

Hi all,

some of my virtual machines use a sriov virtual function as network interface. This vf is added to the vm using pci passthrough.

The network template looks like this:

  <TEMPLATE>
      <BRIDGE><![CDATA[sriov]]></BRIDGE>
      <BRIDGE_TYPE><![CDATA[linux]]></BRIDGE_TYPE>
      <DESCRIPTION><![CDATA[sriov virtual function]]></DESCRIPTION>
      <LABELS><![CDATA[mylabel]]></LABELS>
      <OUTER_VLAN_ID><![CDATA[]]></OUTER_VLAN_ID>
      <PHYDEV><![CDATA[]]></PHYDEV>
      <SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
      <VLAN_ID><![CDATA[]]></VLAN_ID>
      <VN_MAD><![CDATA[dummy]]></VN_MAD>
    </TEMPLATE>

and the (relevant part of the) vm template like this:

    <TEMPLATE>
      <PCI>
        <CLASS><![CDATA[0200]]></CLASS>
        <DEVICE><![CDATA[8090]]></DEVICE>
        <NETWORK><![CDATA[sriov-vf-network]]></NETWORK>
        <NETWORK_UNAME><![CDATA[oneadmin]]></NETWORK_UNAME>
        <SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
        <TYPE><![CDATA[NIC]]></TYPE>
        <VENDOR><![CDATA[1077]]></VENDOR>
      </PCI>
    </TEMPLATE>

With ONe 5.10 the resulting definition is:

  <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    <devices>
      <hostdev mode='subsystem' type='pci' managed='yes'>
        <source>
          <address  domain='0x0000' bus='0x43' slot='0x14' function='0x0'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0'/>
      </hostdev>
    </devices>
  </domain>

With ONe 5.12:

  <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    <devices>
      <interface type='bridge'>
        <source bridge='sriov'/>
        <mac address='aa:05:ca:ba:98:00'/>
        <target dev='one-65-1'/>
        <model type='virtio'/>
      </interface>
    </devices>
  </domain>

Upon deployment virsh tries to create the pci-passtrough interface as a regular bridge, failing to find the “sriov” interface thus bailing out with

error: Cannot get interface MTU on 'sriov': No such device

Anyone got a clue on how to make pci passthrough work again?

2

fwiw – passing the device without type=nic and network=... works fine even though the device doesn’t show up as network device in sunstone. I can live with that but I liked it better by seeing the device as nic.

3

Hello @phil,

I think what is happening is this:

  1. When you updated from 5.10 to 5.12 the configuration files: /var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf and /etc/libvirt/qemu.conf were overwritten with the default values.
  2. As a result the PCI DEVICES are not monitored. You can check it with onehost show <HOST_ID>.
  3. If you have configured a PCI passthrough NIC, and the PCI device is not available, OpenNebula configures an interface of type bridge and model virtio because the network driver used is ‘dummy’. Then, as the ‘srviov’ does not exist and driver is ‘dummy’ (OpenNebula does not create the bridge if not exist), you get the error: error: Cannot get interface MTU on 'sriov': No such device.

To solve this issue, restore your previous configuration in /etc/libvirt/qemu.conf [0] (restart livirtd service), and /var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf [1] (restart opennebula and run onehost forceupdate). After that, try to instantiate the template again.

Hope this helps.

Cheers.

[0] http://docs.opennebula.io/5.12/deployment/open_cloud_host_setup/pci_passthrough.html?highlight=passthrough#qemu-configuration
[1] http://docs.opennebula.io/5.12/deployment/open_cloud_host_setup/pci_passthrough.html?highlight=passthrough#driver-configuration

4

Hi @rdiaz,

thanks for your answer. The kvm-probes.d/pci.conf is still the previous “working” one, it has not been overwritten by the package (the files are marked %config(noreplace) in the rpm) and would have been restored on my next ansible run.
I see all pci devices with onehost show and I can assign them directly to the virtual machines, simply not as TYPE = "NIC":

# pci.conf
:filter:
  - '1077:8070' # QLogic FastLinQ QL41000
  - '1077:8090' # QLogic FastLinQ QL41000 SRIOV VF

:short_address: []
:device_name: []

PCI DEVICES

   VM ADDR    TYPE           NAME
      03:00.0 1077:8070:0200 FastLinQ QL41000 Series 10/25/40/50GbE Controller
      03:00.1 1077:8070:0200 FastLinQ QL41000 Series 10/25/40/50GbE Controller
      85:00.0 1077:8070:0200 FastLinQ QL41000 Series 10/25/40/50GbE Controller
      85:00.1 1077:8070:0200 FastLinQ QL41000 Series 10/25/40/50GbE Controller
      85:00.2 1077:8070:0200 FastLinQ QL41000 Series 10/25/40/50GbE Controller
      85:00.3 1077:8070:0200 FastLinQ QL41000 Series 10/25/40/50GbE Controller
   84 85:02.0 1077:8090:0200 FastLinQ QL41000 Series Gigabit Ethernet Controlle
   84 85:02.1 1077:8090:0200 FastLinQ QL41000 Series Gigabit Ethernet Controlle
   84 85:02.2 1077:8090:0200 FastLinQ QL41000 Series Gigabit Ethernet Controlle
      85:02.3 1077:8090:0200 FastLinQ QL41000 Series Gigabit Ethernet Controlle

I see the passedthough nics in the VM as interface device and they’re working fine.
FWIW: I did not blacklist the corresponding modules and I did not explicitly bind the pci devices to vfio-pci because it’s not needed:

$ ls -l /sys/bus/pci/drivers/vfio-pci/*:*
lrwxrwxrwx 1 root root 0 21. Okt 13:17 /sys/bus/pci/drivers/vfio-pci/0000:85:02.0 -> ../../../../devices/pci0000:80/0000:80:03.1/0000:85:02.0
lrwxrwxrwx 1 root root 0 21. Okt 13:17 /sys/bus/pci/drivers/vfio-pci/0000:85:02.1 -> ../../../../devices/pci0000:80/0000:80:03.1/0000:85:02.1
lrwxrwxrwx 1 root root 0 21. Okt 13:17 /sys/bus/pci/drivers/vfio-pci/0000:85:02.2 -> ../../../../devices/pci0000:80/0000:80:03.1/0000:85:02.2

This should not be relevant to OpenNebula, though.

¯\(ツ)/¯

5

Hi @phil,

Please send the following information:

onetemplate show -x <TEMPLATE_ID>
onevm show -x <VM_ID

For both cases: device with and without type=nic and network=...

Thanks.