Unable to start former firecracker vm as kvm microvm instances

Community Support
1

Hello,

oft loosing the firecracker support due to the last upgrade I tried to convert my firecracker instances to kvm microvms. I changed the virtualisation type to kvm and the machine type to microvm but everytime I get an error at boot that there are no pci busses available.

I would really appreciate any hint.

AUTOMATIC_DS_REQUIREMENTS = "(\"CLUSTERS/ID\" @> 0) & (TM_MAD = \"ssh\")"
AUTOMATIC_NIC_REQUIREMENTS = "(\"CLUSTERS/ID\" @> 0)"
AUTOMATIC_REQUIREMENTS = "(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED)"
CONTEXT = [
  DISK_ID = "2",
  ETH0_DNS = "1.1.1.1 8.8.8.8",
  ETH0_EXTERNAL = "",
  ETH0_GATEWAY = "192.168.0.1",
  ETH0_IP = "192.168.0.105",
  ETH0_IP6 = "",
  ETH0_IP6_GATEWAY = "",
  ETH0_IP6_METHOD = "",
  ETH0_IP6_METRIC = "",
  ETH0_IP6_PREFIX_LENGTH = "",
  ETH0_IP6_ULA = "",
  ETH0_MAC = "02:00:c0:a8:00:69",
  ETH0_MASK = "",
  ETH0_METHOD = "",
  ETH0_METRIC = "",
  ETH0_MTU = "",
  ETH0_NETWORK = "",
  ETH0_SEARCH_DOMAIN = "",
  ETH0_VLAN_ID = "",
  ETH0_VROUTER_IP = "",
  ETH0_VROUTER_IP6 = "",
  ETH0_VROUTER_MANAGEMENT = "",
  FILES_DS = "/home/opennebula/datastores/2/473882e9c17f69595507afcfa6dfdf1b:'init-wrapper' ",
  HOME = "/root",
  LANG = "en_US.UTF-8",
  LANGUAGE = "en_US.UTF-8",
  MEM_LIMIT = "4096M",
  MONGO_AUTHSOURCE = "admin",
  MONGO_DBNAME = "unifi",
  MONGO_HOST = "192.168.0.106",
  MONGO_INITDB_ROOT_PASSWORD = "",
  MONGO_INITDB_ROOT_USERNAME = "root",
  MONGO_PASS = "Kennwort1",
  MONGO_USER = "root",
  NETWORK = "YES",
  PATH = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  PGID = "1000",
  PUID = "1000",
  SET_HOSTNAME = "unifi01",
  SSH_PUBLIC_KEY = "",
  START_SCRIPT_BASE64 = "ZWNobyAtZSAiS2VubndvcnQxXG5LZW5ud29ydDEiIHwgcGFzc3dkIHJvb3QKbW91bnQgL2Rldi92ZGIgL2NvbmZpZwptb3VudCAvZGV2L3ZkYyAvbW50CnNvdXJjZSAvbW50L2NvbnRleHQuc2gKbm9odXAgYmFzaCAvbW50L2luaXQtd3JhcHBlciAgPiAvdmFyL2xvZy91bmlmaS5sb2cgMj4mMSAm",
  TARGET = "hda",
  TZ = "Europe/Berlin" ]
CPU = "1"
DISK = [
  ALLOW_ORPHANS = "FORMAT",
  CLONE = "YES",
  CLONE_TARGET = "SYSTEM",
  CLUSTER_ID = "0",
  DATASTORE = "img_raw",
  DATASTORE_ID = "100",
  DEV_PREFIX = "sd",
  DISK_ID = "0",
  DISK_SNAPSHOT_TOTAL_SIZE = "0",
  DISK_TYPE = "FILE",
  DRIVER = "raw",
  FORMAT = "raw",
  IMAGE = "unifi-network-application666",
  IMAGE_ID = "275",
  IMAGE_STATE = "2",
  IMAGE_UNAME = "oneadmin",
  LN_TARGET = "NONE",
  ORIGINAL_SIZE = "3072",
  READONLY = "NO",
  SAVE = "NO",
  SIZE = "3072",
  SOURCE = "/home/opennebula/datastores/100/0a885186ef827de64dc7210d50a6e6be",
  TARGET = "sda",
  TM_MAD = "shared",
  TM_MAD_SYSTEM = "ssh",
  TYPE = "FILE" ]
DISK = [
  ALLOW_ORPHANS = "FORMAT",
  CLONE = "NO",
  CLONE_TARGET = "SYSTEM",
  CLUSTER_ID = "0",
  DATASTORE = "img_raw",
  DATASTORE_ID = "100",
  DEV_PREFIX = "sd",
  DISK_ID = "1",
  DISK_SNAPSHOT_TOTAL_SIZE = "0",
  DISK_TYPE = "FILE",
  DRIVER = "raw",
  FORMAT = "raw",
  IMAGE = "unifi-data01",
  IMAGE_ID = "276",
  IMAGE_STATE = "8",
  IMAGE_UNAME = "oneadmin",
  LN_TARGET = "NONE",
  ORIGINAL_SIZE = "8048",
  PERSISTENT = "YES",
  READONLY = "NO",
  SAVE = "YES",
  SIZE = "8048",
  SOURCE = "/home/opennebula/datastores/100/3d62a8d5a9c89c6879edff125dc55aa2",
  TARGET = "sdb",
  TM_MAD = "shared",
  TM_MAD_SYSTEM = "ssh",
  TYPE = "FILE" ]
GRAPHICS = [
  LISTEN = "0.0.0.0",
  PORT = "6028",
  TYPE = "VNC" ]
MEMORY = "4096"
MEMORY_RESIZE_MODE = "BALLOONING"
NIC = [
  AR_ID = "0",
  BRIDGE = "extbr",
  BRIDGE_TYPE = "linux",
  CLUSTER_ID = "0",
  GATEWAY = "192.168.0.1",
  IP = "192.168.0.105",
  MAC = "02:00:c0:a8:00:69",
  NAME = "NIC0",
  NETWORK = "vnet-ext",
  NETWORK_ID = "1",
  NETWORK_UNAME = "oneadmin",
  NIC_ID = "0",
  PHYDEV = "enp3s0",
  SECURITY_GROUPS = "0",
  TARGET = "one-128-0",
  VN_MAD = "bridge" ]
OS = [
  FIRMWARE = "BIOS",
  FIRMWARE_SECURE = "NO",
  KERNEL = "/home/opennebula/datastores/0/128/kernel",
  KERNEL_CMD = "console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd",
  KERNEL_DS = "$FILE[IMAGE_ID=2]",
  KERNEL_DS_CLUSTER_ID = "0",
  KERNEL_DS_DSID = "2",
  KERNEL_DS_ID = "2",
  KERNEL_DS_SOURCE = "/home/opennebula/datastores/2/b7f31f8c0cac2c9869e997b1e0db19db",
  KERNEL_DS_TM = "ssh",
  MACHINE = "microvm",
  UUID = "00b33099-0ab2-4061-a046-57852fc53824" ]
SECURITY_GROUP_RULE = [
  PROTOCOL = "ALL",
  RULE_TYPE = "OUTBOUND",
  SECURITY_GROUP_ID = "0",
  SECURITY_GROUP_NAME = "default" ]
SECURITY_GROUP_RULE = [
  PROTOCOL = "ALL",
  RULE_TYPE = "INBOUND",
  SECURITY_GROUP_ID = "0",
  SECURITY_GROUP_NAME = "default" ]
TEMPLATE_ID = "22"
TM_MAD_SYSTEM = "ssh"
VCPU = "2"
VMID = "128"
BACKUP_CONFIG = [
 ]
BACKUP_IDS = [
 ]



Fri Nov 29 12:16:25 2024 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Fri Nov 29 12:16:25 2024 [Z0][VMM][E]: DEPLOY: error: Failed to create domain from /home/opennebula/datastores/0/128/deployment.17 error: XML error: No PCI buses available Could not create domain from /home/opennebula/datastores/0/128/deployment.17 ExitCode: 255
2

Hi,

This is probably not the anticipated answer, but according to the qemu docs, PCI is not supported for the microvms…

It’s a minimalist machine type without PCI nor ACPI support, designed for short-lived guests.

Best Regards,
Anton

3

Yes you are right but i have no idea where to set this up. I thought if I just choose machine type microvm these settings are made.

4

Hello @BenvanDamme ,

just a question: what type of upgrade did you do that cause that your firecracker support don’t work now? I’m thinking about a upgrade in February (during university non-classes weeks) of my OpenNebula systems, that now are running an Ubuntu-22.04-3 Server with OpenNebula-6.8.0. My environment have three servers: one that acts as server and as KVM node, a second one that is only KVM node and a third one that acts as Firecracker node. So it would be interesting to know what happened to you to decide if I, finally, upgrade or not.

Thanks.

5

Could you paste an example of a VM Template - an old one with the firecracker, and one defined for microvm? (onetemplate show -x $TEMPLATE_ID)
Probably we could try finding a way if there are no PCI devices needed…

6

Hi,

this did run before updating to 6.10.

<VMTEMPLATE>
  <ID>9</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>unifi_controller</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>0</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <REGTIME>1700398098</REGTIME>
  <TEMPLATE>
    <CONTEXT>
      <FILES_DS><![CDATA[$FILE[IMAGE="init-wrapper", IMAGE_UNAME="oneadmin"]]]></FILES_DS>
      <HOME><![CDATA[/root]]></HOME>
      <LANG><![CDATA[en_US.UTF-8]]></LANG>
      <LANGUAGE><![CDATA[en_US.UTF-8]]></LANGUAGE>
      <MEM_LIMIT><![CDATA[4096M]]></MEM_LIMIT>
      <NETWORK><![CDATA[YES]]></NETWORK>
      <PATH><![CDATA[/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin]]></PATH>
      <PGID><![CDATA[1000]]></PGID>
      <PUID><![CDATA[1000]]></PUID>
      <SET_HOSTNAME><![CDATA[$NAME]]></SET_HOSTNAME>
      <SSH_PUBLIC_KEY><![CDATA[$USER[SSH_PUBLIC_KEY]]]></SSH_PUBLIC_KEY>
      <START_SCRIPT_BASE64><![CDATA[ZWNobyAtZSAiS2VubndvcnQxXG5LZW5ud29ydDEiIHwgcGFzc3dkIHJvb3QKbW91bnQgL2Rldi92ZGIgL2NvbmZpZwptb3VudCAvZGV2L3ZkYyAvbW50CnNvdXJjZSAvbW50L2NvbnRleHQuc2gKbm9odXAgYmFzaCAvbW50L2luaXQtd3JhcHBlciAgPiAvdmFyL2xvZy91bmlmaS5sb2cgMj4mMSAm]]></START_SCRIPT_BASE64>
      <TZ><![CDATA[Europe/Berlin]]></TZ>
    </CONTEXT>
    <CPU><![CDATA[1]]></CPU>
    <DISK>
      <IMAGE><![CDATA[unifi_controller_ghcr]]></IMAGE>
      <IMAGE_UNAME><![CDATA[oneadmin]]></IMAGE_UNAME>
    </DISK>
    <DISK>
      <IMAGE><![CDATA[unifi-data]]></IMAGE>
      <IMAGE_UNAME><![CDATA[oneadmin]]></IMAGE_UNAME>
    </DISK>
    <GRAPHICS>
      <LISTEN><![CDATA[0.0.0.0]]></LISTEN>
      <TYPE><![CDATA[VNC]]></TYPE>
    </GRAPHICS>
    <HOT_RESIZE>
      <CPU_HOT_ADD_ENABLED><![CDATA[NO]]></CPU_HOT_ADD_ENABLED>
      <MEMORY_HOT_ADD_ENABLED><![CDATA[NO]]></MEMORY_HOT_ADD_ENABLED>
    </HOT_RESIZE>
    <HYPERVISOR><![CDATA[firecracker]]></HYPERVISOR>
    <LOGO><![CDATA[images/logos/alpine.png]]></LOGO>
    <MEMORY><![CDATA[4096]]></MEMORY>
    <MEMORY_RESIZE_MODE><![CDATA[BALLOONING]]></MEMORY_RESIZE_MODE>
    <MEMORY_UNIT_COST><![CDATA[MB]]></MEMORY_UNIT_COST>
    <NIC>
      <NETWORK><![CDATA[vnet-ext]]></NETWORK>
      <NETWORK_UNAME><![CDATA[oneadmin]]></NETWORK_UNAME>
      <SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
    </NIC>
    <OS>
      <FIRMWARE><![CDATA[]]></FIRMWARE>
      <FIRMWARE_SECURE><![CDATA[YES]]></FIRMWARE_SECURE>
      <KERNEL_CMD><![CDATA[console=ttyS0 reboot=k panic=1]]></KERNEL_CMD>
      <KERNEL_DS><![CDATA[$FILE[IMAGE_ID=2]]]></KERNEL_DS>
    </OS>
    <SCHED_REQUIREMENTS><![CDATA[DATASTORE = sys_raw & HYPERVISOR = firecracker]]></SCHED_REQUIREMENTS>
    <TM_MAD_SYSTEM><![CDATA[ssh]]></TM_MAD_SYSTEM>
    <VCPU><![CDATA[2]]></VCPU>

Then I tried to switch to kvm and ended with this, which does not run.

<VMTEMPLATE>
  <ID>22</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>unifi_network_app</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>0</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <REGTIME>1731417131</REGTIME>
  <TEMPLATE>
    <CONTEXT>
      <FILES_DS><![CDATA[$FILE[IMAGE="init-wrapper", IMAGE_UNAME="oneadmin"]]]></FILES_DS>
      <HOME><![CDATA[/root]]></HOME>
      <LANG><![CDATA[en_US.UTF-8]]></LANG>
      <LANGUAGE><![CDATA[en_US.UTF-8]]></LANGUAGE>
      <MEM_LIMIT><![CDATA[4096M]]></MEM_LIMIT>
      <MONGO_AUTHSOURCE><![CDATA[admin]]></MONGO_AUTHSOURCE>
      <MONGO_DBNAME><![CDATA[unifi]]></MONGO_DBNAME>
      <MONGO_HOST><![CDATA[192.168.0.106]]></MONGO_HOST>
      <MONGO_INITDB_ROOT_PASSWORD><![CDATA[Kennwort1]]></MONGO_INITDB_ROOT_PASSWORD>
      <MONGO_INITDB_ROOT_USERNAME><![CDATA[root]]></MONGO_INITDB_ROOT_USERNAME>
      <MONGO_PASS><![CDATA[Kennwort1]]></MONGO_PASS>
      <MONGO_USER><![CDATA[root]]></MONGO_USER>
      <NETWORK><![CDATA[YES]]></NETWORK>
      <PATH><![CDATA[/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin]]></PATH>
      <PGID><![CDATA[1000]]></PGID>
      <PUID><![CDATA[1000]]></PUID>
      <SET_HOSTNAME><![CDATA[$NAME]]></SET_HOSTNAME>
      <SSH_PUBLIC_KEY><![CDATA[$USER[SSH_PUBLIC_KEY]]]></SSH_PUBLIC_KEY>
      <START_SCRIPT_BASE64><![CDATA[ZWNobyAtZSAiS2VubndvcnQxXG5LZW5ud29ydDEiIHwgcGFzc3dkIHJvb3QKbW91bnQgL2Rldi92ZGIgL2NvbmZpZwptb3VudCAvZGV2L3ZkYyAvbW50CnNvdXJjZSAvbW50L2NvbnRleHQuc2gKbm9odXAgYmFzaCAvbW50L2luaXQtd3JhcHBlciAgPiAvdmFyL2xvZy91bmlmaS5sb2cgMj4mMSAm]]></START_SCRIPT_BASE64>
      <TZ><![CDATA[Europe/Berlin]]></TZ>
    </CONTEXT>
    <DISK>
      <IMAGE><![CDATA[unifi-network-application666]]></IMAGE>
      <IMAGE_UNAME><![CDATA[oneadmin]]></IMAGE_UNAME>
    </DISK>
    <DISK>
      <IMAGE><![CDATA[unifi-data01]]></IMAGE>
      <IMAGE_UNAME><![CDATA[oneadmin]]></IMAGE_UNAME>
    </DISK>
    <GRAPHICS>
      <LISTEN><![CDATA[0.0.0.0]]></LISTEN>
      <TYPE><![CDATA[VNC]]></TYPE>
    </GRAPHICS>
    <HOT_RESIZE>
      <CPU_HOT_ADD_ENABLED><![CDATA[NO]]></CPU_HOT_ADD_ENABLED>
      <MEMORY_HOT_ADD_ENABLED><![CDATA[NO]]></MEMORY_HOT_ADD_ENABLED>
    </HOT_RESIZE>
    <HYPERVISOR><![CDATA[kvm]]></HYPERVISOR>
    <LOGO><![CDATA[images/logos/alpine.png]]></LOGO>
    <MEMORY><![CDATA[4096]]></MEMORY>
    <MEMORY_RESIZE_MODE><![CDATA[BALLOONING]]></MEMORY_RESIZE_MODE>
    <MEMORY_UNIT_COST><![CDATA[MB]]></MEMORY_UNIT_COST>
    <NIC>
      <NETWORK><![CDATA[vnet-ext]]></NETWORK>
      <NETWORK_UNAME><![CDATA[oneadmin]]></NETWORK_UNAME>
      <SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
    </NIC>
    <OS>
      <ARCH><![CDATA[x86_64]]></ARCH>
      <FIRMWARE><![CDATA[/usr/share/OVMF/OVMF_CODE.fd]]></FIRMWARE>
      <FIRMWARE_SECURE><![CDATA[NO]]></FIRMWARE_SECURE>
      <KERNEL_CMD><![CDATA[console=ttyS0 reboot=k panic=1]]></KERNEL_CMD>
      <KERNEL_DS><![CDATA[$FILE[IMAGE_ID=2]]]></KERNEL_DS>
      <MACHINE><![CDATA[q35]]></MACHINE>
    </OS>
    <SCHED_REQUIREMENTS><![CDATA[DATASTORE = sys_raw & HYPERVISOR = kvm]]></SCHED_REQUIREMENTS>
    <TM_MAD_SYSTEM><![CDATA[ssh]]></TM_MAD_SYSTEM>
    <VCPU><![CDATA[2]]></VCPU>
  </TEMPLATE>
</VMTEMPLATE>
7

Hi,

in 6.10 the firecracker support is discarded and the documentation says that you could use kvm instead.
kvm does have the machine type microvm so I thought it should be an easy replacement, but it is not.

8

When I run this in the deployed folder the microvm starts fine, but I am unable to transform the command to a opennebula template.
Where do I go wrong?

qemu-system-x86_64 -M microvm,x-option-roms=off,isa-serial=off,rtc=off -no-acpi -enable-kvm -cpu host -nodefaults -no-user-config -nographic -no-reboot -device virtio-serial-device -chardev stdio,id=virtiocon0 -device virtconsole,chardev=virtiocon0 -drive id=root,file=disk.0,format=raw,if=none -device virtio-blk-device,drive=root -kernel kernel -append "console=hvc0 root=/dev/vda rw acpi=off reboot=t panic=-1"
9

Hi,

OpenNebula supports KVM and LXC, with KVM closer to the microvms (sort of). But for KVM it is using libvirt, so if you manage to start a microvm instance via defined libvirt domain XML, there is a chance to see what should be done by OpenNebula to have similar domain XML.

Best Regards,
Anton Todorov

10

Hi Anton,

thank you very much for your help. I tried to google for this issue but it does not look very promising.
I would realy like to know what the creator of this table had in mind when he mentionend kvm as the replacement for firecracker.
https://docs.opennebula.io/6.10/intro_release_notes/release_notes/whats_new.html

Kind Regards,

Benjamin