Firecracker VM starts then powers off after some seconds


I’m able to successfully start a firecracker VM, but after about 30 seconds or so, it powers off. I’ve seen this issue mentioned in the forums here, but no solution that worked for me. I’m able to run a firecracker VM directly myself on the same host manually without issue. I’m using the OS images (Ubuntu, Alpine, Debian–all have the same issue) along with the Kernel Image (Kernel 5.4 x86_64 - Firecracker) from the Apps page. I’ve tried deploying both with and without the network. I’ve ran the network steps for firecracker setup so I don’t think that’s the issue.

Versions of the related components and OS (frontend, hypervisors, VMs):

Frontend:

opennebula-common-onecfg/stable,stable,now 6.8.0-1 all [installed,automatic]
opennebula-common/stable,stable,now 6.8.0-1 all [installed,automatic]
opennebula-context/jammy,jammy 4.14.0-1 all
opennebula-dbgsym/stable 6.8.0-1 amd64
opennebula-fireedge/stable,now 6.8.0-1 amd64 [installed]
opennebula-flow/stable,stable,now 6.8.0-1 all [installed]
opennebula-gate/stable,stable,now 6.8.0-1 all [installed]
opennebula-guacd/stable,now 6.8.0-1 amd64 [installed,automatic]
opennebula-libs/stable,stable,now 6.8.0-1 all [installed,automatic]
opennebula-node-firecracker/stable,now 6.8.0-1 amd64 [installed]
opennebula-node-kvm/stable,stable,now 6.8.0-1 all [residual-config]
opennebula-node-lxc/stable 6.8.0-1 amd64
opennebula-node-lxd/stable 6.8.0-1 amd64
opennebula-provision-data/stable,stable,now 6.8.0-1 all [installed,automatic]
opennebula-provision/stable,stable,now 6.8.0-1 all [installed]
opennebula-rubygems/stable,now 6.8.0-1 amd64 [installed,automatic]
opennebula-sunstone/stable,stable,now 6.8.0-1 all [installed]
opennebula-tools/stable,stable,now 6.8.0-1 all [installed,automatic]
opennebula/stable,now 6.8.0-1 amd64 [installed]

Host:

Firecracker v0.24.2
opennebula-common-onecfg/stable,now 6.8.0-1 all [installed,automatic]
opennebula-common/stable,now 6.8.0-1 all [installed,automatic]
opennebula-context/jammy 4.14.0-1 all
opennebula-dbgsym/stable 6.8.0-1 amd64
opennebula-fireedge/stable 6.8.0-1 amd64
opennebula-flow/stable 6.8.0-1 all
opennebula-gate/stable 6.8.0-1 all
opennebula-guacd/stable 6.8.0-1 amd64
opennebula-libs/stable 6.8.0-1 all
opennebula-node-firecracker/stable,now 6.8.0-1 amd64 [installed]
opennebula-node-kvm/stable,now 6.8.0-1 all [residual-config]
opennebula-node-lxc/stable 6.8.0-1 amd64
opennebula-node-lxd/stable 6.8.0-1 amd64
opennebula-provision-data/stable 6.8.0-1 all
opennebula-provision/stable 6.8.0-1 all
opennebula-rubygems/stable 6.8.0-1 amd64
opennebula-sunstone/stable 6.8.0-1 all
opennebula-tools/stable 6.8.0-1 all
opennebula/stable 6.8.0-1 amd64

Steps to reproduce:

Download the kernel and OS images and use the template below to start a firecracker VM.
I’m testing this on a virtualbox host (ubuntu 22.04) with nested virtualization enabled.

Template:

CONTEXT = [
  NETWORK = "YES",
  SSH_PUBLIC_KEY = "$USER[SSH_PUBLIC_KEY]" ]
CPU = "1"
DISK = [
  IMAGE = "Debian 12",
  IMAGE_UNAME = "oneadmin" ]
GRAPHICS = [
  LISTEN = "0.0.0.0",
  TYPE = "VNC" ]
HOT_RESIZE = [
  CPU_HOT_ADD_ENABLED = "NO",
  MEMORY_HOT_ADD_ENABLED = "NO" ]
MEMORY = "512"
MEMORY_RESIZE_MODE = "BALLOONING"
MEMORY_UNIT_COST = "MB"
NIC = [
  NETWORK = "onprem-cluster-public",
  NETWORK_UNAME = "oneadmin",
  SECURITY_GROUPS = "0" ]
OS = [
  KERNEL_CMD = "console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd",
  KERNEL_DS = "$FILE[IMAGE_ID=6]" ]
VCPU = "2"

Current results:

VM starts successfully but then goes into shutdown. Log:

Tue Mar 19 13:44:18 2024 [Z0][VM][I]: New state is ACTIVE
Tue Mar 19 13:44:18 2024 [Z0][VM][I]: New LCM state is PROLOG
Tue Mar 19 13:44:28 2024 [Z0][VM][I]: New LCM state is BOOT
Tue Mar 19 13:44:28 2024 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/82/deployment.0
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: pre: Running /var/tmp/one/vnm/bridge/pre.d/firecracker
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: pre:
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: ExitCode: 0
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: ExitCode: 0
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/mkdir -p.
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: ExitCode: 0
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/113/82/vm.xml.
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: ExitCode: 0
Tue Mar 19 13:44:29 2024 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/113/82/ds.xml.
Tue Mar 19 13:44:30 2024 [Z0][VMM][I]: Running command sudo -n /usr/sbin/one-clean-firecracker-domain -c /sys/fs/cgroup -v one-82 -t 60
Tue Mar 19 13:44:30 2024 [Z0][VMM][I]: Running command sudo -n /usr/sbin/one-prepare-firecracker-domain -c /sys/fs/cgroup -p 1024 -s /var/lib/one/datastores/113 -v 82
Tue Mar 19 13:44:30 2024 [Z0][VMM][I]: Running command /var/tmp/one/vmm/firecracker/map_context /var/lib/one/datastores/113/82/disk.1 /var/lib/one/datastores/113/82/disk.1
Tue Mar 19 13:44:30 2024 [Z0][VMM][D]: deploy: Creating VM: 'screen -dmS one-82 sudo -n jailer --id one-82 --node 0 --exec-file /usr/bin/firecracker --uid 9869 --gid 9869 -- --config-file deployment.file'
Tue Mar 19 13:44:30 2024 [Z0][VMM][I]: ExitCode: 0
Tue Mar 19 13:44:30 2024 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Tue Mar 19 13:44:30 2024 [Z0][VMM][I]: ExitCode: 0
Tue Mar 19 13:44:30 2024 [Z0][VMM][I]: Successfully execute network driver operation: post.
Tue Mar 19 13:44:30 2024 [Z0][VM][I]: New LCM state is RUNNING
Tue Mar 19 13:46:38 2024 [Z0][LCM][I]: VM running but monitor state is POWEROFF
Tue Mar 19 13:46:38 2024 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Tue Mar 19 13:46:38 2024 [Z0][VM][I]: New state is POWEROFF
Tue Mar 19 13:46:38 2024 [Z0][VM][I]: New LCM state is LCM_INIT

Expected results:

VM should start and stay running.

Do you get the same error with other images or is it just Debian 12 ? Check also the firecracker troubleshooting section.

I get the same issue with all the images I’ve tried. Debian 12. Ubuntu 22.04. Alpine 3.18. So I don’t think it’s the image. And all these images work fine under KVM.

I’ve also tried the same setup with a vultr bare metal machine to see if it might be the nested virtualization. I used minione to set up opennebula, installed the firecracker packages since the server is running both opennebula servers and acting as a host and I’m able to deploy a firecracker VM which just shuts down after about half a minute to a minute. Same results as my local development using the nested VMs.