Firecracker microVM fails with "Permission denied"

Community Support
1

Hi,

I’m running some test with OpenNebula and Firecracker. I have configured a ONE-6.4.0 with a Firecracker node. Then, I have downloaded “bash dockerhub” image, a Firecracker Kernel and I have created a virtual network where I will attach some instances.
When I instantiate my Firecracker microVM, process fails (after PROLOG state) and terminate with these messages:

Wed Nov 9 13:17:52 2022 [Z0][IPM][D]: Message received: DEPLOY FAILURE 45 Running command sudo -n /usr/sbin/one-clean-firecracker-domain -c /sys/fs/cgroup -v one-45 -t 60 Running command sudo -n /usr/sbin/one-prepare-firecracker-domain -c /sys/fs/cgroup -p 1024 -s /var/lib/one/datastores/0 -v 45 /usr/sbin/one-prepare-firecracker-domain: línea 74: /sys/fs/cgroup/cpu/firecracker/one-45/cpu.shares: Permiso denegado There was an error deploying the microVM. Check oned.log. Running command sudo -n /usr/sbin/one-clean-firecracker-domain -c /sys/fs/cgroup -v one-45 -t 60 -o ExitCode: 255

I have checked “sudo” configuration in my Firecracker node. All seems correct (and, also, I have rewrite /etc/sudoers.d/opennebula-node-firecracker as following:
oneadmin ALL=(ALL:ALL) NOPASSWD: ONE_FIRECRACKER, ONE_NET, ONE_OVS, ONE_LXC, ONE_LXD
because in /etc/sudoers.d/opennebula line “Cmnd_Alias ONE_FIRECRACKER” doesn’t include /bin/mount and other tools. Also, I have seen that last line is commented, so I have added “Cmnd_Alias”.

However, when microVM start, all seems OK in firecracker-node, but process finaly fails with “Permission Denied”.

What can I do?

Thanks.

2

Hi @Daniel_Ruiz_Molina,

Is it possible that you’re using cgroups v2? I think only cgroups v1 is supported by the Firecracker drivers, so maybe that’s causing the error.

3

Hi @cgonzalez,

Yes, Debian 11 runs by default Cgroup v2. I have added “systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller” to GRUB_CMDLINE_LINUX in /etc/default/grub, rewrited /boot/grub/grub.cfg, but now system boots with Cgroup v1 but Firecracker VMs keeps failing.

The error messages is this:

Mon Nov 14 11:31:37 2022 [Z0][IPM][D]: Message received: DEPLOY FAILURE 54 Running command sudo -n /usr/sbin/one-clean-firecracker-domain -c /sys/fs/cgroup -v one-54 -t 60 Running command sudo -n /usr/sbin/one-prepare-firecracker-domain -c /sys/fs/cgroup -p 1024 -s /var/lib/one/datastores/0 -v 54 Running command /var/tmp/one/vmm/firecracker/map_context /var/lib/one/datastores/0/54/disk.1 /var/lib/one/datastores/0/54/disk.1 DEBUG: deploy: Creating VM: ‘screen -dmS one-54 sudo -n jailer --id one-54 --node 0 --exec-file /usr/bin/firecracker --uid 9869 --gid 9869 – --config-file deployment.file’ MicroVM process did not start. Running command sudo -n /usr/sbin/one-clean-firecracker-domain -c /sys/fs/cgroup -v one-54 -t 60 -o ExitCode: 255

Thanks.

4

Hi @Daniel_Ruiz_Molina,

Can you see any error at: /var/lib/one/datastores/<sys_ds>/<vm_id>/logs.fifo [1]? Also, one of the common mistakes is trying to deploy a MicroVM with a NIC attached without properly configuring the networking drivers, note that Firecracker requires some extra configuration steps [2].

[1] Firecracker Driver — OpenNebula 6.4.2 documentation
[2] Firecracker Driver — OpenNebula 6.4.2 documentation

5

Hi @cgonzalez

1000 thousands thanks!! I had forgotten to run [2] about network post-configuration with Firecracker VMs. Now, after copying “pre” and “clean” scripts, I’m running 5 Firecracker VMs with no problem.

Thanks again!

1 Like
6

With 1.9, when building on linux for windows i could’nt cache with go build -i because of permission denied on /usr/local/go/pkg_window…, the build for a very small app was 4s, now it’s 0.150s !

7

I think only cgroups v1 is supported by the Firecracker drivers, so maybe that’s causing the error.netgear login

pikashow apk