A practical example of installing OpenNebula with Firecracker by using miniONE

mehrdad-yousefi · May 26, 2023, 6:37pm

Hi,

I’m using OpenNebula installed by miniONE. First, I tried to use it by using QEMU/KVM hypervisor and it works pretty fine. Now, I’m interested to use it with Firecracker/KVM hypervisor. Here it is how I installed it, since the latest miniONE script doesn’t support provisioning the Firecracker automatically since version 5.12 (which I’m wondering why you dropped supporting it… anyway):

wget 'https://github.com/OpenNebula/minione/releases/latest/download/minione' && bash minione --yes --password admin --ssh-pubkey /root/.ssh/id_rsa.pub --vm-password admin --bridge-interface br0 --nat-interface eth0 --vnet-address 192.168.0.0 --vnet-netmask 255.255.255.0 --vnet-gateway 192.168.0.7 --vnet-ar-ip-start 192.168.0.169 --vnet-ar-ip-count 10 --marketapp-name 'Ubuntu 22.04'

ip addr add 192.168.0.7/24 dev br0
ip link set br0 up

apt-get update
apt-get -y install opennebula-node-firecracker

onehost delete 0
onehost create 12.75.0.3 -i firecracker -v firecracker
onehost list

Note, that I’m installing OpenNebula inside a Docker container. Everything works fine for this installation (if there is a logical problem here, please point it out). Now, I have my root filesystems as a ext4 format file as well as my kernel compiled and have a vmlinux. Note that I’m already using Firecracker and it works fine by using its CLI. My purpose of trying to use OpenNebula is to have a dashboard that can manage VMs instead of using Firecracker’s CLI.

Now, my question is how would I export my ext4 root filesystems into OpenNebula? Is it going to be an image with OS or DATABLOCK type or it needs to be exported as a RAMDISK?

I used both image OS and DATABLOCK types and it starts the VM but it would be shutdown without showing an error. I know that Firecracker needs non-persistent images, and I’m pretty sure my images are non-persistent.

Also, when I create template and gives the path of my compiled kernel vmlinux file, it doesn’t show up in the template or KERNEL_DS is not there. So, it seems it needs to be added later on manually. Is it a known bug? Thanks for your help.

This is my template:

NAME="Test"
CPU="6"
MEMORY="4096"
DISK=[
  IMAGE="Ubuntu 22.04",
  IMAGE_UNAME="oneadmin" ]
GRAPHICS=[
  LISTEN="0.0.0.0",
  TYPE="VNC" ]
OS=[
  KERNEL_CMD="console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd",
  KERNEL_DS="$FILE[IMAGE=\"Kernel\"]" ]

And this is the output of logs.fifo:

Running Firecracker v0.24.2
2023-05-26T21:56:41.279399814 [one-2:main] Disk size 196648 is not a multiple of sector size 512; the remainder will not be visible to the guest.
2023-05-26T21:56:41.320645490 [one-2:main] Artificially kick devices.
2023-05-26T21:56:41.320673919 [one-2:main] kick block disk.0.
2023-05-26T21:56:41.320710025 [one-2:main] Successfully started microvm that was configured from one single json
2023-05-26T21:56:41.590438733 [one-2:main] block: activate event
2023-05-26T21:56:42.627868132 [one-2:main] Vmm is stopping.

dclavijo · May 30, 2023, 8:08pm

By default, minione deploys a qcow2 storage on image and system datastores. This is incompatible with firecracker. You need to user raw images.

Take a look at the following datastore and images templates for reference

root@minione-8926:~# onedatastore show 100
DATASTORE 100 INFORMATION
ID             : 100
NAME           : local_raw_images
USER           : oneadmin
GROUP          : oneadmin
CLUSTERS       : 0
TYPE           : IMAGE
DS_MAD         : fs
TM_MAD         : ssh
BASE PATH      : /var/lib/one//datastores/100
DISK_TYPE      : FILE
STATE          : READY

DATASTORE CAPACITY
TOTAL:         : 38.6G
FREE:          : 34.6G
USED:          : 4G
LIMIT:         : -

PERMISSIONS
OWNER          : um-
GROUP          : u--
OTHER          : ---

DATASTORE TEMPLATE
ALLOW_ORPHANS="YES"
CLONE_TARGET="SYSTEM"
DISK_TYPE="FILE"
DS_MAD="fs"
LN_TARGET="SYSTEM"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
TM_MAD="ssh"
TYPE="IMAGE_DS"

IMAGES
3
root@minione-8926:~# onedatastore show 102
DATASTORE 102 INFORMATION
ID             : 102
NAME           : local_raw_sys
USER           : oneadmin
GROUP          : oneadmin
CLUSTERS       : 0
TYPE           : SYSTEM
DS_MAD         : -
TM_MAD         : ssh
BASE PATH      : /var/lib/one//datastores/102
DISK_TYPE      : FILE
STATE          : READY

DATASTORE CAPACITY
TOTAL:         : -
FREE:          : -
USED:          : -
LIMIT:         : -

PERMISSIONS
OWNER          : um-
GROUP          : u--
OTHER          : ---

DATASTORE TEMPLATE
ALLOW_ORPHANS="YES"
DISK_TYPE="FILE"
DS_MIGRATE="YES"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
SHARED="NO"
TM_MAD="ssh"
TYPE="SYSTEM_DS"

IMAGES

and the image

root@minione-8926:~# oneimage show 3
IMAGE 3 INFORMATION
ID             : 3
NAME           : alpine
USER           : oneadmin
GROUP          : oneadmin
LOCK           : None
DATASTORE      : local_raw_images
TYPE           : OS
REGISTER TIME  : 05/30 19:49:44
PERSISTENT     : No
SOURCE         : /var/lib/one//datastores/100/65810155f9903627fbd0ec4f0027b462
PATH           : docker://alpine?size=2048&filesystem=ext4&format=raw
FORMAT         : raw
SIZE           : 2G
STATE          : used
RUNNING_VMS    : 1

PERMISSIONS
OWNER          : um-
GROUP          : ---
OTHER          : ---

IMAGE TEMPLATE
DEV_PREFIX="vd"
FROM_APP="136"
FROM_APP_MD5="5b9de9f2756d1df622e3cbef3dedb95c"
FROM_APP_NAME="alpine"

VIRTUAL MACHINES

  ID USER     GROUP    NAME                                                                        STAT  CPU     MEM HOST                                                     TIME
   1 oneadmin oneadmin local_raw_alpine-1                                                          runn    1    768M localhost                                            0d 00h05

And the resulting VMs

root@minione-8926:~# onevm list
  ID USER     GROUP    NAME                                                                        STAT  CPU     MEM HOST                                                     TIME
   1 oneadmin oneadmin local_raw_alpine-1                                                          runn    1    768M localhost                                            0d 00h06
   0 oneadmin oneadmin alpine_dh_fc-0                                                              poff    1    768M localhost                                            0d 00h18

VM 0 is deployed as you did and it results in the same error

root@minione-8926:~# cat /var/lib/one/datastores/0/0/logs.fifo
Running Firecracker v0.24.2
2023-05-30T19:40:42.437249538 [one-0:main] Disk size 196648 is not a multiple of sector size 512; the remainder will not be visible to the guest.
2023-05-30T19:40:42.486825520 [one-0:main] Artificially kick devices.
2023-05-30T19:40:42.486865428 [one-0:main] kick block disk.0.
2023-05-30T19:40:42.486874038 [one-0:main] kick block disk.1.
2023-05-30T19:40:42.499816746 [one-0:main] Successfully started microvm that was configured from one single json
2023-05-30T19:40:43.476946456 [one-0:main] block: activate event
2023-05-30T19:40:43.483353404 [one-0:main] block: activate event
2023-05-30T19:40:44.597997155 [one-0:main] Vmm is stopping.

in the case of VM 1 you get this log

root@minione-8926:~# cat /var/lib/one/datastores/102/1/logs.fifo
Running Firecracker v0.24.2
2023-05-30T19:53:04.176785019 [one-1:main] Disk size 2147483842 is not a multiple of sector size 512; the remainder will not be visible to the guest.
2023-05-30T19:53:04.234149224 [one-1:main] Artificially kick devices.
2023-05-30T19:53:04.234174253 [one-1:main] kick block disk.0.
2023-05-30T19:53:04.234178234 [one-1:main] kick block disk.1.
2023-05-30T19:53:04.234603368 [one-1:main] Successfully started microvm that was configured from one single json
2023-05-30T19:53:04.950343532 [one-1:main] block: activate event
2023-05-30T19:53:04.954820766 [one-1:main] block: activate event
2023-05-30T19:53:04.964602534 [one-1:fc_vcpu 0] Failed to trigger i8042 kbd interrupt (disabled by guest OS)
2023-05-30T19:53:04.981184930 [one-1:fc_vcpu 0] Failed to trigger i8042 kbd interrupt (disabled by guest OS)
2023-05-30T19:53:07.002527637 [one-1:fc_api] The API server received a Get request on "/machine-config".
2023-05-30T19:53:07.002561896 [one-1:fc_api] The request was executed successfully. Status code: 200 OK.
2023-05-30T19:53:07.002569096 [one-1:fc_api] Total previous API call duration: 46 us.
2023-05-30T19:53:07.002741299 [one-1:fc_api] The API server received a Get request on "/".
2023-05-30T19:53:07.002748559 [one-1:fc_api] Total previous API call duration: 9 us.
2023-05-30T19:53:12.165674659 [one-1:fc_api] The API server received a Get request on "/machine-config".
2023-05-30T19:53:12.165721197 [one-1:fc_api] The request was executed successfully. Status code: 200 OK.
2023-05-30T19:53:12.165728497 [one-1:fc_api] Total previous API call duration: 58 us.
2023-05-30T19:53:12.165935239 [one-1:fc_api] The API server received a Get request on "/".
2023-05-30T19:53:12.165946610 [one-1:fc_api] Total previous API call duration: 13 us.
2023-05-30T19:53:15.679059323 [one-1:fc_api] The API server received a Get request on "/machine-config".
2023-05-30T19:53:15.679129841 [one-1:fc_api] The request was executed successfully. Status code: 200 OK.
2023-05-30T19:53:15.679143860 [one-1:fc_api] Total previous API call duration: 91 us.
2023-05-30T19:53:15.679596384 [one-1:fc_api] The API server received a Get request on "/".
2023-05-30T19:53:15.679619842 [one-1:fc_api] Total previous API call duration: 27 us.
2023-05-30T19:53:17.319156745 [one-1:fc_api] The API server received a Get request on "/machine-config".
2023-05-30T19:53:17.319197533 [one-1:fc_api] The request was executed successfully. Status code: 200 OK.

and the VM attach process running with no issues

root@minione-8926:~# ps aux | grep -i screen
oneadmin   50255  0.0  0.1   8704  2324 ?        Ss   19:53   0:00 SCREEN -dmS one-1 sudo -n jailer --id one-1 --node 0 --exec-file /usr/bin/firecracker --uid 9869 --gid 9869 -- --config-file deployment.file
root       53015  0.0  0.1   8156  2564 pts/0    S+   20:01   0:00 grep --color=auto -i screen

As you can see, deploying minione with firecracker requires some extra steps due to the limitations and additional setup required. Enabling minione --firecracker means also supporting such deployment. Since the goal of minione is to quickly try out OpenNebula, we decided to drop the firecracker deployment mode a while ago.

This being said, if you want to quickly try out OpenNebula with firecracker, it would have come handy. If you think this is a good feature to have, please open a request on the minione repository. LXC support was added as well after FireCracker and LXD were dropped.

mehrdad-yousefi · May 31, 2023, 6:15pm

I’m able to resolve this issue by using OpenNebula 5.12, but I think it would nice to have it on the latest version as well.

Topic		Replies	Views
About Firecracker on miniONE Questions	3	894	July 27, 2020
I can't run Dockers with FireCracker Community Support	3	464	February 25, 2022
OpenNebula + Firecracker: Building the Future of On-Premises Serverless Computing Announcements	4	497	July 27, 2020
Unable to run linux container Community Support	2	427	May 11, 2022
Using native Docker containers in OpenNebula 6.0 Community Support	2	723	February 23, 2022

A practical example of installing OpenNebula with Firecracker by using miniONE

Related topics