Good day everyone,
We’ve been trying to evaluate OpenNebula for our small business and have come to encounter many issues. I would like to address these and ask if anyone could give us advice, tips and fixes for any.
First and foremost, we want to set up our server infrastructure with our own hardware - right now, we have three machines:
A colocated bare-metal and VPS in Frankfurt and a KVM in USE with the following hardware:
AMD EPYC 7352, 128GB DDR4-3200RDIMMs (Tegu),
2 Core 10GB RAM 40GB SSD (SDNS),
3 Core 8GB RAM 80GB SSD (USE-GOFB).
Tegu is supposed to be our frontend and host our needs for cloud storage, matrix server, web & dns server, processing and internal servers as well as hosting prototypes. Will also host autorative DNS server.
SDNS is supposed to be a secondary DNS server, I thought of hosting Sunstone on it as well.
USE-GOFB is a geo-offloading server that will also act as the ternary DNS server.
Issue #1 - LXD and Datastores.
Now, the first hurdle we ran into was deploying OpenNebula. We had several issues, but we solved these. However, we want to use LXD Containers - the problem is that every time we try to use the LXC Hypervisor, we get a driver error:
DEPLOY: INFO: deploy: No block device on /var/lib/one/datastores/0/3/mapper/disk.0 No block device on /var/lib/one/datastores/0/3/mapper/disk.1 lxc-create: one-3: conf.c: chown_mapped_root: 3095 Error chowning /var/lib/lxc-one/3/disk.0 lxc-create: one-3: lxccontainer.c: do_storage_create: 1295 Error chowning "/var/lib/lxc-one/3/disk.0" to container root lxc-create: one-3: conf.c: suggest_default_idmap: 4699 You must either run as root, or define uid mappings lxc-create: one-3: conf.c: suggest_default_idmap: 4700 To pass uid mappings to lxc-create, you could create lxc-create: one-3: conf.c: suggest_default_idmap: 4701 ~/.config/lxc/default.conf: lxc-create: one-3: conf.c: suggest_default_idmap: 4702 lxc.include = /etc/lxc/default.conf lxc-create: one-3: conf.c: suggest_default_idmap: 4703 lxc.idmap = u 0 600100001 65537 lxc-create: one-3: conf.c: suggest_default_idmap: 4704 lxc.idmap = g 0 600100001 65537 lxc-create: one-3: lxccontainer.c: do_lxcapi_create: 1877 Failed to create (none) storage for one-3 lxc-create: one-3: tools/lxc_create.c: main: 331 Failed to create container one-3 There was an error creating the containter. ExitCode: 255
Now, here’s a little background on our datastores:
We are hosting our front-end on a 256GB NVMe. We then have 2x512GB NVMes, a 1TB SSD, 2x4TB Hybrid-Drives, 2x2TB Hybrid-Drives and 4x3TB (R5) HDDs in Hardware. Because of that, we had to set up Datastores with Symlinks - and I think this might cause this.
Because OpenNebula actually doesn’t tell you how to add other drives in the server anywhere - probably because it assumes that you will have an LVM partition that makes all the drives into one partition for any node - we had to do it with symlinks. We don’t want to put all data in one LVM partition because there are some cases where we have I/O intensive applications, and others where we don’t.
I still don’t know how we did it, but that’s because I had to offload that to a different sysadmin that figured out how - I’m not that good with servers at all, however, I own the business - so I built the server, deployed it, and did the best to my abilities to help set it up, since it was around holidays.
I did verify and attempt to reinstall packages needed multiple times. We are using RockyLinux 8.5 - Basically RHEL.
Issue #2 - Un-Official Docker-Images
I tried to create an image for:
matrixdotorg/synapse, seafileltd/seafile-mc and hestiacp (a fork of VestaCP that just looks nicer).
Synapse and Seafile-MC didn’t work, HestiaCP did. Here’s the error I’m getting trying to install it to any of our datastores:
Wed Dec 29 12:23:43 2021:
Error copying image in the datastore: INFO: cp: Copying local image docker://matrixdotorg/synapse?size=2048&filesystem=xfs&format=raw&tag=latest to the image repository
ERROR: cp: Command “set -e -o pipefail; /var/lib/one/remotes/datastore/fs/…/downloader.sh ‘docker://matrixdotorg/synapse?size=2048&filesystem=xfs&format=raw&tag=latest’ ‘/var/lib/one//datastores/104/ed2809cf23a37186ef160458c7108dbf’” failed:
jq: error (at :118): Cannot iterate over null (null) Error copying Error copying docker://matrixdotorg/synapse?size=2048&filesystem=xfs&format=raw&tag=latest to /var/lib/one//datastores/104/ed2809cf23a37186ef160458c7108dbf
I get this error through CLI and Sunstone. I tried many times with different file formats, partition types - nothing really helped. Note that while attempting to install Docker, we were using the opennebula-node-kvm package. Kubernetes also failed to install, I don’t have the error anymore.
Issue #2.5 - IDs
Due to so many attempts that failed, we’re pushing up the image IDs and VM IDs - any way to clear an ID after it has been deleted? Its a bit offsetting having 0,1,17,37,42,44,56 with no IDs in between.
Because these are our major problems right now, I’ll leave it be like this.
Thank you for anyone to have read this and thank you in advance for any advice on how to solve these issues!
All the best,