Problem creating KVM hosts on former libvirt/virsh working station

Hello everyone,

I’m trying to add previously hosted VMs from libvirt (with virsh cli in particular).
I installed OpenNebula on top of the running system (Ubuntu 20.04 Server LTS).

Since I had still some work to do on the running VMs I changed
user = "root" group = "root" in /etc/libvirt/qemu.confand thought I could change it later with ease once I’m done.

Now, I installed the KVM Node according to the documentation, but they won’t run. For about a minute they are stuck at INIT state in the sunstone WebUI and exit with an error.
From /var/log/one/oned.log I find the following notes.:

oned.log
Fri Nov 20 14:39:10 2020 [Z0][ONE][E]: Error monitoring Host frontend (6):
Fri Nov 20 14:39:10 2020 [Z0][ONE][E]: Error monitoring Host node (7):
Fri Nov 20 14:39:11 2020 [Z0][ReM][D]: Req:6224 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:39:11 2020 [Z0][ReM][D]: Req:6224 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:39:11 2020 [Z0][ReM][D]: Req:3504 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:39:11 2020 [Z0][ReM][D]: Req:3504 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:39:11 2020 [Z0][ReM][D]: Req:7072 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:39:11 2020 [Z0][ReM][D]: Req:7072 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:39:41 2020 [Z0][ReM][D]: Req:1264 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:39:41 2020 [Z0][ReM][D]: Req:1264 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:39:41 2020 [Z0][ReM][D]: Req:6720 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:39:41 2020 [Z0][ReM][D]: Req:6720 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:39:41 2020 [Z0][ReM][D]: Req:9248 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:39:41 2020 [Z0][ReM][D]: Req:9248 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:39:43 2020 [Z0][AuM][D]: Message received: AUTHENTICATE SUCCESS 9 -

Fri Nov 20 14:39:43 2020 [Z0][ReM][D]: Req:7072 UID:0 IP:127.0.0.1 one.hostpool.info invoked
Fri Nov 20 14:39:43 2020 [Z0][ReM][D]: Req:7072 UID:0 one.hostpool.info result SUCCESS, "<HOST_POOL><HOST><ID..."
Fri Nov 20 14:39:43 2020 [Z0][ReM][D]: Req:9248 UID:0 IP:127.0.0.1 one.user.info invoked , 0, false
Fri Nov 20 14:39:43 2020 [Z0][ReM][D]: Req:9248 UID:0 one.user.info result SUCCESS, "<USER><ID>0</ID><GID..."
Fri Nov 20 14:40:04 2020 [Z0][DBM][I]: Purging obsolete LogDB records: 0 records purged. Log state: 0,0 - 0,0
Fri Nov 20 14:40:04 2020 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 0.
Fri Nov 20 14:40:11 2020 [Z0][ReM][D]: Req:32 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:40:11 2020 [Z0][ReM][D]: Req:32 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:40:11 2020 [Z0][ReM][D]: Req:5072 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:40:11 2020 [Z0][ReM][D]: Req:5072 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:40:11 2020 [Z0][ReM][D]: Req:3808 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:40:11 2020 [Z0][ReM][D]: Req:3808 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:40:41 2020 [Z0][ReM][D]: Req:2304 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:40:41 2020 [Z0][ReM][D]: Req:2304 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:40:41 2020 [Z0][ReM][D]: Req:8224 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:40:41 2020 [Z0][ReM][D]: Req:8224 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:40:41 2020 [Z0][ReM][D]: Req:7664 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:40:41 2020 [Z0][ReM][D]: Req:7664 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:41:04 2020 [Z0][DBM][I]: Purging obsolete LogDB records: 0 records purged. Log state: 0,0 - 0,0
Fri Nov 20 14:41:04 2020 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 0.
Fri Nov 20 14:41:11 2020 [Z0][ReM][D]: Req:4144 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:41:11 2020 [Z0][ReM][D]: Req:4144 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:41:11 2020 [Z0][ReM][D]: Req:1232 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:41:11 2020 [Z0][ReM][D]: Req:1232 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:41:11 2020 [Z0][ReM][D]: Req:2976 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:41:11 2020 [Z0][ReM][D]: Req:2976 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:41:41 2020 [Z0][ReM][D]: Req:688 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:41:41 2020 [Z0][ReM][D]: Req:688 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:41:41 2020 [Z0][ReM][D]: Req:8768 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:41:41 2020 [Z0][ReM][D]: Req:8768 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:41:41 2020 [Z0][ReM][D]: Req:3344 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:41:41 2020 [Z0][ReM][D]: Req:3344 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:42:04 2020 [Z0][DBM][I]: Purging obsolete LogDB records: 0 records purged. Log state: 0,0 - 0,0
Fri Nov 20 14:42:04 2020 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 0.
Fri Nov 20 14:42:11 2020 [Z0][ReM][D]: Req:2912 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:42:11 2020 [Z0][ReM][D]: Req:2912 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:42:11 2020 [Z0][ReM][D]: Req:2848 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:42:11 2020 [Z0][ReM][D]: Req:2848 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:42:11 2020 [Z0][ReM][D]: Req:7296 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:42:11 2020 [Z0][ReM][D]: Req:7296 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:42:41 2020 [Z0][ReM][D]: Req:3552 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:42:41 2020 [Z0][ReM][D]: Req:3552 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:42:41 2020 [Z0][ReM][D]: Req:5504 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:42:41 2020 [Z0][ReM][D]: Req:5504 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:42:41 2020 [Z0][ReM][D]: Req:5936 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:42:41 2020 [Z0][ReM][D]: Req:5936 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:43:04 2020 [Z0][DBM][I]: Purging obsolete LogDB records: 0 records purged. Log state: 0,0 - 0,0
Fri Nov 20 14:43:04 2020 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 0.
Fri Nov 20 14:43:11 2020 [Z0][ReM][D]: Req:4944 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:43:11 2020 [Z0][ReM][D]: Req:4944 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:43:11 2020 [Z0][ReM][D]: Req:5376 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:43:11 2020 [Z0][ReM][D]: Req:5376 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:43:11 2020 [Z0][ReM][D]: Req:5184 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:43:11 2020 [Z0][ReM][D]: Req:5184 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:43:14 2020 [Z0][InM][D]: Monitoring datastore default (1)
Fri Nov 20 14:43:14 2020 [Z0][InM][D]: Monitoring datastore files (2)
Fri Nov 20 14:43:15 2020 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Fri Nov 20 14:43:15 2020 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Fri Nov 20 14:43:41 2020 [Z0][ReM][D]: Req:4928 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Nov 20 14:43:41 2020 [Z0][ReM][D]: Req:4928 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>-1<..."
Fri Nov 20 14:43:41 2020 [Z0][ReM][D]: Req:8096 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:43:41 2020 [Z0][ReM][D]: Req:8096 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:43:41 2020 [Z0][ReM][D]: Req:8016 UID:0 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1
Fri Nov 20 14:43:41 2020 [Z0][ReM][D]: Req:8016 UID:0 one.vmpool.infoextended result SUCCESS, "<VM_POOL></VM_POOL>"
Fri Nov 20 14:44:04 2020 [Z0][DBM][I]: Purging obsolete LogDB records: 0 records purged. Log state: 0,0 - 0,0
Fri Nov 20 14:44:04 2020 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 0.

I tried to change the user and group attribute back to oneadmin, but without success.

ssh-keyscan frontend node >> /var/lib/one/.ssh/known_hosts seems to cannot find the host name.

ssh-keyscan

getaddrinfo frontend: Name or service not known
getaddrinfo frontend: Name or service not known
getaddrinfo frontend: Name or service not known
getaddrinfo frontend: Name or service not known
getaddrinfo frontend: Name or service not known
getaddrinfo node: Name or service not known
getaddrinfo node: Name or service not known
getaddrinfo node: Name or service not known
getaddrinfo node: Name or service not known
getaddrinfo node: Name or service not known

I have no idea what is causing this issue and I’m lost. Adding new hosts just repeats the same issue. Any suggestion how to solve this issue?