Can't create VMs

Community Support
1

ENV

  • OS: Ubuntu 20.04 Server
  • Ver: [5.12.0.3]

Vm Error Log:

Fri May 13 06:56:29 2022 [Z0][VM][I]: New state is ACTIVE
Fri May 13 06:56:29 2022 [Z0][VM][I]: New LCM state is PROLOG
Fri May 13 06:56:31 2022 [Z0][VM][I]: New LCM state is BOOT
Fri May 13 06:56:31 2022 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/355/deployment.0
Fri May 13 06:56:33 2022 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Fri May 13 06:56:34 2022 [Z0][VMM][I]: ExitCode: 0
Fri May 13 06:56:34 2022 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Fri May 13 06:56:37 2022 [Z0][VMM][I]: Command execution failed (exit code: 255): /var/lib/one/remotes/vmm/kvm/deploy-tweaks '/var/lib/one/vms/355/deployment.0' 'localhost' 355 localhost
Fri May 13 06:56:37 2022 [Z0][VMM][E]: [deploy-tweaks] Could not create domain from /var/lib/one/vms/355/deployment.0
Fri May 13 06:56:37 2022 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/0/355/deployment.0
Fri May 13 06:56:37 2022 [Z0][VMM][I]: error: internal error: process exited while connecting to monitor: 2022-05-13T06:56:37.079799Z qemu-system-x86_64: -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-105-one-355/master-key.aes: Unable to read /var/lib/libvirt/qemu/domain-105-one-355/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-105-one-355/master-key.aes”: Permission denied
Fri May 13 06:56:37 2022 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/0/355/deployment.0
Fri May 13 06:56:37 2022 [Z0][VMM][I]: ExitCode: 0
Fri May 13 06:56:37 2022 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Fri May 13 06:56:37 2022 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Fri May 13 06:56:37 2022 [Z0][VMM][E]: Error deploying virtual machine: [deploy-tweaks] Could not create domain from /var/lib/one/vms/355/deployment.0
Fri May 13 06:56:37 2022 [Z0][VM][I]: New LCM state is BOOT_FAILURE

oned.log

Fri May 13 06:56:34 2022 [Z0][ReM][D]: Req:9568 UID:0 IP:127.0.0.1 one.vm.info invoked , 355, false
Fri May 13 06:56:34 2022 [Z0][ReM][D]: Req:9568 UID:0 one.vm.info result SUCCESS, "<VM><ID>355</ID><UID..."
Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: LOG I 355 Command execution failed (exit code: 255): /var/lib/one/remotes/vmm/kvm/deploy-tweaks '/var/lib/one/vms/355/deployment.0' 'localhost' 355 localhost

Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: LOG E 355 [deploy-tweaks] Could not create domain from /var/lib/one/vms/355/deployment.0

Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: LOG I 355 error: Failed to create domain from /var/lib/one//datastores/0/355/deployment.0

Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: LOG I 355 error: internal error: process exited while connecting to monitor: 2022-05-13T06:56:37.079799Z qemu-system-x86_64: -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-105-one-355/master-key.aes: Unable to read /var/lib/libvirt/qemu/domain-105-one-355/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-105-one-355/master-key.aes”: Permission denied

Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: LOG E 355 Could not create domain from /var/lib/one//datastores/0/355/deployment.0

Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: LOG I 355 ExitCode: 0

Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: LOG I 355 Successfully execute network driver operation: clean.

Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: LOG I 355 Failed to execute virtualization driver operation: deploy.

Fri May 13 06:56:37 2022 [Z0][VMM][D]: Message received: DEPLOY FAILURE 355 [deploy-tweaks] Could not create domain from /var/lib/one/vms/355/deployment.0

Fri May 13 06:56:38 2022 [Z0][InM][E]: Error parsing VM_STATE: syntax error, unexpected VARIABLE, expecting EQUAL or EQUAL_EMPTY at line 0, columns 10:12

other

When i cli run:

/var/lib/one/remotes/vmm/kvm/deploy-tweaks ‘/var/lib/one/vms/355/deployment.0’ ‘localhost’ 355 localhost

Response:

root@sdpserver: /var/lib/one/remotes/vmm/kvm/deploy-tweaks '/var/lib/one/vms/355/deployment.0' 'localhost' 355 localhost
/var/lib/one/remotes/vmm/kvm/deploy-tweaks: line 67: remotes/datastore/xpath.rb: No such file or directory
ERROR MESSAGE --8<------
[deploy-tweaks] Can't get SYSTEM datastore ID for one-355
ERROR MESSAGE ------>8--
2

Hello Shuai,

You should resolve the following error. It looks like a permission error, but you should dig dipper why is this happening:

Fri May 13 06:56:37 2022 [Z0][VMM][I]: error: internal error: process exited while connecting to monitor: 2022-05-13T06:56:37.079799Z qemu-system-x86_64: -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-105-one-355/master-key.aes: Unable to read /var/lib/libvirt/qemu/domain-105-one-355/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-105-one-355/master-key.aes”: Permission denied

Also, as a rule of thumb - You must never run a script owned by oneadmin as root!

Best Regards,
Anton Todorov

3

thx.
i check this error:

Fri May 13 06:56:37 2022 [Z0][VMM][I]: error: internal error: process exited while connecting to monitor: 2022-05-13T06:56:37.079799Z qemu-system-x86_64: -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-105-one-355/master-key.aes: Unable to read /var/lib/libvirt/qemu/domain-105-one-355/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-105-one-355/master-key.aes”: Permission denied

root@sdpserver:~# cd /var/lib/libvirt/qemu/domain-105-one-355/
-bash: cd: /var/lib/libvirt/qemu/domain-105-one-355/: No such file or directory

so, domain-105-one-355 directory not exist.

and i run CLI with oneadmin user:

oneadmin@sdpserver:/var/lib/libvirt/qemu$ /var/lib/one/remotes/vmm/kvm/deploy-tweaks ‘/var/lib/one/vms/355/deployment.0’ ‘localhost’ 355 localhost
cp: cannot stat ‘’$‘\342\200\230’‘/var/lib/one/vms/355/deployment.0’$‘\342\200\231’: Permission denied
/var/lib/one/remotes/vmm/kvm/deploy-tweaks: line 67: remotes/datastore/xpath.rb: Permission denied
ERROR MESSAGE --8<------
[deploy-tweaks] Can’t get SYSTEM datastore ID for one-355
ERROR MESSAGE ------>8–

4

I think that this is the root cause of your issues

the deploy-tweaks script is intended to by opennebula. to run it manually you should chdir to the oneadmin home first. The following should work:

oneadmin@frontend$ cd /var/lib/one
oneadmin@frontend$ remotes/vmm/kvm/deploy-tweaks /var/lib/one/vms/355/deployment.0 localhost 355 localhost

But most probably you’ll get the same error…

Hope this helps,
Anton Todorov

5

yes, error again:

root@sdpserver:~# su oneadmin
oneadmin@sdpserver:/root$ cd /var/lib/one/
oneadmin@sdpserver:~$ remotes/vmm/kvm/deploy-tweaks /var/lib/one/vms/355/deployment.0 localhost 355 localhost
ERROR MESSAGE --8<------
[deploy-tweaks] Could not create domain from /var/lib/one/vms/355/deployment.0
ERROR MESSAGE ------>8--
error: Failed to create domain from /var/lib/one//datastores/0/355/deployment.0
error: internal error: process exited while connecting to monitor: 2022-05-18T06:08:04.505198Z qemu-system-x86_64: -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-106-one-355/master-key.aes: Unable to read /var/lib/libvirt/qemu/domain-106-one-355/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-106-one-355/master-key.aes”: Permission denied
ERROR MESSAGE --8<------
Could not create domain from /var/lib/one//datastores/0/355/deployment.0
ERROR MESSAGE ------>8--

i change qemu directory permission:

root@sdpserver:/var/lib/libvirt# chmod 755 -R qemu

and run again:

oneadmin@sdpserver:~$ remotes/vmm/kvm/deploy-tweaks /var/lib/one/vms/355/deployment.0 localhost 355 localhost
one-355

is working !!!

thanks!!!