I have set up a front end and a single node to run kvm and lxc vms to test. I downloaded a few images from the marketplace (both kvm and lxc), set up a simple bridged network and tried instantiating some vms. I have tried kvm and lxc, I tried with ssh after verifying that ssh w/o password works and I tried shared with NFS after verifying that I could create a file from the Node and delete it from the front end in /var/lib/one. In every case, the vm fails to start with the following error:
Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/deploy
‘/var/lib/one//datastores/0/26/deployment.0’ ‘lxd01’ 26 lxd01
Of course, the exact paths vary depending on lxd or kvm and the vm number. After that, a few more errors which likely stem from this one, but this is the first error.
Any help would be greatly appreciated.
Hello @doug,
Does the error shows more information apart from what you shared? you can check /var/log/one/oned.log and the VM en log (/var/log/one/<vm_id>)
Here is the VM log…
Sun Mar 8 20:17:11 2020 [Z0][VM][I]: New state is ACTIVE
Sun Mar 8 20:17:11 2020 [Z0][VM][I]: New LCM state is PROLOG
Sun Mar 8 20:20:27 2020 [Z0][VM][I]: New LCM state is BOOT
Sun Mar 8 20:20:27 2020 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/26/deployment.0
Sun Mar 8 20:20:31 2020 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Sun Mar 8 20:20:32 2020 [Z0][VMM][I]: pre: Executed "sudo ip link add name br005 type bridge ".
Sun Mar 8 20:20:32 2020 [Z0][VMM][I]: pre: Executed “sudo ip link set br005 up”.
Sun Mar 8 20:20:32 2020 [Z0][VMM][I]: pre: Executed "sudo ip link add link ens18 name ens18.5 mtu 1500 type vlan id 5 ".
Sun Mar 8 20:20:32 2020 [Z0][VMM][I]: pre: Executed “sudo ip link set ens18.5 up”.
Sun Mar 8 20:20:32 2020 [Z0][VMM][I]: pre: Executed “sudo ip link set ens18.5 master br005”.
Sun Mar 8 20:20:32 2020 [Z0][VMM][I]: ExitCode: 0
Sun Mar 8 20:20:32 2020 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/deploy ‘/var/lib/one//datastores/0/26/deployment.0’ ‘lxd01’ 26 lxd01
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: deploy: Processing disk 0
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: deploy: Using raw filesystem mapper for /var/lib/one/datastores/0/26/disk.0
Sun Mar 8 20:20:36 2020 [Z0][VMM][E]: deploy: do_map: losetup: /var/lib/one/datastores/0/26/disk.0: failed to set up loop device: Permission denied
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: deploy: Mapping disk at /var/snap/lxd/common/lxd/storage-pools/default/containers/one-26/rootfs using device
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: deploy: Processing disk 0
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: deploy: Using raw filesystem mapper for /var/lib/one/datastores/0/26/disk.0
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: deploy: Unmapping disk at /var/snap/lxd/common/lxd/storage-pools/default/containers/one-26/rootfs
Sun Mar 8 20:20:36 2020 [Z0][VMM][E]: deploy: Failed to detect block device from /var/snap/lxd/common/lxd/storage-pools/default/containers/one-26/rootfs
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: deploy: Unmapping disk at /var/lib/one/datastores/0/26/mapper/disk.1
Sun Mar 8 20:20:36 2020 [Z0][VMM][E]: deploy: Failed to detect block device from /var/lib/one/datastores/0/26/mapper/disk.1
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: /var/tmp/one/vmm/lxd/deploy:64:in `’: failed to setup container storage (RuntimeError)
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: ExitCode: 1
Sun Mar 8 20:20:36 2020 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Sun Mar 8 20:20:36 2020 [Z0][VMM][E]: Error deploying virtual machine
Sun Mar 8 20:20:36 2020 [Z0][VM][I]: New LCM state is BOOT_FAILURE
Thanks for helping!
Does anyone have any ideas to try? I would live to continue testing this platform but I’m at a loss here. Thanks,
Hello @doug, this seems to be the error, oneadmin needs to be able to map the disk file into a loopback device, try to manually map it and debug it. The command should be sudo losetup -f --show $dsrc replace $dsrc with the image file path, in this VM it would be /var/lib/one/datastores/0/26/disk.0
Thank you for the reply! I manually mapped disk.0 to the /dev/loop0. There were no reported errors and I was able to mount the loop device. It was an empty 1GB virtual disk. I was able to map it with oneadmin without sudo. Does the script run under a different user account?
The command should be run by oneadmin (or the user you select when installing from source) with sudo. However thanks for pointing it works without sudo, the drivers needs to point to /sbin/losetup because in some environments there is no /usr/sbin/losetup
oneadmin@ubuntu1804-lxd-nfs-5-11-80-2219a-0:~$ losetup
WARNING:root:could not open file '/etc/apt/sources.list'
Command 'losetup' is available in '/sbin/losetup'
The command could not be located because '/sbin' is not included in the PATH environment variable.
This is most likely caused by the lack of administrative privileges associated with your user account.
losetup: command not found
In your case, for some reason the user running the deploy script doesn’t have permissions to setup a loop device.
I believe the script should be running under oneadmin as I did not compile from source. I installed from opennebula repo using the opennebula guide. How can I verify which user is running the script? Also, I ran the previous command without sudo because the oneadmin password I use for sunstone didn’t work on the cli. So I set the oneadmin password from the root account and was able to sudo but it didn’t change anything. Could the permissions issue possibly be a result of the previous error?
Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/deploy '/var/lib/one//datastores/0/29/deployment.0
How can I verify which user is running the script?
The script is called deploy, you can filter processes with ps on the virtualization node and filter with the word “deploy”, run that while deploying the container.
You can manually run the deploy script by passing the arguments and the stdin it requires, the stdin is the VM XML, you can look for it in /var/log/one/oned.log in the frontend and try to reproduce it.