"Please Create a Storage Pool"

Community Support
1

Hi everyone, I’m attempting to provision an LXD container on OpenNebula 6.0.1 running on Debian 10. I have created three datastores, one for images, one for files and one for system. When I attempt to create the container it enters the BOOT_FAILED state. Within the log there is a message: “No storage pool found. Please create a new storage pool.”

Prior to this error message, the following command fails: “cat << EOT | /var/tmp/one/vmm/lxd/deploy”. This command seems to be malformed to me, but I’m not sure how it’s supposed to look.

As far as I can tell I have added the three datastores to my single cluster. I don’t see any options in the interface to create a storage pool, nor have I found any documentation regarding it. How would I remedy this?

2

Seems you haven’t initialized LXD at all yet. Install the opennebula-node-lxd package on the host where you intend to use LXD, and please, make sure you are using the version 3 of LXD

root@ubuntu1804-lxd-marketplace-6-0-2-6395a-0:~# lxd --version
3.0.3
3

Hi dann1!

Thanks so much for your response. I really appreciate you taking the time to help me out.

At the time I received this error message, I had the opennebula-node-lxd package installed. Additionally, I have LXD version 3.0.4 installed on my node.

I’ve since done some additional work. I’ve found that I am able to correctly deploy the “Virtual Service Router” which runs on top of LXD. However, I am still unable to deploy any virtual machines utilizing the same technology.

When you say initialize, do you mean I should run “lxd init”? I had done so and walked through the setup process. This allowed me to create containers via the LXC command on the node’s CLI, however it made no difference to Open Nebula.

I’ve found that both /snap/bin/lxd and /snap/bin/lxc function normally. However, /snap/lxd/current/bin/lxd gives me an error.

/snap/lxd/current/bin/lxd: error while loading shared libraries: libdqlite.so.0: cannot open shared object file: No such file or directory

It appears that this specific executable is the one used by OpenNebula to provision its containers. Here’s the log for one of my failed containers:

Sun May 16 12:43:58 2021 [Z0][VM][I]: New state is ACTIVE
Sun May 16 12:43:58 2021 [Z0][VM][I]: New LCM state is PROLOG
Sun May 16 12:44:13 2021 [Z0][VM][I]: New LCM state is BOOT
Sun May 16 12:44:13 2021 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/26/deployment.0
Sun May 16 12:44:13 2021 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Sun May 16 12:44:13 2021 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/deploy '/var/lib/one//datastores/100/26/deployment.0' 'LXDHost' 26 LXDHost
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Processing disk 0
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Using raw filesystem mapper for /var/lib/one/datastores/100/26/disk.0
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Mapping disk at /var/snap/lxd/common/lxd/storage-pools/default/containers/one-26/rootfs using device /dev/loop3
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Mounting /dev/loop3 at /var/snap/lxd/common/lxd/storage-pools/default/containers/one-26/rootfs
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Mapping disk at /var/lib/one/datastores/100/26/mapper/disk.1 using device /dev/loop4
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Mounting /dev/loop4 at /var/lib/one/datastores/100/26/mapper/disk.1
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: --- Starting container ---
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Name: one-26
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Remote: unix://
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Architecture: x86_64
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Created: 2021/05/16 17:44 UTC
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Status: Stopped
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Type: persistent
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Profiles: default
Sun May 16 12:44:27 2021 [Z0][VMM][I]:
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Log:
Sun May 16 12:44:27 2021 [Z0][VMM][I]:
Sun May 16 12:44:27 2021 [Z0][VMM][I]: lxc one-26 20210516174419.234 ERROR start - start.c:proc_pidfd_open:1607 - Function not implemented - Failed to send signal through pidfd
Sun May 16 12:44:27 2021 [Z0][VMM][I]: lxc one-26 20210516174419.305 ERROR start - start.c:start:2104 - No such file or directory - Failed to exec "/sbin/init"
Sun May 16 12:44:27 2021 [Z0][VMM][I]: lxc one-26 20210516174419.305 ERROR sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 7)
Sun May 16 12:44:27 2021 [Z0][VMM][I]: lxc one-26 20210516174419.305 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:851 - Received container state "ABORTING" instead of "RUNNING"
Sun May 16 12:44:27 2021 [Z0][VMM][I]: lxc one-26 20210516174419.305 ERROR start - start.c:__lxc_start:2019 - Failed to spawn container "one-26"
Sun May 16 12:44:27 2021 [Z0][VMM][I]: lxc 20210516174419.347 WARN commands - commands.c:lxc_cmd_rsp_recv:135 - Connection reset by peer - Failed to receive response for command "get_state"
Sun May 16 12:44:27 2021 [Z0][VMM][I]:
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Processing disk 0
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Using raw filesystem mapper for /var/lib/one/datastores/100/26/disk.0
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Unmapping disk at /var/snap/lxd/common/lxd/storage-pools/default/containers/one-26/rootfs
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Umounting disk mapped at /dev/loop3
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Unmapping disk at /var/lib/one/datastores/100/26/mapper/disk.1
Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Umounting disk mapped at /dev/loop4
Sun May 16 12:44:27 2021 [Z0][VMM][I]: /var/tmp/one/vmm/lxd/client.rb:102:in `wait': {"type"=>"sync", "status"=>"Success", "status_code"=>200, "operation"=>"", "error_code"=>0, "error"=>"", "metadata"=>{"id"=>"f04cd21a-97b5-4fba-b15e-72576ebe361a", "class"=>"task", "description"=>"Starting container", "created_at"=>"2021-05-16T12:44:17.851505347-05:00", "updated_at"=>"2021-05-16T12:44:17.851505347-05:00", "status"=>"Failure", "status_code"=>400, "resources"=>{"containers"=>["/1.0/containers/one-26"]}, "metadata"=>nil, "may_cancel"=>false, "err"=>"Failed to run: /snap/lxd/current/bin/lxd forkstart one-26 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/one-26/lxc.conf: ", "location"=>"none"}} (LXDError)
Sun May 16 12:44:27 2021 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:517:in `wait?'
Sun May 16 12:44:27 2021 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:529:in `change_state'
Sun May 16 12:44:27 2021 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:211:in `start'
Sun May 16 12:44:27 2021 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/deploy:70:in `<main>'
Sun May 16 12:44:27 2021 [Z0][VMM][I]: ExitCode: 1
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Sun May 16 12:44:27 2021 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Sun May 16 12:44:27 2021 [Z0][VMM][E]: Error deploying virtual machine
Sun May 16 12:44:27 2021 [Z0][VM][I]: New LCM state is BOOT_FAILURE

Is there a way to switch what LXD executable is used to provision containers? I know LXD is functioning correctly on the system, I’m just unsure of how I should fix the executable Open Nebula uses. I looked up libdqlite.so.0 and found this post. I’ve ensured that I’m running on the 3.0/stable channel but to no avail. Would a symlink potentially work?

4

Looking at the VM log, when starting the container, lxd cannot find the /sbin/init executable inside the container rootfs. Check your container disk.0 image content, and confirm whether /sbin/init is really there or not. You should be able to do so by mounting the container image file.

Sun May 16 12:44:27 2021 [Z0][VMM][I]: deploy: Using raw filesystem mapper for
/var/lib/one/datastores/100/26/disk.0

mount /var/lib/one/datastores/100/26/disk.0 /mnt

There should be a full Linux FS inside /mnt.

/snap/lxd/current/bin/lxd: error while loading shared libraries: libdqlite.so.0: cannot open shared object file: No such file or directory

Regarding this. The whole purpose of using snap packages is for avoiding this kind of issues. The lxd snap is supposed to provide every single system conflict-free dependency required for lxd.

Is there a way to switch what LXD executable is used to provision containers?

For the most part the LXD Driver directly uses the REST API exposed at /var/snap/lxd/common/lxd/unix.socket. There are a few instances where the executable is used. In your case, the deploy action only runs the lxc command when opening the VNC connection.

By default this is the executables used on a LXD node.

root@ubuntu2004-lxd-ssh-6-0-2-0836e-2:~# snap list
Name  Version  Rev    Tracking       Publisher   Notes
core  16-2.50  11081  latest/stable  canonical✓  core
lxd   3.0.4    11348  3.0/stable     canonical✓  -
root@ubuntu2004-lxd-ssh-6-0-2-0836e-2:~# which lxd
/snap/bin/lxd
root@ubuntu2004-lxd-ssh-6-0-2-0836e-2:~# which lxc
/usr/bin/lxc
5

I went ahead and mounted the virtual hard disks. You’re right about them missing /sbin/init. In fact, they’re completely empty. I tried to download several images from the Linux Containers market place as well as “Nginx - LXD” from “Turnkey Linux Containers” and they all come out empty. They take up the proper amount of space, but upon mounting the image, there’s no contents at all to note.

Could this be due to an aborted download? If so, does OpenNebula log the requests it makes anywhere? Using TCPDump I can see some traffic moving over port 443 when I download a new image from the marketplace. Obviously, it’s encrypted so I don’t get much further than that.

6

Look for clues in /var/log/one/oned.log. The image import process is basically:

  • create the raw image
  • dump the container image tar.gz contents inside the image
  • auto contextualization logic

You might want to debug this script which should be at /var/lib/one/remotes/datastore/lxd_downloader.sh

7

Hey dann1,

I wanted to thank you for all of your help. I’ve fixed the problem. It turns out that the uuidgen binary was missing from my system. I’m now able to create LXD containers. I really appreciate you taking the time to help me out with this. Have a great day!

1 Like