Problems Creating LXD Containers from Sunstone

Community Support Network
1

Problem:

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/deploy '/var/lib/one//datastores/0/15/deployment.0' 'north-america' 15 north-america

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Overriding container

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Using raw filesystem mapper for /var/lib/one/datastores/0/15/disk.0

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Mapping disk at /var/lib/lxd/storage-pools/default/containers/one-15/rootfs using device /dev/loop0

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Resizing filesystem ext4 on /dev/loop0

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Mounting /dev/loop0 at /var/lib/lxd/storage-pools/default/containers/one-15/rootfs

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG E 15 deploy: mount_on?: Mount detected in /var/lib/one/datastores/0/15/mapper/disk.1

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Using raw filesystem mapper for /var/lib/one/datastores/0/15/disk.0

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Unmapping disk at /var/lib/lxd/storage-pools/default/containers/one-15/rootfs

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Umounting disk mapped at /dev/loop0

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG I 15 deploy: Unmapping disk at /var/lib/one/datastores/0/15/mapper/disk.1

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG E 15 deploy: Failed to detect block device from /var/lib/one/datastores/0/15/mapper/disk.1

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG E 15 deploy: failed to dismantle container storage

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG I 15 /var/tmp/one/vmm/lxd/client.rb:145:in `initialize': undefined method `[]' for #<LXDError:0x000000021f20a0> (NoMethodError)

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG I 15 from /var/tmp/one/vmm/lxd/deploy:86:in `exception'

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG I 15 from /var/tmp/one/vmm/lxd/deploy:86:in `raise'

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG I 15 from /var/tmp/one/vmm/lxd/deploy:86:in `rescue in <main>'

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG I 15 from /var/tmp/one/vmm/lxd/deploy:74:in `<main>'

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG I 15 ExitCode: 1

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: LOG I 15 Failed to execute virtualization driver operation: deploy.

Mon Mar 11 00:26:29 2019 [Z0][VMM][D]: Message received: DEPLOY FAILURE 15 -

I’ve downloaded ubuntu_bionic from Linux Containers data stores, added my image to a VM template like so:

CONTEXT = [
  NETWORK = "YES",
  SSH_PUBLIC_KEY = "$USER[SSH_PUBLIC_KEY]" ]
CPU = "1"
DISK = [
  IMAGE = "ubuntu_bionic - LXD",
  IMAGE_UNAME = "oneadmin" ]
GRAPHICS = [
  LISTEN = "0.0.0.0",
  TYPE = "VNC" ]
HYPERVISOR = "lxd"
LXD_SECURITY_NESTING = "no"
LXD_SECURITY_PRIVILEGED = "no"
MEMORY = "1024"
MEMORY_UNIT_COST = "MB"
OS = [
  BOOT = "" ]
VCPU = "1"

Container always gets to “BOOT_FAILURE”. I can create new lxd containers manually (lxc launch ubuntu:18), its just within ON/sunstone that I can’t.

2

Hello,

Could you try to start another VM?

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG E 15 deploy: mount_on?: Mount detected in /var/lib/one/datastores/0/15/mapper/disk.1

This log entry shows you already tried to deploy the container and it failed, or, if you have the previous log history you can show it.

3

I mitigated the issue by using a KVM image instead, that seems to work no problems. I can post log of a brand new container later. It looks the same, specifically this part:

initialize': undefined method[]’ for #<LXDError:0x000000021f20a0> (NoMethodError)`

4

That line is an exception error already being dealt with.

5

Great to know! I’ll add more info later so you can look into it, in case it’s something else.

6

I’m also experiencing this. In both of our logs, it is disk.1 causing issues. That is the contextualization iso.

I’m beginning to wonder if nfs may have something to do with it. @JPaulMora Are you running NFS as your system datastore?

7

Sun Mar 10 23:20:06 2019 [Z0][VMM][D]: Message received: LOG E 15 deploy: mount_on?: Mount detected in /var/lib/one/datastores/0/15/mapper/disk.1

This lines tells that prior to the context disk mount in the driver flow, it had been mounted in that directory, so, the container had previously failed somehow and the mount wasn’t dealt with

8

The automated tests are run to check the driver correct operation are also run on NFS environments and so far so good. There are a few patches for it in the code, regarding the mapper side of the drivers when dealing with symbolic links

9

@Brady_Deetz At the moment of the bug I was not using NFS. It seems the bug has been found (and fixed). Read about it here: https://github.com/OpenNebula/one/issues/3098. Now just wait to 5.8.1 release or apply the patch yourself if you’re brave enough.