[Solved - Workaround] 5.0.2 fs_lvm - can't create virtual machines

Hi

I’m evaluating OpenNebula and trying to use fs_lvm to store System images. I have followed the instructions at http://docs.opennebula.org/5.0/deployment/open_cloud_storage_setup/lvm_drivers.html# and I have two datastores.

Datastore 102 (System):

ID             : 102
NAME           : lvm_system
USER           : oneadmin
GROUP          : oneadmin
CLUSTERS       : 0
TYPE           : SYSTEM
DS_MAD         : -
TM_MAD         : fs_lvm
BASE PATH      : /var/lib/one//datastores/102
DISK_TYPE      : FILE
STATE          : READY
DATASTORE CAPACITY
TOTAL:         : 300G
FREE:          : 300G
USED:          : 0M
LIMIT:         : -
PERMISSIONS
OWNER          : um-
GROUP          : u--
OTHER          : ---
DATASTORE TEMPLATE
DISK_TYPE="FILE"
DS_MIGRATE="YES"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
SHARED="YES"
TM_MAD="fs_lvm"
TYPE="SYSTEM_DS"

Datastore 103 (Image):

ID             : 103
NAME           : production
USER           : oneadmin
GROUP          : oneadmin
CLUSTERS       : 0
TYPE           : IMAGE
DS_MAD         : fs
TM_MAD         : fs_lvm
BASE PATH      : /var/lib/one//datastores/103
DISK_TYPE      : BLOCK
STATE          : READY
DATASTORE CAPACITY
TOTAL:         : 50G
FREE:          : 25.2G
USED:          : 24.8G
LIMIT:         : -
PERMISSIONS
OWNER          : um-
GROUP          : u--
OTHER          : ---
DATASTORE TEMPLATE
CLONE_TARGET="SYSTEM"
DISK_TYPE="BLOCK"
DS_MAD="fs"
LN_TARGET="SYSTEM"
SAFE_DIRS="/var/tmp /tmp"
TM_MAD="fs_lvm"
TYPE="IMAGE_DS"

When I try to create a virtual machine I get this error:

Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85 Command execution fail: /var/lib/one/remotes/tm/fs_lvm/ln OPENNEBULA:/var/lib/one//datastores/103/427efeaa5a86a4c676877757036b61d0 XXXXXXXXX:/var/lib/one//datastores/10$
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG E 85 ln: Command "    set -e
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85 mkdir -p /var/lib/one/datastores/102/85
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85 sudo lvcreate -L603M -n lv-one-85-0 vg-one-102
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85 qemu-img convert -O raw "/var/lib/one/datastores/103/427efeaa5a86a4c676877757036b61d0" "/dev/vg-one-102/lv-one-85-0"
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85 rm -f "/var/lib/one/datastores/102/85/disk.0"
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85 ln -s "/dev/vg-one-102/lv-one-85-0" "/var/lib/one/datastores/102/85/disk.0"" failed: qemu-img: Could not open '/var/lib/one/datastores/103/427efeaa5a86a4c676877757036b61d0$
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85 qemu-img: Could not open '/var/lib/one/datastores/103/427efeaa5a86a4c676877757036b61d0'
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG E 85 Error cloning /var/lib/one/datastores/103/427efeaa5a86a4c676877757036b61d0 to lv-one-85-0
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: LOG I 85 ExitCode: 1
Tue Aug  9 08:28:39 2016 [Z0][TM][D]: Message received: TRANSFER FAILURE 85 Error cloning /var/lib/one/datastores/103/427efeaa5a86a4c676877757036b61d0 to lv-one-85-0

This would appear to be caused by the compute node not having anything on the /var/lib/one/datastores/103 folder. The files are visible on the front-end but not on the compute nodes.

I tried to use NFS to share the folder to the compute nodes but I am still not able to create a virtual machine despite the files being visible.

Tue Aug  9 09:23:09 2016 [Z0][VM][I]: New state is ACTIVE
Tue Aug  9 09:23:09 2016 [Z0][VM][I]: New LCM state is PROLOG
Tue Aug  9 09:23:09 2016 [Z0][VM][I]: New LCM state is BOOT
Tue Aug  9 09:23:09 2016 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/89/deployment.0
Tue Aug  9 09:23:10 2016 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Tue Aug  9 09:23:10 2016 [Z0][VMM][I]: ExitCode: 0
Tue Aug  9 09:23:10 2016 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Tue Aug  9 09:23:10 2016 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/102/89/deployment.0' 'XXXXXXXXX' 89 XXXXXXXXX
Tue Aug  9 09:23:10 2016 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/102/89/deployment.0
Tue Aug  9 09:23:10 2016 [Z0][VMM][I]: error: Cannot access storage file '/var/lib/one//datastores/102/89/disk.0' (as uid:9869, gid:9869): No such file or directory
Tue Aug  9 09:23:10 2016 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/102/89/deployment.0
Tue Aug  9 09:23:10 2016 [Z0][VMM][I]: ExitCode: 255
Tue Aug  9 09:23:10 2016 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Tue Aug  9 09:23:10 2016 [Z0][VMM][E]: Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/102/89/deployment.0
Tue Aug  9 09:23:10 2016 [Z0][VM][I]: New LCM state is BOOT_FAILURE

OpenNebula created the correct LVS but it would seem OpenNebula doesn’t symlink them to the correct files and thus the creation fails.

[root@XXXXXXX ~]# lvdisplay vg-one-102
  --- Logical volume ---
  LV Path                /dev/vg-one-102/lv-one-89-0
  LV Name                lv-one-89-0
  VG Name                vg-one-102
  LV UUID                Ojd5nY-8DQe-b69F-BIIA-Zm8X-Nxz7-Q84amJ
  LV Write Access        read/write
  LV Creation host, time XXXXXXX , 2016-08-09 19:23:04 +0300
  LV Status              available
  # open                 0
  LV Size                19.53 GiB
  Current LE             5000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:11

Contents of datastore:

[root@NXXXXXXXXX ~]# ls -la /var/lib/one/datastores/102/89/
total 368
drwxrwxr-x 2 oneadmin oneadmin     38 Aug  9 19:23 .
drwxrwxr-x 3 oneadmin oneadmin     15 Aug  9 19:23 ..
-rw-rw-r-- 1 oneadmin oneadmin   1078 Aug  9 19:23 deployment.0
-rw-r--r-- 1 oneadmin oneadmin 372736 Aug  9 19:23 disk.1

Any ideas or should I try to use some other datastore formats? We have high-end SAN available which I would like to use with OpenNebula to store both images and system data.

Thank you.

1 Like

Alright. I couldn’t fix it so instead I created clvm cluster and use it as shared datastore for all the nodes.

1 Like