Failure to deploy images from qcow2 datastore to qcow2 system datastores

Hello, I’ve deployed ONE 5.6.1 in a PoC with 1 frontend, 3 workers and 1 NFS server, all using Ubuntu 16.04 (latest)

  • NFS server (export file):
    /var/lib/one/datastores 192.168.200.0/24(rw,sync,no_subtree_check,no_root_squash)

  • Frontend and workers mount it as this (fstab file):
    poc4:/var/lib/one/datastores /var/lib/one/datastores nfs defaults 0 0

  • Datastores:
    oneadmin@onm:~$ onedatastore list
    ID NAME SIZE AVAIL CLUSTERS IMAGE
    2 files 836.2G 99% 0
    118 IMAGES 836.2G 99% 0
    122 SYSTEM 836.2G 99% 0
    123 IMAGES-SSH 836.2G 99% 0

    oneadmin@onm:~$ onedatastore show 118
    DATASTORE 118 INFORMATION
    ID : 118
    NAME : IMAGES
    USER : oneadmin
    GROUP : oneadmin
    CLUSTERS : 0
    TYPE : IMAGE
    DS_MAD : fs
    TM_MAD : qcow2
    BASE PATH : /var/lib/one//datastores/118
    DISK_TYPE : FILE
    STATE : READY

    DATASTORE CAPACITY
    TOTAL: : 836.2G
    FREE: : 829.6G
    USED: : 6.6G
    LIMIT: : -

    PERMISSIONS
    OWNER : um-
    GROUP : u–
    OTHER : —

    DATASTORE TEMPLATE
    ALLOW_ORPHANS=“NO”
    CLONE_TARGET=“SYSTEM”
    DISK_TYPE=“FILE”
    DRIVER=“qcow2”
    DS_MAD=“fs”
    LN_TARGET=“NONE”
    RESTRICTED_DIRS="/"
    SAFE_DIRS="/var/tmp"
    TM_MAD=“qcow2”
    TYPE=“IMAGE_DS”

    IMAGES
    oneadmin@onm:~$ onedatastore show 122
    DATASTORE 122 INFORMATION
    ID : 122
    NAME : SYSTEM
    USER : oneadmin
    GROUP : oneadmin
    CLUSTERS : 0
    TYPE : SYSTEM
    DS_MAD : -
    TM_MAD : qcow2
    BASE PATH : /var/lib/one//datastores/122
    DISK_TYPE : FILE
    STATE : READY

    DATASTORE CAPACITY
    TOTAL: : 836.2G
    FREE: : 829.6G
    USED: : 6.6G
    LIMIT: : -

    PERMISSIONS
    OWNER : um-
    GROUP : u–
    OTHER : —

    DATASTORE TEMPLATE
    ALLOW_ORPHANS=“NO”
    DISK_TYPE=“FILE”
    DS_MIGRATE=“YES”
    RESTRICTED_DIRS="/"
    SAFE_DIRS="/var/tmp"
    SHARED=“YES”
    TM_MAD=“qcow2”
    TYPE=“SYSTEM_DS”

    IMAGES
    oneadmin@onm:~$ onedatastore show 123
    DATASTORE 123 INFORMATION
    ID : 123
    NAME : IMAGES-SSH
    USER : oneadmin
    GROUP : oneadmin
    CLUSTERS : 0
    TYPE : IMAGE
    DS_MAD : fs
    TM_MAD : ssh
    BASE PATH : /var/lib/one//datastores/123
    DISK_TYPE : FILE
    STATE : READY

    DATASTORE CAPACITY
    TOTAL: : 836.2G
    FREE: : 829.6G
    USED: : 6.6G
    LIMIT: : -

    PERMISSIONS
    OWNER : um-
    GROUP : u–
    OTHER : —

    DATASTORE TEMPLATE
    ALLOW_ORPHANS=“NO”
    CLONE_TARGET=“SYSTEM”
    DISK_TYPE=“FILE”
    DS_MAD=“fs”
    LN_TARGET=“SYSTEM”
    RESTRICTED_DIRS="/"
    SAFE_DIRS="/var/tmp"
    TM_MAD=“ssh”
    TYPE=“IMAGE_DS”
    IMAGES
    61
    71


When I try to deploy a VM from DS 118 (TM_MAD=qcow2) I get this error:
Wed Oct 31 17:06:34 2018 [Z0][VMM][D]: Message received: LOG I 751 Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy ‘/var/lib/one//datastores/122/751/deployment.0’ ‘poc3’ 751 poc3

Wed Oct 31 17:06:34 2018 [Z0][VMM][D]: Message received: LOG I 751 error: Failed to create domain from /var/lib/one//datastores/122/751/deployment.0

Wed Oct 31 17:06:34 2018 [Z0][VMM][D]: Message received: LOG I 751 error: internal error: process exited while connecting to monitor: 2018-10-31T16:06:34.208427Z qemu-system-x86_64: -drive file=/var/lib/one//datastores/122/751/disk.0,format=qcow2,if=none,id=drive-ide0-0-0,cache=none: Could not open backing file: Could not open '/var/lib/one/datastores/118/8b7445c6f42731319d2933c3a8939acc': Permission denied

Wed Oct 31 17:06:34 2018 [Z0][VMM][D]: Message received: LOG E 751 Could not create domain from /var/lib/one//datastores/122/751/deployment.0

Wed Oct 31 17:06:34 2018 [Z0][VMM][D]: Message received: LOG I 751 ExitCode: 255

Wed Oct 31 17:06:34 2018 [Z0][VMM][D]: Message received: LOG I 751 Failed to execute virtualization driver operation: deploy.

Wed Oct 31 17:06:34 2018 [Z0][VMM][D]: Message received: DEPLOY FAILURE 751 Could not create domain from /var/lib/one//datastores/122/751/deployment.0

What I see in the deployed hosts:
davidp@poc3:/var/lib/one/datastores/122/751$ ls -l
total 368
-rw-rw-r-- 1 oneadmin oneadmin 1128 Oct 31 17:06 deployment.0
lrwxrwxrwx 1 oneadmin oneadmin 13 Oct 31 17:06 disk.0 -> disk.0.snap/0
drwxrwxr-x 2 oneadmin oneadmin 34 Oct 31 17:06 disk.0.snap
-rw-r–r-- 1 oneadmin oneadmin 372736 Oct 31 17:06 disk.1
davidp@poc3:/var/lib/one/datastores/122/751$ ls -l disk.0.snap/
total 196
-rw-r–r-- 1 oneadmin oneadmin 197120 Oct 31 17:06 0
lrwxrwxrwx 1 oneadmin oneadmin 1 Oct 31 17:06 disk.0.snap -> .
davidp@poc3:/var/lib/one/datastores/122/751$

When I deploy the same image from DS 123 (TM_MAD=ssh) it works.

Any hint about the source of this issue?
Thanks in advance.

Please take a look to this thread Could not open backing file it was the same problem as you. You have to check all the permissions, maybe libvirt has no access to different folder that are needed.

Hello Alejandro, I’m looking for some kind of “official solution” for v5.6.1 as it support Ubuntu 16.04, removing apparmor is not described as a requirement in the deployment guide.
Thanks in any way.

P.D. Verified that with user oneadmin I can rw files at /var/lib/one/datastores and below.

This is not an OpenNebula problem, it seems that the apparmor is not well configured. Please check all the configuration is correct and the permissions to access all the files are correct too.

Hello, just sharing the solution:
Effectively as highlighted by Alejandro the problem was related to permission but not the kind ones you can solve with chmod, it was related to apparmor policies.
After a deeper understanding of what apparmor does I created as root at all nodes (frontend doesn’t need it) a file in “/etc/apparmor.d/tunables/home.d/opennebula” with this content:
davidp@poc1:~$ cat /etc/apparmor.d/tunables/home.d/opennebula
@{HOMEDIRS}+=/var/lib/one/datastores

Followed by “# service apparmor restart”

After that the problems is gone :slight_smile:

As reported in this topic it would be helpful to include a call to apparmor stuff in the deployment guide for mounted datastores.
Thanks everybody!

1 Like

Hi @davidp, i’m late but thank you for sharing your solution. I was in a similar position: apparmor was blocking the creation of the file required to start the vms. Setting this parameter as you suggests fixed all my issues.

This happens only when you try to use two datastores shared via two separate nfs shares, if you use the whole /var/lib/one/datastores in a single nfs share, as suggested even in some post in this forum, everythong works as expected.

It would be nice some hint in the documentation like the part that suggest to disable SELinux for CentOS, just my two cents.

Thank you, that solved my issue!