Launching ephemeral VMs fails with permission denied on 5.12.0.3

Hello,

I’ve been investigating a weird issue for the last couple of days and can’t really figure it out: I’m on Debian 10, and ONE 5.12.0.3.

When trying to spin up an ephemeral VM, launch fails with this error in ONE:

Sun Jan 17 11:34:30 2021 : Error deploying virtual machine: Could not create domain from /var/lib/one/datastores/100/89/deployment.0

oned.log has some more details:

Sun Jan 17 11:34:30 2021 [Z0][VMM][D]: Message received: LOG I 89 Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy '/var/lib/one/datastores/100/89/deployment.0' 'MYHOSTNAME' 89 MYHOSTNAME
Sun Jan 17 11:34:30 2021 [Z0][VMM][D]: Message received: LOG I 89 error: Failed to create domain from /var/lib/one/datastores/100/89/deployment.0
Sun Jan 17 11:34:30 2021 [Z0][VMM][D]: Message received: LOG I 89 error: internal error: process exited while connecting to monitor: 2021-01-17T11:34:30.046811Z qemu-system-x86_64: -drive file=/var/lib/one/datastores/100/89/disk.0,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback: Could not open backing file: Could not open '/var/lib/one/datastores/103/65ad126e4ee862a2f85c00fe466b515b': Permission denied
Sun Jan 17 11:34:30 2021 [Z0][VMM][D]: Message received: LOG E 89 Could not create domain from /var/lib/one/datastores/100/89/deployment.0

I can create a new persistent VM from the same template without issues. Also, this happens with existing templates and new ones freshly downloaded from the Marketplace.

Permissions don’t seem to have anything wrong: qemu runs as oneadmin, and this user is able to access /var/lib/one/datastores/103/65ad126e4ee862a2f85c00fe466b515b.

Topics on this forum about similar issues suggested to:

  • disable AppArmor -> I don’t have it
  • check /etc/libvirt/qemu.conf is correct -> it is

Any help or hint would be appreciated. Happy to share more details in case I’ve missed something.

Thanks
Giorgio

Hi, it can be some issue with system datastore. When you are running vm with persistent image, then image datastore is used. Can you share your datastores templates?

Not sure I know what a “datastore template” is?

Here are my datastores anyway:

ID,USER,GROUP,NAME,SIZE,AVAIL,CLUSTERS,IMAGES,TYPE,DS,TM,STAT
103,oneadmin,oneadmin,templates,25G,74%,100,8,img,fs,qcow2,on
102,oneadmin,oneadmin,tier2-images,591.5G,24%,100,5,img,fs,qcow2,on
101,oneadmin,oneadmin,tier1-images,97.9G,68%,100,7,img,fs,qcow2,on
100,oneadmin,oneadmin,system,25G,100%,100,0,sys,-,qcow2,on

Note the error seems to be on “103”. Permissions seem to be the same across all of them:

drwxr-x---  7 oneadmin oneadmin 4.0K Dec  2  2019 .
drwxr-x--- 12 oneadmin oneadmin 4.0K Jan 17 16:14 ..
drwxr-xr-x 10 oneadmin oneadmin   10 Jan 17 11:53 100
drwxr-xr-x 10 oneadmin oneadmin   17 Jan 16 15:28 101
drwxr-xr-x  8 oneadmin oneadmin   13 Jan 16 15:32 102
drwxr-xr-x  3 oneadmin oneadmin   11 Jan 17 11:54 103
drwxr-xr-x  2 oneadmin oneadmin 4.0K Jan 17 16:24 .isofiles

And mounts:

tank-tier1/images on /var/lib/one/datastores/101 type zfs (rw,xattr,noacl)
tank-tier1/templates on /var/lib/one/datastores/103 type zfs (rw,xattr,noacl)
tank-tier1/system on /var/lib/one/datastores/100 type zfs (rw,xattr,noacl)
tank-tier2/images on /var/lib/one/datastores/102 type zfs (rw,xattr,noacl)

Thanks!
Giorgio

What is output of:

  • ls -la /var/lib/one/datastores/103/65ad126e4ee862a2f85c00fe466b515b
  • qemu-img info /var/lib/one/datastores/100/89/disk.0