11 error: can't canonicalize path '/var/lib/one//datastores/0/11/disk.1': Permission denied

I’m wondering what coudl be the root cause of this message. My VM deployments are failing with the following:

Wed Mar 30 02:32:07 2016 [Z0][ImM][D]: Datastore files (2) successfully monitored.
Wed Mar 30 02:32:11 2016 [Z0][TM][D]: Message received: TRANSFER SUCCESS 11 -

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 ExitCode: 0

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 Successfully execute network driver operation: pre.

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy ‘/var/lib/one//datastores/0/11/deployment.0’ ‘mdskvm-p01’ 11 mdskvm-p01

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 error: Failed to create domain from /var/lib/one//datastores/0/11/deployment.0

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 error: can’t canonicalize path ‘/var/lib/one//datastores/0/11/disk.1’: Permission denied

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG E 11 Could not create domain from /var/lib/one//datastores/0/11/deployment.0

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 ExitCode: 255

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 Failed to execute virtualization driver operation: deploy.

Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: DEPLOY FAILURE 11 Could not create domain from /var/lib/one//datastores/0/11/deployment.0

Wed Mar 30 02:32:17 2016 [Z0][ImM][D]: Datastore system (0) successfully monitored.
Wed Mar 30 02:32:17 2016 [Z0][InM][D]: Host mdskvm-p01 (2) successfully monitored.

My mount NFS is fine and I managed to deploy earlier but recently changed network settings:

[oneadmin@opennebula01 one]$ cat /etc/exports
/var/lib/one/ *(rw,sync,no_subtree_check,root_squash)
[oneadmin@opennebula01 one]$ showmount -e
Export list for opennebula01:
/var/lib/one *
[oneadmin@opennebula01 one]$ ssh mdskvm-p01
Warning: Permanently added ‘mdskvm-p01,192.168.0.60’ (ECDSA) to the list of known hosts.
Last login: Wed Mar 30 01:33:00 2016
[oneadmin@mdskvm-p01 ~]$
[oneadmin@mdskvm-p01 ~]$
[oneadmin@mdskvm-p01 ~]$
[oneadmin@mdskvm-p01 ~]$

[root@mdskvm-p01 ~]# mount|grep one
192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70)
[root@mdskvm-p01 ~]#

Cheers,
Tom

I’ve solved this by updating the oneadmin password in the DB.