ERROR: clone: Command failed: No such file or directory

Hey All,

Having this issue with instantiating VM’s. I’ve removed UserKnownHostsFile /dev/null which solved the key issue however this issue about copying still persists:

[oneadmin@opennebula01 one]$ cat 20.log
Thu Mar 31 01:16:00 2016 [Z0][DiM][I]: New VM state is ACTIVE.
Thu Mar 31 01:16:00 2016 [Z0][LCM][I]: New VM state is PROLOG.
Thu Mar 31 01:16:00 2016 [Z0][TM][I]: Command execution fail: /var/lib/one/remotes/tm/shared/clone opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 mdskvm-p01:/var/lib/one//datastores/0/20/disk.0 20 1
Thu Mar 31 01:16:00 2016 [Z0][TM][I]: clone: Cloning /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 in mdskvm-p01:/var/lib/one//datastores/0/20/disk.0
Thu Mar 31 01:16:00 2016 [Z0][TM][E]: clone: Command “cd /var/lib/one/datastores/0/20; cp /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 /var/lib/one/datastores/0/20/disk.0” failed: cp: cannot stat ‘/var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7’: No such file or directory
Thu Mar 31 01:16:00 2016 [Z0][TM][E]: Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/20/disk.0
Thu Mar 31 01:16:00 2016 [Z0][TM][I]: ExitCode: 1
Thu Mar 31 01:16:00 2016 [Z0][TM][E]: Error executing image transfer script: Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/20/disk.0
Thu Mar 31 01:16:00 2016 [Z0][DiM][I]: New VM state is FAILED
[oneadmin@opennebula01 one]$

[oneadmin@opennebula01 ~]$ /var/lib/one/remotes/tm/shared/clone opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 mdskvm-p01:/var/lib/one//datastores/0/20/disk.0 20 1
INFO: clone: Cloning /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 in mdskvm-p01:/var/lib/one//datastores/0/20/disk.0
ERROR: clone: Command “cd /var/lib/one/datastores/0/20; cp /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 /var/lib/one/datastores/0/20/disk.0” failed: cp: cannot stat ‘/var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7’: No such file or directory
ERROR MESSAGE --8<------
Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/20/disk.0
ERROR MESSAGE ------>8–
[oneadmin@opennebula01 ~]$

[oneadmin@opennebula01 one]$ ls -altri /var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7
ls: cannot access /var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7: No such file or directory
[oneadmin@opennebula01 one]$ ls -altri /var/lib/one//datastores/0/20/
total 0
68474908 drwxrwxr-x 6 oneadmin oneadmin 42 Mar 31 01:16 …
34700810 drwxrwxr-x 2 oneadmin oneadmin 6 Mar 31 01:16 .
[oneadmin@opennebula01 one]$

NFS is properly mounted and selinux is disabled on all servers:

[oneadmin@mdskvm-p01 ~]$ ls -altri /var/lib/one/datastores/1
total 0
101870807 drwxrwxr-x 2 oneadmin oneadmin 6 Mar 30 18:58 .
68474907 drwxrwxr-x 5 oneadmin oneadmin 30 Mar 30 18:58 …
[oneadmin@mdskvm-p01 ~]$ mount|grep one
192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70)
[oneadmin@mdskvm-p01 ~]$

And accessible via the controller:

[oneadmin@opennebula01 .ssh]$ ssh mdskvm-p01
Last login: Thu Mar 31 01:29:25 2016
[oneadmin@mdskvm-p01 ~]$

Cheers,
TK

The third datastore is ssh and after some reason, seems having it ssh will work but is there a way to try to pick another datastore while instantiating? Sorry, still new to this:

[oneadmin@opennebula01 datastores]$ onedatastore list -x|grep -E "ID|BASE|TM_MAD"
0
0
0
<TM_MAD></TM_MAD>
<BASE_PATH></BASE_PATH>
<CLUSTER_ID>-1</CLUSTER_ID>
<BASE_PATH></BASE_PATH>
<TM_MAD></TM_MAD>
1
0
0
<TM_MAD></TM_MAD>
<BASE_PATH></BASE_PATH>
<CLUSTER_ID>-1</CLUSTER_ID>
1
<BASE_PATH></BASE_PATH>
<TM_MAD></TM_MAD>
2
0
0
<TM_MAD></TM_MAD>
<BASE_PATH></BASE_PATH>
<CLUSTER_ID>-1</CLUSTER_ID>
<BASE_PATH></BASE_PATH>
<TM_MAD></TM_MAD>
[oneadmin@opennebula01 datastores]$

This is a default trial install of OpenNebula.

Cheers,
TK

Tried with ssh instead of shared, still didn’t work:

Thu Mar 31 02:01:00 2016 [Z0][TM][D]: Message received: LOG I 22 Command execution fail: /var/lib/one/remotes/tm/ssh/clone opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 mdskvm-p01:/var/lib/one//datastores/0/22/disk.0 22 1

Thu Mar 31 02:01:00 2016 [Z0][TM][D]: Message received: LOG I 22 clone: Cloning opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 in /var/lib/one/datastores/0/22/disk.0

Thu Mar 31 02:01:00 2016 [Z0][TM][D]: Message received: LOG E 22 clone: Command “scp -r opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 mdskvm-p01:/var/lib/one//datastores/0/22/disk.0” failed: /var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7: No such file or directory

Thu Mar 31 02:01:00 2016 [Z0][TM][D]: Message received: LOG E 22 Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/22/disk.0

Thu Mar 31 02:01:00 2016 [Z0][TM][D]: Message received: LOG I 22 ExitCode: 1

Thu Mar 31 02:01:00 2016 [Z0][TM][D]: Message received: TRANSFER FAILURE 22 Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/22/disk.0