VM Migration between physical hosts

I’m trying to migrate a VM from one set of physical nodes, to another. The VM is sitting on a gluster volume (GV01) on host1. I’m trying to move it to the second gluster volume on GV02 on the second host cluster. The image attached illustrates this scenario.

How to do this properly? Do I need to mount GV01 on the second physical cluster then? Appears I do otherwise the migration logic doesn’t find the mounted GV01 from which to move the VM to GV02. Here’s what happens:

[oneadmin@one01 datastores]$
[oneadmin@one01 datastores]$ bash 0x /var/lib/one/remotes/tm/shared/mv mdskvm-p01.nix.mds.xyz:/var/lib/one//datastores/118/32 mdskvm-p05.nix.mds.xyz:/var/lib/one//datastores/123/32 32 123
bash: 0x: No such file or directory
[oneadmin@one01 datastores]$ bash -x /var/lib/one/remotes/tm/shared/mv mdskvm-p01.nix.mds.xyz:/var/lib/one//datastores/118/32 mdskvm-p05.nix.mds.xyz:/var/lib/one//datastores/123/32 32 123
+ SRC=mdskvm-p01.nix.mds.xyz:/var/lib/one//datastores/118/32
+ DST=mdskvm-p05.nix.mds.xyz:/var/lib/one//datastores/123/32
+ VMID=32
+ DSID=123
+ '[' -z '' ']'
+ TMCOMMON=/var/lib/one/remotes/tm/tm_common.sh
+ . /var/lib/one/remotes/tm/tm_common.sh
++ export LANG=C
++ LANG=C
++ '[' -z '' ']'
++ ONE_LOCAL_VAR=/var/lib/one
++ ONE_LIB=/usr/lib/one
++ DS_DIR=/var/lib/one/datastores
++ ONE_SH=/usr/lib/one/sh
++ . /usr/lib/one/sh/scripts_common.sh
+++ export LANG=C
+++ LANG=C
+++ export PATH=/bin:/sbin:/usr/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/var/lib/one/.local/bin:/var/lib/one/bin
+++ PATH=/bin:/sbin:/usr/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/var/lib/one/.local/bin:/var/lib/one/bin
+++ AWK=awk
+++ BASH=/bin/bash
+++ CUT=cut
+++ CEPH=ceph
+++ DATE=date
+++ DD=dd
+++ DF=df
+++ DU=du
+++ GREP=grep
+++ ISCSIADM=iscsiadm
+++ LVCREATE=lvcreate
+++ LVREMOVE=lvremove
+++ LVCHANGE=lvchange
+++ LVSCAN=lvscan
+++ LVEXTEND=lvextend
+++ LVS=lvs
+++ LN=ln
+++ MD5SUM=md5sum
+++ MKFS=mkfs
+++ MKISOFS=genisoimage
+++ MKSWAP=mkswap
+++ QEMU_IMG=qemu-img
+++ RADOS=rados
+++ RBD=rbd
+++ READLINK=readlink
+++ RM=rm
+++ CP=cp
+++ SCP=scp
+++ SED=sed
+++ SSH=ssh
+++ SUDO=sudo
+++ SYNC=sync
+++ TAR=tar
+++ TGTADM=tgtadm
+++ TGTADMIN=tgt-admin
+++ TGTSETUPLUN=tgt-setup-lun-one
+++ TR=tr
+++ VGDISPLAY=vgdisplay
+++ VMKFSTOOLS=vmkfstools
+++ WGET=wget
++++ uname -s
+++ '[' xLinux = xLinux ']'
+++ SED='sed -r'
++++ basename /var/lib/one/remotes/tm/shared/mv
+++ SCRIPT_NAME=mv
++ fix_dir_slashes mdskvm-p01.nix.mds.xyz:/var/lib/one//datastores/118/32
++ dirname mdskvm-p01.nix.mds.xyz:/var/lib/one//datastores/118/32/file
++ sed -r 's/\/+/\//g'
+ SRC=mdskvm-p01.nix.mds.xyz:/var/lib/one/datastores/118/32
++ fix_dir_slashes mdskvm-p05.nix.mds.xyz:/var/lib/one//datastores/123/32
++ dirname mdskvm-p05.nix.mds.xyz:/var/lib/one//datastores/123/32/file
++ sed -r 's/\/+/\//g'
+ DST=mdskvm-p05.nix.mds.xyz:/var/lib/one/datastores/123/32
++ arg_path mdskvm-p01.nix.mds.xyz:/var/lib/one/datastores/118/32
+++ echo mdskvm-p01.nix.mds.xyz:/var/lib/one/datastores/118/32
+++ sed -r 's/^[^:]*:(.*)$/\1/'
++ ARG_PATH=/var/lib/one/datastores/118/32
++ fix_dir_slashes /var/lib/one/datastores/118/32
++ dirname /var/lib/one/datastores/118/32/file
++ sed -r 's/\/+/\//g'
+ SRC_PATH=/var/lib/one/datastores/118/32
++ arg_path mdskvm-p05.nix.mds.xyz:/var/lib/one/datastores/123/32
+++ echo mdskvm-p05.nix.mds.xyz:/var/lib/one/datastores/123/32
+++ sed -r 's/^[^:]*:(.*)$/\1/'
++ ARG_PATH=/var/lib/one/datastores/123/32
++ fix_dir_slashes /var/lib/one/datastores/123/32
++ dirname /var/lib/one/datastores/123/32/file
++ sed -r 's/\/+/\//g'
+ DST_PATH=/var/lib/one/datastores/123/32
++ arg_host mdskvm-p01.nix.mds.xyz:/var/lib/one/datastores/118/32
++ echo mdskvm-p01.nix.mds.xyz:/var/lib/one/datastores/118/32
++ sed -r 's/^([^:]*):.*$/\1/'
+ SRC_HOST=mdskvm-p01.nix.mds.xyz
++ arg_host mdskvm-p05.nix.mds.xyz:/var/lib/one/datastores/123/32
++ echo mdskvm-p05.nix.mds.xyz:/var/lib/one/datastores/123/32
++ sed -r 's/^([^:]*):.*$/\1/'
+ DST_HOST=mdskvm-p05.nix.mds.xyz
++ dirname /var/lib/one/datastores/123/32
+ DST_DIR=/var/lib/one/datastores/123
++ dirname /var/lib/one/datastores/118/32
+ SRC_DS_DIR=/var/lib/one/datastores/118
++ basename /var/lib/one/datastores/118/32
+ SRC_VM_DIR=32
++ is_disk /var/lib/one/datastores/123/32
++ echo /var/lib/one/datastores/123/32
++ grep '/disk\.[0-9]\+'
++ '[' 1 -eq 0 ']'
++ echo 0
+ '[' 0 -eq 1 ']'
+ '[' /var/lib/one/datastores/118/32 == /var/lib/one/datastores/123/32 ']'
+ ssh_exec_and_log mdskvm-p05.nix.mds.xyz 'mv /var/lib/one/datastores/118/32 /var/lib/one/datastores/123/32' 'Error moving VM files to another System DS: /var/lib/one/datastores/118/32 to /var/lib/one/datastores/123/32 in mdskvm-p05.nix.mds.xyz'
+ ssh_exec_and_log_no_error mdskvm-p05.nix.mds.xyz 'mv /var/lib/one/datastores/118/32 /var/lib/one/datastores/123/32' 'Error moving VM files to another System DS: /var/lib/one/datastores/118/32 to /var/lib/one/datastores/123/32 in mdskvm-p05.nix.mds.xyz'
++ ssh mdskvm-p05.nix.mds.xyz bash -s
+ SSH_EXEC_ERR='mv: cannot stat '\''/var/lib/one/datastores/118/32'\'': No such file or directory'
+ SSH_EXEC_RC=1
+ '[' 1 -ne 0 ']'
+ log_error 'Command "mv /var/lib/one/datastores/118/32 /var/lib/one/datastores/123/32" failed: mv: cannot stat '\''/var/lib/one/datastores/118/32'\'': No such file or directory'
+ log_function ERROR 'Command "mv /var/lib/one/datastores/118/32 /var/lib/one/datastores/123/32" failed: mv: cannot stat '\''/var/lib/one/datastores/118/32'\'': No such file or directory'
+ echo 'ERROR: mv: Command "mv /var/lib/one/datastores/118/32 /var/lib/one/datastores/123/32" failed: mv: cannot stat '\''/var/lib/one/datastores/118/32'\'': No such file or directory'
ERROR: mv: Command "mv /var/lib/one/datastores/118/32 /var/lib/one/datastores/123/32" failed: mv: cannot stat '/var/lib/one/datastores/118/32': No such file or directory
+ '[' -n 'Error moving VM files to another System DS: /var/lib/one/datastores/118/32 to /var/lib/one/datastores/123/32 in mdskvm-p05.nix.mds.xyz' ']'
+ error_message 'Error moving VM files to another System DS: /var/lib/one/datastores/118/32 to /var/lib/one/datastores/123/32 in mdskvm-p05.nix.mds.xyz'
+ echo 'ERROR MESSAGE --8<------'
ERROR MESSAGE --8<------
+ echo 'Error moving VM files to another System DS: /var/lib/one/datastores/118/32 to /var/lib/one/datastores/123/32 in mdskvm-p05.nix.mds.xyz'
Error moving VM files to another System DS: /var/lib/one/datastores/118/32 to /var/lib/one/datastores/123/32 in mdskvm-p05.nix.mds.xyz
+ echo 'ERROR MESSAGE ------>8--'
ERROR MESSAGE ------>8--
+ return 1
+ EXEC_RC=1
+ '[' 1 '!=' 0 ']'
+ exit 1
[oneadmin@one01 datastores]$

And what about the local disks? How can I move VM’s off of those?

Thx,
TK

So I’ve mounted GV1 on host PH3 under the same expected path. It worked. I got the VM across. However, still wondering about the local storage drives. If I’m going to follow the method I used for the Gluster volume GV1, how will I mount these local drives over on PH3 or PH4 to migrate too? Via SSHFS as I do on the OpenNebula Sunstone VM? I have all the datastores for all the physicals mounted on the OpenNebula Sunstone VM be it via Gluster or SSHFS.

The datastore area and mounting drives is a bit of a grey area for me. Some parts I understand. Others I figure out when I use them.

Cheers,
TK