Moving images to another datastore?

Product Support
1

Hi all,

I use OpenNebula on top of Ceph datastore. Recently I have added NVMe storage pool to my Ceph cluster, and created a new ONe datastore ceph_nvme for it. Now I want to move some of my running VMs’ disks to this new datastore. What is the recommended way how to do it with minimum VM downtime?

Some of the issues I ran into:

  • Save As cannot save the image (or a snapshot) to a different datastore
  • Clone works across datastores, but fails when the source image has snapshots
  • Snapshots can be flattened only in POWEROFF state, not in RUNNING, UNDEPLOYED, and not for detached images.
  • Some of my VMs run on top of non-persistent images, which on Ceph work the same way as persistent ones, except that they lose data on VM destroy or when the image is detached.
  • On some VMs which have not-so-stateful data, it would be nice to create snapshot, copy this snapshot over to the new datastore, shutdown the VM, unplug the old disk and plug in the new one.

So far I am trying this:

  • Shut down the VM to the POWEROFF state (not UNDEPLOYED)
  • Save the image using Save As when it has snapshots or is non-persistent
  • Clone the original image (if possible) or the saved image to the new datastore
  • Make the cloned image persistent from the top ‘…’ button
  • Remember to check whether the old image is virtio-block based (vda) or virtio-scsi based (sda)
  • Unplug the old image (non-persistent image loses data now)
  • Plug the cloned image using the same bus as before
  • Resume (start) the VM
  • Remove the “save as” image and/or the original image

This is requires two copies instead of one for some VMs. Do you have any recommendations how to do it better?

Thanks!

-Yenya

2

Hi Yenya,

Your workflow is quite inventive and may still be useful when moving only a specific disk or Image.

However, if the goal is to move all disks of an existing VM to the NVMe-backed Ceph pool, the simpler supported approach is an offline datastore migration. Create a Ceph System Datastore pointing to the NVMe pool and migrate the VM to it while keeping it on the same host:

onevm migrate <VM_ID> <CURRENT_HOST_ID> <TARGET_SYSTEM_DS_ID> --poff

OpenNebula will power off the VM, move its disks to the target datastore, and start it again. Ceph live storage migration is not supported, so some downtime is still required.

This moves the active VM disks, but not necessarily the original base Images in the Image Datastore.

3

Thanks. One more question, though: my ceph_nvme datastore is an image datastore, not system one. Should I make a system datastore on top of the same NVMe-based Ceph pool? For my original HDD-based Ceph pool, I have two datastores (system and image) on top of the same pool.

Thanks!

-Yenya

4

Hey,

Personally I would recommend you to have system DS on that pool as well. Depends on the number of Ceph nodes, OSDs and networking, but you should notice a performance increase, too.

5

OK, I created a new system datastore(*) on ceph_nvme pool, and tried to run

# onevm migrate $VM_ID $CURRENT_HOST $NEW_SYSTEM_DATASTORE_ID --poff
[one.vm.migrate] System datastore migration not supported by driver

Is it supposed to work?

(*) FWIW, my old system datastore on HDD-based pool and my new datastore on NVMe-based pool have the following differences in onedatastore show (+ is the new NVMe-based one, - is the original one; note for example commas versus spaces in CEPH_HOST)

-DATASTORE 0 INFORMATION     
+DATASTORE 114 INFORMATION                                                        

-ID             : 0  
+ID             : 114                             

-NAME           : system                   
+NAME           : system_nvme                

-BASE PATH      : /var/lib/one//datastores/0
+BASE PATH      : /var/lib/one//datastores/114

-CEPH_HOST="mon1 mon2 mon3"
+CEPH_HOST="mon1,mon2,mon3"

-POOL_NAME="one"
+POOL_NAME="one_nvme"

+DATASTORE_CAPACITY_CHECK="NO"
+NFS_AUTO_ENABLE="NO"
+NO_DECOMPRESS="NO"
+PERSISTENT_SNAPSHOTS="YES"
+QCOW2_STANDALONE="NO"
+RESTIC_SPARSIFY="NO"
+RSYNC_SPARSIFY="NO"

Thanks!

-Yenya

6

Hi Yenya,

Thanks for testing this, and apologies for the earlier incorrect suggestion.

Ceph-to-Ceph System Datastore migration is not currently implemented in OpenNebula 7.2. The default DS_MIGRATE = "NO" setting for the Ceph transfer driver is intentional and should not be changed.

Therefore, your original workflow is currently the correct OpenNebula-managed approach. For persistent Images without snapshots, you can clone them directly to the NVMe Image Datastore. For non-persistent VM disks, or Images with snapshot dependencies, the intermediate Save As and/or flatten operation is unfortunately required.

A manual Ceph rbd copy could avoid an intermediate copy at the storage level, but OpenNebula would not manage the associated metadata changes, so this would not be a supported procedure.

7

@Rostyslav_Yatsyshyn - no problem. It would be nice to have. As for rbd copy approach: It would be interesting to have support for “wild images” the same way as OpenNebula supports detecting “wild VMs” - running VMs about which ONe does not know anything. Then it would be possible to move the image by arbitrary means, and ONe would then “adopt” them with correct parameters later. I ran rbd ls oneon my cluster, and found about 150 images which had name with one- prefix, but which OpenNebula did not list in oneimage list. Looking at the number[s] in their name, they belonged to VMs deleted long time ago. Monitoring the datastore could detect it.

The link right next to your and @brunorro’s name “OpenNebula Team” leads to a non-existing page https://forum.opennebula.io/g/team - you might want to fix it.

8

Thanks for pointing that out! That page exists but it has restricted access.

9

Hi Yenya,

The idea is understandable, but adopting “wild” RBD images may be problematic for the same reason we moved away from wild VM detection in newer releases: such resources may have been created outside OpenNebula and may not follow the layout, naming, metadata, or configuration expected by OpenNebula.

For this reason, I think implementing the feature request for migration between Ceph datastores would be a safer and more controlled approach:

OpenNebula would remain aware of the source and destination images throughout the operation and could properly manage snapshots, VM disk references, metadata, and cleanup.

10

did you check if fireedge is actually looking in the right spot for those view configs? i notice you’ve got views under the root and also duplicated under admin/user — might be a path issue where it’s only picking up the top-level admin and user dirs