Why IMAGES_DS parameters affects SYSTEM_DS?

Development
1

Hi, I have architecture design. I’m using two types of system datastores: eg fs_lvm and shared and I’m not understand, why I can’t simple single images datastore in this case.

Eg when you creating new images datastore, you must specify parameters like:

TM_MAD = fs_lvm
DISK_TYPE = "BLOCK"

In this case, all images instantiated from this datastore will use fs_lvm driver, and deploy image into block device.

If we change our images datastore parameters to:

TM_MAD = shared
DISK_TYPE = "FILE"

Image will be deployed as a file even if system datastore managed by fs_lvm driver.

I can’t use same images datastore for both types of system datastores, because images inherit options from images datastore, but parameters from system datastore are ignored, that’s crazy!

Is there any specific reason for that, or any ideas how can we solve that?

I already wrote issue about that:

Today I faced with another one

One of possible solution can be to replace it like:

-if [ "${TYPE}" != "BLOCK" ]; then
+if [ "${DISK_TYPE}" != "BLOCK" ]; then
     exit 0
 fi

In this case it will take DISK_TYPE parameter from the images_ds but not from system_ds, so we still have the same problem here.

Now I want to decide future design of these options and how to inherit them.

Thanks for attention!

2

A VM could have only one SYSTEM datastore and multiple IMAGE datastores. Generally (with some exclusions) OpenNebula use the TM_MAD of the IMAGE datastore to provide the disks for the VM because the IMAGE’s TM_MAD knows how to deal with the media on the given datastore.
IMHO It’s not the SYSTEM’s TM_MAD business to know how to import image from a IMAGE datastore because the IMAGE datastore could be anything. As I said there are exclusions and IMHO they are wrong but it is a topic for another discussion.

I think you are looking for a feature to transfer (import/export?) image from one storage to another?

Yep. The TM_MAD of the IMAGE datastore knows how to deal with the images within :slight_smile:
The TM_MAD should be used only for the images in the SYSTEM datastore - the volatile disks.

I’ve already have a discussion on similar topic with OpenNebula and IMHO the LN_TARGET and COPY_TARGET variables could be used for this. But currently their values are only informative, used for OpenNebula’s internals - changing them do not change the behaviour of the given TM_MAD.

There is another issue buried there, though. The disks created by SYSTEM datastore’s TM_MAD - the volatile disks - are always defined in the domain XML as files. It is hard-coded in the OpenNebula core and there is no option to change it. But I’ve already created a workaround in addon-storpool.

I really appreciate the opened discussion! This way we could share and improve our knowledge of OpenNebula, show the good things and address the issues that could be improved.

Cheers,
Anton

3

Let me just extend Anton’s response. This is the structure of the storage drivers:

             +-----------+               +-------------+    TM_MAD
             |           |               |             |
             | Image     |  TM_MAD       | System      +---------+
 DS_MAD      | Datastore |               | Datastore   |         |
             |           +-------------->+             |         |
+----------->+           |               |             |         |
             |           +<--------------+             +<--------+
             |           |               |             |
             +-----------+               +-------------+
  • DS_MAD is to manage images in the image datastore
  • TM_MAD is to move images from the images datastore to the system one, and back
  • TM_MAD is also to move images across hypervisors within the same system datastore

Images datastores cannot work with any system datastore. This is the main reason image datastore attributes are used together with the system ones. Note that most operations are optimized assuming this behavior. So as Anton suggests there is no universal export/import functionality that will seriously impact on the performance of storage operations.

In some circumstances it is interesting that an Image datastore is able to work with different types of system datastores. This has been also implemented in opennebula with the ability to specify different transfer modes. For example ceph image ds can work with a ceph system ds and a ssh system ds. Note also that even in this case some parameters needs also to be specify in the image DS in order to properly schedule the storage.

So if you want your drivers to work with different transfer modes you can follow the same approach. OpenNebula includes the infrastructure needed to schedule the storage in this scenario (e.g. assign one system DS or other based on different policies and/or constraints)

4

Well, following documentation we can see that there is:

  • datastore_mad - needed for manage images on the images datastore
  • tm_mad - knows how to copy or link volumes from images datastore to system one and back

It is should be enough, but problem is that when you copying volumes from images datastore to system one, the tm_mad assigned to the images datastore is called but it is always using parameters assigned to system datastore.

And after all we found another tm_mad parameter which assigned to system datastore, and needed only for operating with volatile disks, eg mv them to another system_ds or copy to images datastore.

This is totally confusing!

This is right think, main problem is that usually it is not working, eg you can’t copy from ssh images datastore to ceph system datastore. Or from shared to fs_lvm, it might work but image will be as a file not logical volume.

But I want to have this functionality, because I have one images datastore with a lot of images and configured templates, and I don’t want to have another one.

I agree, my idea is to decide and add ability to develop this kind of cross datastore actions.

  1. First my idea was to change the logic of the tm_mad execution to always use driver specified for the system datastore, it should know how to copy from each type of system_ds.
    This is breaking changes, but as far I can see it is not so harmful, because current implementation of tm_mad drivers is suppose to use them in pairs, eg. if images datastore have TM_MAD=fs_lvm this is also required to have TM_MAD=fs_lvm for the system datastore.

  2. Another option is teach every tm driver to work with the right system ds, eg. as you said if images datastore have TM_MAD=ssh, but system datastore have TM_MAD=fs_lvm, then ssh transport driver should know how to place the drive as logical volume.

In this case we can add similar hook like we’ve done with vmm driver, remember vmm/kvm/save.ceph, and vmm/kvm/restore.ceph actions, so we can have something like that, eg:
tm/ssh/clone.fs_lvm will know how to copy to fs_lvm, or tm/ceph/cpds.shared will know how to copy from ceph to shared

This can be simple implemented in any storage driver without changing core logic.
In case with copying from shared to fs_lvm it can be just symlink like:

tm/shared/clone.fs_lvm --> ../fs_lvm/clone

and small hook for check target tm_mad added to each action.

if [ -f "${DRIVER_PATH}/clone.${TM_MAD}" ]; then
    exec "${DRIVER_PATH}/clone.${TM_MAD}" "$@"
fi

What is your opinion about it? Do you see another way how to solve that?

5

Yes again, consider this is needed because the deployment is made in two steps:

  • Allocation some prechecks are made and used by the scheduler (e.g. capacity/cluster requirements)
  • Deployment after the final system ds is scheduled (e.g. quotas)

The second option is the right one, adapt the driver to work with different system ds (the ones that actually make sense). As I said this has not been implemented in general for performance reasons. in this way:

  • TM in image ds is used to “transfer” images to the system ds, and change format
  • TM in system ds is used to “transfer” images across hypervisors

Again, OpenNebula is already prepared to work like this and should consider quotas in the right places if you properly set the attributes in oned.conf mentioned by Anton (LN_TARGET etc…)

6

OK, thanks for the discussion guys!

@ruben can you agree that creating this kind of hooks is the right way for add support for different tm_driver targets for the existing tm drivers?

better than

and better than implementation this functionality inside the same script body

Or maybe better to implement this functionality to one_tm executor, for not overload current drivers by additional requests to API and additional checks?
It will also provide an opportunity to extend standard drivers with custom actions without modifying them.

7

I was thinking in something like:

The hook is within the action. However your proposal seems much more clean. This could be even implemented in the generic tm_mad driver (here: https://github.com/OpenNebula/one/blob/master/src/tm_mad/one_tm.rb#L99) Although this would need some minimal support from oned.

8

OK, I’ll describe this proposal in feature request, thanks!

9
1 Like