# Why IMAGES\_DS parameters affects SYSTEM\_DS?

**URL:** <https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072>\
**Category:** Integration Support\
**Created:** [March 26, 2019, 11:22am UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072 "2019-03-26T11:22:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![kvaps](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/kvaps/32/4496_2.png) [@kvaps](https://forum.opennebula.io/u/kvaps)\
**Post date:** [March 26, 2019, 11:22am UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/1 "2019-03-26T11:22:23Z")

</div>

Hi, I have architecture design. I’m using two types of system datastores: eg `fs_lvm` and `shared` and I’m not understand, why I can’t simple single images datastore in this case.

Eg when you creating new images datastore, you must specify parameters like:

```auto
TM_MAD = fs_lvm
DISK_TYPE = "BLOCK"

```

In this case, all images instantiated from this datastore will use `fs_lvm` driver, and deploy image into block device.

If we change our images datastore parameters to:

```auto
TM_MAD = shared
DISK_TYPE = "FILE"

```

Image will be deployed as a file even if system datastore managed by `fs_lvm` driver.

I can’t use same images datastore for both types of system datastores, because images inherit options from images datastore, but parameters from system datastore are ignored, that’s crazy!

Is there any specific reason for that, or any ideas how can we solve that?

I already wrote issue about that:

> <https://github.com/OpenNebula/one/issues/2354>
>
> \*\*Description\*\*
> You have one common NFS datastore for storing images.
> You have… few clusters with different system datastores with different types: example \`fs\_lvm\` and \`shared\`
> You want to use this images datastore for deploying VMs to both systems.
> For \*\*images\*\* datastore you can set:
> \* \`TM\_MAD=fs\_lvm\` -then it will work with \`fs\_lvm\` but will not work with \`shared\` system ds (return errors like device not found)
> \* \`TM\_MAD=shared\` -then it will work with \`shared\` but will work wrong with \`fs\_lvm\` system ds (it will create files instead devices)
> 
> \*\*To Reproduce\*\*
> \* Create \`fs\_lvm\` and \`shared\` systemds.
> \* Create shared another one for images.
> \* Try to apply mentioned options for it.
> 
> \*\*Expected behavior\*\*
> Driver will take decision based on TM\_MAD parameter of system datastore (not images)
> like for \`shared\` system datastore it will operate files, and for \`fs\_lvm\` datastore it will operate lvm partitions
> 
> \*\*Details\*\*
> - Affected Component: Storage
> - Hypervisor: KVM
> - Version: 5.6.0
> 
> 
> 
> 
> 
> 
> 
> 
> \## Progress Status
> \- \[\] Branch created 
> \- \[\] Code committed to development branch
> \- \[\] Testing - QA
> \- \[\] Documentation 
> \- \[\] Release notes - resolved issues, compatibility, known issues
> \- \[\] Code committed to upstream release/hotfix branches
> \- \[\] Documentation committed to upstream release/hotfix branches

Today I faced with another one

> <https://github.com/OpenNebula/one/issues/3130>
>
> \*\*Description\*\*
> 
> Here is specific check for that
> 
> https://github.com/OpenNeb…ula/one/blob/release-5.8.0/src/tm\_mad/fs\_lvm/mv#L86-L88
> 
> \`\`\`bash
> if \["${TYPE}" != "BLOCK" \]; then
> exit 0
> fi
> \`\`\`
> 
> But disk type is always
> \`\`\`
> TYPE = "FILE"
> \`\`\`
> 
> \*\*To Reproduce\*\*
> \* Configure lvm datastore according the \[documentation\](docs.opennebula.org/5.8/deployment/open\_cloud\_storage\_setup/lvm\_drivers.html)
> \* Instantiate new VM
> \* Shutdown it, and try to migrate datastore
> 
> \*\*Expected behavior\*\*
> VM will be migrated with all data
> 
> \*\*Actual behavior\*\*
> Only metadata directory will be moved, VM will continue use logical volume from the old datastore
> 
> \*\*Details\*\*
> - Affected Component: Storage drivers
> - Hypervisor: KVM
> - Version: 5.6 5.8
> 
> \*\*Additional context\*\*
> Add any other context about the problem here.
> 
> 
> 
> 
> 
> 
> 
> 
> \## Progress Status
> \- \[\] Branch created 
> \- \[\] Code committed to development branch
> \- \[\] Testing - QA
> \- \[\] Documentation 
> \- \[\] Release notes - resolved issues, compatibility, known issues
> \- \[\] Code committed to upstream release/hotfix branches
> \- \[\] Documentation committed to upstream release/hotfix branches

One of possible solution can be to replace it like:

```diff
-if ["${TYPE}" != "BLOCK"]; then
+if ["${DISK_TYPE}" != "BLOCK"]; then
     exit 0
 fi

```

In this case it will take `DISK_TYPE` parameter from the `images_ds` but not from `system_ds`, so we still have the same problem here.

Now I want to decide future design of these options and how to inherit them.

Thanks for attention!

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [April 17, 2019, 9:31am UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/2 "2019-04-17T09:31:31Z")

</div>

> [@kvaps](#):
>
> Hi, I have architecture design. I’m using two types of system datastores: eg `fs_lvm` and `shared` and I’m not understand, why I can’t simple single images datastore in this case.

A VM could have only one SYSTEM datastore and multiple IMAGE datastores. Generally (with some exclusions) OpenNebula use the TM\_MAD of the IMAGE datastore to provide the disks for the VM because the IMAGE’s TM\_MAD knows how to deal with the media on the given datastore.  
IMHO It’s not the SYSTEM’s TM\_MAD business to know how to import image from a IMAGE datastore because the IMAGE datastore could be anything. As I said there are exclusions and IMHO they are wrong but it is a topic for another discussion.

I think you are looking for a feature to transfer (import/export?) image from one storage to another?

> [@kvaps](#):
>
> Image will be deployed as a file even if system datastore managed by `fs_lvm` driver.

Yep. The TM\_MAD of the IMAGE datastore knows how to deal with the images within 🙂  
The TM\_MAD should be used only for the images in the SYSTEM datastore - the volatile disks.

I’ve already have a discussion on similar topic with OpenNebula and IMHO the LN\_TARGET and COPY\_TARGET variables could be used for this. But currently their values are only informative, used for OpenNebula’s internals - changing them do not change the behaviour of the given TM\_MAD.

There is another issue buried there, though. The disks created by SYSTEM datastore’s TM\_MAD - the volatile disks - are always defined in the domain XML as files. It is hard-coded in the OpenNebula core and there is no option to change it. But I’ve already created a workaround in addon-storpool.

I really appreciate the opened discussion! This way we could share and improve our knowledge of OpenNebula, show the good things and address the issues that could be improved.

Cheers,  
Anton

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [April 17, 2019, 11:19am UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/3 "2019-04-17T11:19:52Z")

</div>

Let me just extend Anton’s response. This is the structure of the storage drivers:

```auto
             +-----------+ +-------------+ TM_MAD
             | | | |
             | Image | TM_MAD | System +---------+
 DS_MAD | Datastore | | Datastore | |
             | +-------------->+ | |
+----------->+ | | | |
             | +<--------------+ +<--------+
             | | | |
             +-----------+ +-------------+

```

- DS\_MAD is to manage images in the image datastore
- TM\_MAD is to move images from the images datastore to the system one, and back
- TM\_MAD is also to move images across hypervisors within the same system datastore

Images datastores **cannot work** with any system datastore. This is the main reason image datastore attributes are used together with the system ones. Note that most operations are optimized assuming this behavior. So as Anton suggests there is no universal export/import functionality that will seriously impact on the performance of storage operations.

In some circumstances it is interesting that an Image datastore is able to work with different types of system datastores. This has been also implemented in opennebula with the ability to specify different transfer modes. [For example ceph image ds can work with a ceph system ds and a ssh system ds](http://docs.opennebula.org/5.8/deployment/open_cloud_storage_setup/ceph_ds.html#ssh-mode). Note also that even in this case some parameters needs also to be specify in the image DS in order to properly schedule the storage.

So if you want your drivers to work with different transfer modes you can follow the same approach. OpenNebula includes the infrastructure needed to schedule the storage in this scenario (e.g. assign one system DS or other based on different policies and/or constraints)

---

<div class="post-metadata">

**Author:** ![kvaps](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/kvaps/32/4496_2.png) [@kvaps](https://forum.opennebula.io/u/kvaps)\
**Post date:** [April 17, 2019, 11:40am UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/4 "2019-04-17T11:40:46Z")

</div>

Well, following documentation we can see that there is:

- **datastore\_mad** - needed for manage images on the images datastore
- **tm\_mad** - knows how to copy or link volumes from images datastore to system one and back

It is should be enough, but problem is that when you copying volumes from images datastore to system one, the **tm\_mad** assigned to the images datastore is called but it is always using parameters assigned to system datastore.

And after all we found another **tm\_mad** parameter which assigned to system datastore, and needed only for operating with volatile disks, eg mv them to another system\_ds or copy to images datastore.

This is totally confusing!

> [@atodorov\_storpool](#):
>
> The TM\_MAD of the IMAGE datastore knows how to deal with the images within 🙂

This is right think, main problem is that usually it is not working, eg you can’t copy from **ssh** images datastore to **ceph** system datastore. Or from **shared** to **fs\_lvm** , it might work but image will be as a file not logical volume.

But I want to have this functionality, because I have one images datastore with a lot of images and configured templates, and I don’t want to have another one.

> [@ruben](#):
>
> Images datastores **cannot work** with any system datastore. This is the main reason image datastore attributes are used together with the system ones. Note that most operations are optimized assuming this behavior. So as Anton suggests there is no universal export/import functionality that will seriously impact on the performance of storage operations.

I agree, my idea is to decide and add ability to develop this kind of cross datastore actions.

1. First my idea was to change the logic of the **tm\_mad** execution to always use driver specified for the system datastore, it should know how to copy from each type of system\_ds.  
This is breaking changes, but as far I can see it is not so harmful, because current implementation of **tm\_mad** drivers is suppose to use them in pairs, eg. if images datastore have `TM_MAD=fs_lvm` this is also required to have `TM_MAD=fs_lvm` for the system datastore.

2. Another option is teach every tm driver to work with the right system ds, eg. as you said if images datastore have `TM_MAD=ssh`, but system datastore have `TM_MAD=fs_lvm`, then **ssh** transport driver should know how to place the drive as logical volume.

In this case we can add similar hook like we’ve done with vmm driver, remember **vmm/kvm/save.ceph** , and **vmm/kvm/restore.ceph** actions, so we can have something like that, eg:  
**tm/ssh/clone.fs\_lvm** will know how to copy to **fs\_lvm** , or **tm/ceph/cpds.shared** will know how to copy from **ceph** to **shared**

This can be simple implemented in any storage driver without changing core logic.  
In case with copying from **shared** to **fs\_lvm** it can be just symlink like:

```auto
tm/shared/clone.fs_lvm --> ../fs_lvm/clone

```

and small hook for check target **tm\_mad** added to each action.

```auto
if [-f "${DRIVER_PATH}/clone.${TM_MAD}"]; then
    exec "${DRIVER_PATH}/clone.${TM_MAD}" "$@"
fi

```

What is your opinion about it? Do you see another way how to solve that?

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [April 17, 2019, 11:51am UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/5 "2019-04-17T11:51:28Z")

</div>

Yes again, consider this is needed because the deployment is made in two steps:

- Allocation some prechecks are made and used by the scheduler (e.g. capacity/cluster requirements)
- Deployment after the final system ds is scheduled (e.g. quotas)

The second option is the right one, adapt the driver to work with different system ds (the ones that actually make sense). As I said this has not been implemented in general for performance reasons. in this way:

- TM in image ds is used to “transfer” images to the system ds, and change format
- TM in system ds is used to “transfer” images across hypervisors

Again, OpenNebula is already prepared to work like this and should consider quotas in the right places if you properly set the attributes in oned.conf mentioned by Anton (LN\_TARGET etc…)

---

<div class="post-metadata">

**Author:** ![kvaps](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/kvaps/32/4496_2.png) [@kvaps](https://forum.opennebula.io/u/kvaps)\
**Post date:** [April 17, 2019, 12:08pm UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/6 "2019-04-17T12:08:06Z")

</div>

OK, thanks for the discussion guys!

@ruben can you agree that creating this kind of hooks is the right way for add support for different tm\_driver targets for the existing tm drivers?

> **[OpenNebula/one](https://github.com/OpenNebula/one/compare/master...kvaps:bug-2354-2)**
>
> OpenNebula cloud management platform. Contribute to OpenNebula/one development by creating an account on GitHub.

better than

> <https://github.com/OpenNebula/one/pull/2615/files>

and better than implementation this functionality inside the same script body

Or maybe better to implement this functionality to **one\_tm** executor, for not overload current drivers by additional requests to API and additional checks?  
It will also provide an opportunity to extend standard drivers with custom actions without modifying them.

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [April 17, 2019, 12:20pm UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/7 "2019-04-17T12:20:20Z")

</div>

I was thinking in something like:

> <https://github.com/OpenNebula/one/blob/master/src/tm_mad/ceph/clone#L120>

The hook is within the action. However your proposal seems much more clean. This could be even implemented in the generic tm\_mad driver (here: [https://github.com/OpenNebula/one/blob/master/src/tm\_mad/one\_tm.rb#L99](https://github.com/OpenNebula/one/blob/master/src/tm_mad/one_tm.rb#L99)) Although this would need some minimal support from oned.

---

<div class="post-metadata">

**Author:** ![kvaps](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/kvaps/32/4496_2.png) [@kvaps](https://forum.opennebula.io/u/kvaps)\
**Post date:** [April 17, 2019, 12:32pm UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/8 "2019-04-17T12:32:52Z")

</div>

OK, I’ll describe this proposal in feature request, thanks!

---

<div class="post-metadata">

**Author:** ![kvaps](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/kvaps/32/4496_2.png) [@kvaps](https://forum.opennebula.io/u/kvaps)\
**Post date:** [April 17, 2019, 1:00pm UTC](https://forum.opennebula.io/t/why-images-ds-parameters-affects-system-ds/7072/9 "2019-04-17T13:00:55Z")

</div>

> <https://github.com/OpenNebula/one/issues/3244>