# CEPH parameters clarification **URL:** \ **Category:** Product Support\ **Created:** [September 7, 2017, 7:36pm UTC](https://forum.opennebula.io/t/ceph-parameters-clarification/4901 "2017-09-07T19:36:24Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\ **Post date:** [September 7, 2017, 7:36pm UTC](https://forum.opennebula.io/t/ceph-parameters-clarification/4901/1 "2017-09-07T19:36:24Z")
Hi, I’m trying to configure ceph storage as system\_ds and image\_ds for first time and some parts of the documentation are a little bit confusing for me: [http://docs.opennebula.org/5.4/deployment/open\_cloud\_storage\_setup/ceph\_ds.html#ceph-datastore](http://docs.opennebula.org/5.4/deployment/open_cloud_storage_setup/ceph_ds.html#ceph-datastore) In Datastore Layount I can see the following warning. > In this case context disk and auxiliar files (deployment description and chekpoints) are stored locally in the nodes. In the nodes, where is stored this data? If this data is stored locally, live migration can be performed? Regarding to system datastore: **ceph** (only with local FS on the DS directory) → As ceph is a distributed object, I cannot understand what does local FileSystem means on the Datastore directory… **shared** for shared transfer mode (only with shared FileSystem) → Is this a simple shared file server published by a ceph metadata server? On the other hand: **BRIDGE\_LIST: List of storage bridges to access the Ceph cluster**. I have been googling a little bit but I have not been able to find references to it. What is this parameter for? Thanks a lot and sorry for this newbee questions Ó * * * **Versions of the related components and OS (frontend, hypervisors, VMs): Opennbula 5.4, KVM** **Steps to reproduce: N/A** **Current results:N/A** **Expected results:N/A** ---
**Author:** ![heathen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/heathen/32/4720_2.png) [@heathen](https://forum.opennebula.io/u/heathen)\ **Post date:** [September 9, 2017, 6:40pm UTC](https://forum.opennebula.io/t/ceph-parameters-clarification/4901/2 "2017-09-09T18:40:17Z")
I’ve come here with almost the same question: In previous ONE 5.x versions if you have SYSTEM\_DS and IMAGE\_DS with the same TM\_MAD type (ceph) then live migration worked and only VM metadata files were moving via ssh driver. But this sometimes led to errors in case of VM host failure and VM HA triggers activated (at least because of a bug I found in ceph driver, but it is already fixed). BRIDGE\_LIST is the list of the [proxy] hosts to make transition between ceph pool and external sources/destinations. For example, if you upload image via Sunstone it will be received at the Sunstone host first, then ssh’ed to BRIDGE host and from there should be uploaded to ceph pool. If you have different (non-ceph) system datastore type and keep images in ceph datastore, at the time you create VM ONE will download\convert image from ceph pool to file on one of the BRIDGE hosts and ssh it to the hypervisor host. And my question to everybody: Now I’m a little bit confused with ceph TM\_MAD options. What is the difference between ceph TM\_MAD types? What is the best option to have images and VM disks on ceph and VMs metadata on a shared filesystem (cephfs, for ex.) to prevent any data movements in case of migration/HA triggers? Should system storage be created with ceph or shared TM\_MAD? And what about ceph image datastorage in this case? Thanks! ---
**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\ **Post date:** [September 9, 2017, 8:33pm UTC](https://forum.opennebula.io/t/ceph-parameters-clarification/4901/3 "2017-09-09T20:33:01Z")
Hi, > [@heathen](#): > > Now I’m a little bit confused with ceph TM\_MAD options. What is the difference between ceph TM\_MAD types? When you set TM\_MAD to _shared_ for the SYSTEM DS it is not Ceph related anymore. I it is just a shared filesystem - the Contextualization ISO,Volatile disk images and the checkpoint file during VM suspend/or clod move/ will be a QCOW2 files on the shared filesystem. > [@heathen](#): > > What is the best option to have images and VM disks on ceph and VMs metadata on a shared filesystem (cephfs, for ex.) to prevent any data movements in case of migration/HA triggers? You could use altered ceph TM\_MAD that do not copy data / delete data during VM migration 😉 Edit: > [@heathen](#): > > And what about ceph image datastorage in this case? There TM\_MAD used for the SYSTEM\_DS does not affect the behavior of the IMAGE DS. BR, Anton Todorov ---
**Author:** ![heathen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/heathen/32/4720_2.png) [@heathen](https://forum.opennebula.io/u/heathen)\ **Post date:** [September 9, 2017, 9:20pm UTC](https://forum.opennebula.io/t/ceph-parameters-clarification/4901/4 "2017-09-09T21:20:47Z")
Anton, thanks a lot for your reply! > [@atodorov\_storpool](#): > > When you set TM\_MAD to shared for the SYSTEM DS it is not Ceph related anymore. Yes, I get it. I see that even when I set SYSTEM\_DS with TM\_MAD=shared and IMAGE\_DS with DS\_MAD=ceph and TM\_MAD=ceph then disk images still live within ceph pool and ONE does not copy them to the system ds. But this confuses me even more 🙂 And I don’t understand why manual offers to create a different system\_ds with TM\_MAD=ceph in case of ceph image\_ds if everything works with TM\_MAD=shared? What behavior difference would expected between these options? I’ve tried to dig to the drivers code but probably did it not long enough 🙂 ---
**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\ **Post date:** [September 9, 2017, 10:17pm UTC](https://forum.opennebula.io/t/ceph-parameters-clarification/4901/5 "2017-09-09T22:17:02Z")
> [@heathen](#): > > But this confuses me even more 🙂 And I don’t understand why manual offers to create a different system\_ds with TM\_MAD=ceph in case of ceph image\_ds if everything works with TM\_MAD=shared? What behavior difference would expected between these options? When _shared_ is used you have a mix in the VM disks - some of them are on ceph datastore, other are files on a filesystem. With _ceph_ (or other distributed storage 😉 ) you’ll have consistency on all VM disk’s backing store. That’s the reason to hint that it is not so hard to tweak the ceph driver to work on shared filesystem. Our [addon-storpool](https://github.com/OpenNebula/addon-storpool) prove that it is possible to handle both ssh and shared backed modes - when _storpool_ is used as TM\_MAD it is matter of a configuration variable to switch between _ssh_ or _shared_ mode. Personally I prefer the ssh backed mode because in our recommended setup the only difference is that you have one service to care for less. All “metadata” (domain xml and contextualization iso) files are (re)generated on every (re)deployment of a VM so it is not issue for the VM recovery after HOST failure case. On another hand with the ceph TM\_MAD there is a corner case when the host fail just after VM state is stored ( VM suspend or cold migrate) - it is possible to lose the checkpoint file that holds the VM state. So it all depends on your use case and/or requirements and preferences - each one has its pros and cons. BR, Anton Todorov ---
**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\ **Post date:** [September 10, 2017, 7:33am UTC](https://forum.opennebula.io/t/ceph-parameters-clarification/4901/6 "2017-09-10T07:33:00Z")
Hi Anton, Thanks a lot for your clarifications! > [@atodorov\_storpool](#): > > All “metadata” (domain xml and contextualization iso) files are (re)generated on every (re)deployment of a VM so it is not issue for the VM recovery after HOST failure case. If images\_ds and system\_ds are ceph… Can you please advance where are contextualization files created in kvm hosts? Can they be placed in a shared fs? Does it have se nse? The idea is to allow opennebula to run the vm in the “best” host every startup instead of keeping the host. Thanks a lot ---
**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\ **Post date:** [September 10, 2017, 2:15pm UTC](https://forum.opennebula.io/t/ceph-parameters-clarification/4901/7 "2017-09-10T14:15:43Z")
Hi oscar. > [@oscar](#): > > If images\_ds and system\_ds are ceph… Can you please advance where are contextualization files created in kvm hosts? Can they be placed in a shared fs? Does it have se nse? Looking at the ceph’s TM\_MAD [tm\_mad/common/context](https://github.com/OpenNebula/one/blob/master/src/tm_mad/common/context#L96) The contextualization ISO image is created as a file on the front-end and then transferred to the KVM host as file. The procedure is same for _ssh_,_shared_ and _ceph_ TM\_MADs. (Hmm. so scratch the previous posts where i am saying that the context iso is on a ceph volume. There are a lot of improvements provided by our addon so I’ve totally missed counting this as another “extra”. I was thinking that it is the standard behavior for all distributed storages ☹ ) As already said the issue with ceph’s TM\_MAD is that when there is a shared filesystem underneath it is not aware of that and it is copying and deleting files when VMs are migrated or undeployed ([1](https://github.com/OpenNebula/one/blob/master/src/tm_mad/ceph/premigrate#L70), [2](https://github.com/OpenNebula/one/blob/master/src/tm_mad/ceph/mv)). Can anyone test _ceph_ on a shared filesystem as SYSTEM DS TM\_MAD but borrowing/replacing/ _mv_,_premigrate_,_postmigrate_ and _failmigrate_ from the _shared_ TM\_MAD? Reading the code it looks like these are the only files that mess such setup… > [@oscar](#): > > The idea is to allow opennebula to run the vm in the “best” host every startup instead of keeping the host. When you set a VM to ‘pending’ the scheduler is deciding where to run the VM. BR, Anton Todorov