# Multiple CEPH datastores **URL:** \ **Category:** Product Support\ **Created:** [April 1, 2016, 11:06am UTC](https://forum.opennebula.io/t/multiple-ceph-datastores/2039 "2016-04-01T11:06:25Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Yenya](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/yenya/32/4624_2.png) [@Yenya](https://forum.opennebula.io/u/Yenya)\ **Post date:** [April 1, 2016, 11:06am UTC](https://forum.opennebula.io/t/multiple-ceph-datastores/2039/1 "2016-04-01T11:06:25Z")
I would like to have multiple CEPH-based datastores. What can and what cannot be shared between them? Can they be on the same CEPH cluster? (I suppose yes) Can they share the same CEPH pool? (probably not, as images are named “one-” starting from zero - can datastores define their own prefix instead of “one-”?) Can they share the same CEPH user/secret? Can they share the same libvirt secret UUID? Thanks! ---
**Author:** ![knawnd](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/knawnd/32/6119_2.png) [@knawnd](https://forum.opennebula.io/u/knawnd)\ **Post date:** [December 23, 2021, 1:43pm UTC](https://forum.opennebula.io/t/multiple-ceph-datastores/2039/2 "2021-12-23T13:43:21Z")
@Yenya , did you managed to enable multiple ceph-based DSes? I am trying to add a second ceph-based datastore into our opennebula 5.12.0.4 installation. I am using the same list of hosts in BRIDGE\_LIST (there are 3 Raft HA ONE Front-end nodes) attribute, the same CEPH\_SECRET and CEPH\_USER. The CEPH\_HOST attribute for each of two ceph-based DS has different values. The file /etc/ceph/ceph.conf on all 3 Raft HA FNs has ```auto [global] STAGING_DIR=/var/tmp RBD_FORMAT=2 [mon.a] mon addr = 192.168.220.121:6789 192.168.220.122:6789 192.168.220.123:6789 POOL_NAME=cloud [mon.b] mon addr = 192.168.0.18:6789 192.168.0.19:6789 192.168.0.20:6789 POOL_NAME=cloudssd ``` Hosts listed in the [mon.a] section is for HDD ceph-based DS (#135). Hosts listed in the [mon.b] section is for SSD ceph-based DS (#136). ‘onedatastore list’ command shows sometimes real capacity of DS #135 (and at that time the capacity of DS #136 is 0) and sometimes it’s vice versa. When I am trying to create an empty block device in DS #136 (at the moment when it has non-zero capacity) I am getting an error as below: ```auto Thu Dec 23 15:38:31 2021 [Z0][ImM][I]: rbd --id cloud create --image-format 2 cloudssd/one-3674 --size 1048576 || exit $?" failed: rbd: error opening pool 'cloudssd': (2) No such file or directory ``` It seems for me the OpenNebula can’t operate on two ceph-based DS and randomly choosing DS settings for each of ceph-based DS. I guess to work properly each ceph command has to have a list of ceph mon hosts e.g. ```auto rbd --id cloud create --image-format 2 cloudssd/one-3674 --size 1048576 -m 192.168.0.18:6789 -m 192.168.0.19:6789 -m 192.168.0.20:6789 ``` So I wonder if someone managed to configure several ceph-based DS in the OpenNebula? If someone did then I would appreciate a receipt. ---
**Author:** ![Yenya](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/yenya/32/4624_2.png) [@Yenya](https://forum.opennebula.io/u/Yenya)\ **Post date:** [January 3, 2022, 12:14pm UTC](https://forum.opennebula.io/t/multiple-ceph-datastores/2039/3 "2022-01-03T12:14:04Z")
@knawnd: so you want to use not only two Ceph _pools_, but instead two different Ceph _clusters_. I don’t believe this is supported in ONe, and even in Ceph itself, using more than one cluster in `ceph.conf` is apparently deprecated from v16 on. Do you have any special reason why use two clusters instead of one with two pools (and one set of mons for both pools)? Using more pools from one Ceph cluster should work (altough I did not test it myself). ---
**Author:** ![knawnd](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/knawnd/32/6119_2.png) [@knawnd](https://forum.opennebula.io/u/knawnd)\ **Post date:** [January 11, 2022, 4:21am UTC](https://forum.opennebula.io/t/multiple-ceph-datastores/2039/4 "2022-01-11T04:21:32Z")
> [@Yenya](#): > > @knawnd: so you want to use not only two Ceph _pools_ , but instead two different Ceph _clusters_ . exactly > [@Yenya](#): > > I don’t believe this is supported in ONe it seems for me it shouldn’t be hard to add such feature: one needs to add to rbd commands ceph mon hosts as in my example above. > [@Yenya](#): > > even in Ceph itself, using more than one cluster in `ceph.conf` is apparently deprecated from v16 on. I was not aware of that. I couldn’t found such info in the ceph docs. I would appreciate a link on such “feature” in the ceph docs. > [@Yenya](#): > > Do you have any special reason why use two clusters instead of one with two pools (and one set of mons for both pools)? We had initially HDD-based ceph storage and older ceph version. But now would like to add SSD-based one running on newest ceph version. ---
**Author:** ![Yenya](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/yenya/32/4624_2.png) [@Yenya](https://forum.opennebula.io/u/Yenya)\ **Post date:** [January 11, 2022, 9:00am UTC](https://forum.opennebula.io/t/multiple-ceph-datastores/2039/5 "2022-01-11T09:00:50Z")
@knawnd: I meant this: [https://docs.ceph.com/en/pacific/rados/configuration/common/#running-multiple-clusters-deprecated](https://docs.ceph.com/en/pacific/rados/configuration/common/#running-multiple-clusters-deprecated) But reading this more carefully, it seems that only running the server-based parts of multiple Ceph clusters on the same hardware is deprecated, not using multiple clusters from the same client. As for me, I have been upgrading my Ceph cluster from the initial setup with Ceph 8 (IIRC) and CentOS 7.1 through several generations of hardware, up to the present setup of Ceph 15 on CentOS 8 Stream. Ceph is really well upgradable in-place.