# Multiple ceph cluster possible

**URL:** <https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234>\
**Category:** Operations\
**Created:** [January 11, 2022, 12:15am UTC](https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234 "2022-01-11T00:15:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hoang\_Hien\_Nguyen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/hoang_hien_nguyen/32/6221_2.png) [@Hoang\_Hien\_Nguyen](https://forum.opennebula.io/u/Hoang_Hien_Nguyen)\
**Post date:** [January 11, 2022, 12:15am UTC](https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234/1 "2022-01-11T00:15:48Z")

</div>

I am looking for the way to configure Opennebula to work with two different ceph clusters (not same cluster with different pool).  
I already tried different ceph config file path, user, user key path but ONE can not monitor second cluster and return size 0.

---

<div class="post-metadata">

**Author:** ![knawnd](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/knawnd/32/6119_2.png) [@knawnd](https://forum.opennebula.io/u/knawnd)\
**Post date:** [January 11, 2022, 4:15am UTC](https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234/2 "2022-01-11T04:15:14Z")

</div>

There is another [thread](https://forum.opennebula.io/t/multiple-ceph-datastores/2039) on the same topic.

---

<div class="post-metadata">

**Author:** ![Hoang\_Hien\_Nguyen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/hoang_hien_nguyen/32/6221_2.png) [@Hoang\_Hien\_Nguyen](https://forum.opennebula.io/u/Hoang_Hien_Nguyen)\
**Post date:** [January 11, 2022, 8:01am UTC](https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234/3 "2022-01-11T08:01:30Z")

</div>

Thank you.  
I read that topic and there will be no support multiple ceph clusters. That leave me the only option to export cephfs for ONE.

---

<div class="post-metadata">

**Author:** ![agemuend](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/agemuend/32/7211_2.png) [@agemuend](https://forum.opennebula.io/u/agemuend)\
**Post date:** [January 11, 2022, 3:59pm UTC](https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234/4 "2022-01-11T15:59:54Z")

</div>

I think you misunderstood the thread. It is possible, we use two clusters for example.

---

<div class="post-metadata">

**Author:** ![Hoang\_Hien\_Nguyen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/hoang_hien_nguyen/32/6221_2.png) [@Hoang\_Hien\_Nguyen](https://forum.opennebula.io/u/Hoang_Hien_Nguyen)\
**Post date:** [January 12, 2022, 4:25am UTC](https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234/5 "2022-01-12T04:25:26Z")

</div>

Please guide me, I try add another cluster pool:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/opennebula/original/2X/d/d176b0baa69e8725923b2e2c2d919bf01c0d6e0a.png)

and got monitor error:  
Command execution failed (exit code: 255): /var/lib/one/remotes/datastore/ceph/monitor  
I try to run cmd used by that script and everything working  
**ceph --conf /etc/ceph/hddceph.conf --id hdd df detail --format xml**  
\<total\_bytes\>320017929338880\</total\_bytes\>\<total\_avail\_bytes\>320017759584256\</total\_avail\_bytes\>\<total\_used\_bytes\>169754624\</total\_used\_bytes\>\<total\_used\_raw\_bytes\>169754624\</total\_used\_raw\_bytes\>\<total\_used\_raw\_ratio\>5.3045346248836722e-07\</total\_used\_raw\_ratio\>\<num\_osds\>20\</num\_osds\>\<num\_per\_pool\_osds\>18\</num\_per\_pool\_osds\>\<num\_per\_pool\_omap\_osds\>18\</num\_per\_pool\_omap\_osds\>\<stats\_by\_class\>\<total\_bytes\>320017929338880\</t …

---

<div class="post-metadata">

**Author:** ![agemuend](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/agemuend/32/7211_2.png) [@agemuend](https://forum.opennebula.io/u/agemuend)\
**Post date:** [January 12, 2022, 8:27am UTC](https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234/6 "2022-01-12T08:27:34Z")

</div>

We use a second completely different cluster. The important part is to have the different ceph conf and if required the different keyring file both for the clients and for libvirt as a secret. We created a second folder /etc/ceph2 and added the other monitor nodes and fixed keyring entry in the ceph.conf there, e.g.

[client]  
keyring=/etc/ceph2/ceph.keyring

Have you tried the monitor command as the user that runs oned, e.g. oneadmin? Is there any further error message?

---

<div class="post-metadata">

**Author:** ![knawnd](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/knawnd/32/6119_2.png) [@knawnd](https://forum.opennebula.io/u/knawnd)\
**Post date:** [February 8, 2022, 1:29pm UTC](https://forum.opennebula.io/t/multiple-ceph-cluster-possible/10234/7 "2022-02-08T13:29:56Z")

</div>

@agemuend , thanks a lot for sharing your experience and tricks! They helped me to add second ceph cluster as OpenNebula datastore (the same libvirt secret across both ceph clusters, each ceph DS has own set of ceph mon hosts, the same hosts listed in BRIDGE\_LIST, separate ceph config files for each ceph DS).

One of the issue I had was ONE ceph monitor script failed and because of that DS size was zero.

The reason for that error was in from format of keyfile used in the ONE ceph monitor script: that file (keyfile for ceph) needed to have just a key without any other information.