I am looking for the way to configure Opennebula to work with two different ceph clusters (not same cluster with different pool).
I already tried different ceph config file path, user, user key path but ONE can not monitor second cluster and return size 0.
There is another thread on the same topic.
Thank you.
I read that topic and there will be no support multiple ceph clusters. That leave me the only option to export cephfs for ONE.
I think you misunderstood the thread. It is possible, we use two clusters for example.
Please guide me, I try add another cluster pool:
and got monitor error:
Command execution failed (exit code: 255): /var/lib/one/remotes/datastore/ceph/monitor
I try to run cmd used by that script and everything working
ceph --conf /etc/ceph/hddceph.conf --id hdd df detail --format xml
<total_bytes>320017929338880</total_bytes><total_avail_bytes>320017759584256</total_avail_bytes><total_used_bytes>169754624</total_used_bytes><total_used_raw_bytes>169754624</total_used_raw_bytes><total_used_raw_ratio>5.3045346248836722e-07</total_used_raw_ratio><num_osds>20</num_osds><num_per_pool_osds>18</num_per_pool_osds><num_per_pool_omap_osds>18</num_per_pool_omap_osds><stats_by_class><total_bytes>320017929338880</t …
We use a second completely different cluster. The important part is to have the different ceph conf and if required the different keyring file both for the clients and for libvirt as a secret. We created a second folder /etc/ceph2 and added the other monitor nodes and fixed keyring entry in the ceph.conf there, e.g.
[client]
keyring=/etc/ceph2/ceph.keyring
Have you tried the monitor command as the user that runs oned, e.g. oneadmin? Is there any further error message?
@agemuend , thanks a lot for sharing your experience and tricks! They helped me to add second ceph cluster as OpenNebula datastore (the same libvirt secret across both ceph clusters, each ceph DS has own set of ceph mon hosts, the same hosts listed in BRIDGE_LIST, separate ceph config files for each ceph DS).
One of the issue I had was ONE ceph monitor script failed and because of that DS size was zero.
The reason for that error was in from format of keyfile used in the ONE ceph monitor script: that file (keyfile for ceph) needed to have just a key without any other information.