Deploying OpenNebula with Ceph Backend

Community Support
1

I am attempting to set up a new OpenNebula server and use a Ceph cluster as the backend resources for the OpenNebula VMs.

I have 5 servers/nodes running out of VMware:
opennebula01 - hosting OpenNebula front-end
ceph-controller01 - hosting the Ceph-mon/manager/metadata
ceph01 - hosting Ceph-osd (Nested Virtualization Enabled in VMware)
ceph02 - hosting Ceph-osd (Nested Virtualization Enabled in VMware)
ceph03 - hosting Ceph-osd (Nested Virtualization Enabled in VMware)

I have set up the ceph cluster without any issues and the cluster is healthy.

ceph -s

cluster:
id: { removed for the post }
health: HEALTH_OK

services:
mon: 1 daemons, quorum ceph-controller01 (age 23h)
mgr: ceph-controller01(active, since 23h)
mds: 1/1 daemons up
osd: 3 osds: 3 up (since 23h), 3 in (since 25h)

data:
volumes: 1/1 healthy
pools: 3 pools, 65 pgs
objects: 2.58k objects, 10 GiB
usage: 20 GiB used, 130 GiB / 150 GiB avail
pgs: 65 active+clean

ceph fs ls

name: cephfs, metadata pool: metadata, data pools: [data ]

I currently have the Ceph cluster filesystem mounted on the OpenNebula server but I just do not know what the next steps are and how I can integrate the Ceph cluster into OpenNebula itself. I have looked for documentation but there is very little and it is not documented well enough for me to understand what nodes need what packages, files, etc.

Any help would be greatly appreciated!

2

Everything is described here. This will lead you to a setup where you’ll use a ceph cluster to serve block storage to the virtual machines. Once you have the ceph cluster setup, you basically have to setup the OpenNebula frontend and nodes as ceph clients with the oneadmin user. The datastores will be contained within a ceph pool dedicated to OpenNebula.

If you want to use cephfs instead of rbd, then you’ll have to follow a setup similar to this one but mounting the cephfs instead of an nfs share. You’ll have to mount the fs in the datastore directory /var/lib/one/datastore/<ds_id> on each host and the frontend.

3

Thank you. After a lot of troubleshooting, I added the Ceph cluster as NFS and linked it to the datastore directories.