Ceph rbd datastore is unavailable but libvirt is OK

Community Support
1

Hi All,

I encounter the problem - ceph datastore is unavailable in spite of the fact that I can access it by libvirt or qemu-img directly.

Libvirt is OK:

[root@one-sandbox ceph]# virsh pool-list rbd
Name             State       Autostart
testrbdpool      active     no

[root@one-sandbox ceph]# virsh pool-info testrbdpool
Name:           testrbdpool
UUID:           fa73e15b-a70f-3570-a418-eb57f1a8d092
State:          running
Persistent:     yes
Autostart:      no
Capacity:       43.56 GiB
Allocation:     145.00
Available:      38.75 GiB

But Opennebula does not find any space there:

[oneadmin@one-sandbox ~]$ onedatastore list
ID NAME     SIZE AVAIL CLUSTER  IMAGES TYPE DS      TM      STAT
0  system   9.8G  61%   -       0      sys   -     shared  on
1  default  9.8G  64%   -       1      img   fs    shared  on
2  files    9.8G  64%   -       0      fil   fs    ssh     on
100 cephds  0M -     -          0      img   ceph  ceph    on

Opennebula ceph-based datastore config is:

[oneadmin@one-sandbox ~]$ onedatastore show 100
DATASTORE 100 INFORMATION
ID             : 100
NAME           : cephds
USER           : oneadmin
GROUP          : oneadmin
CLUSTER        : -
TYPE           : IMAGE
DS_MAD         : ceph
TM_MAD         : ceph
BASE PATH      : /var/lib/one//datastores/100
DISK_TYPE      : RBD
STATE          : READY

DATASTORE CAPACITY
TOTAL:         : 0M
FREE:          : 0M
USED:          : 0M
LIMIT:         : -

PERMISSIONS
OWNER          : um-
GROUP          : u--
OTHER          : ---

DATASTORE TEMPLATE
BASE_PATH="/var/lib/one//datastores/"
BRIDGE_LIST="ceph1 ceph2 ceph3"
CEPH_HOST="ceph1 ceph2 ceph3"
CEPH_SECRET="f45cf98c-9d22-45f6-a8f5-7710c0412c03"
CEPH_USER="libvirt"
CLONE_TARGET="SELF"
DISK_TYPE="RBD"
DS_MAD="ceph"
LN_TARGET="NONE"
POOL_NAME="testrbdpool"
TM_MAD="ceph"
TYPE="IMAGE_DS"

More info about ceph related config:

[root@one-sandbox ceph]# cat /etc/libvirt/secrets/f45cf98c-9d22-45f6-a8f5-7710c0412c03.xml
<secret ephemeral='no' private='no'>
  <uuid>f45cf98c-9d22-45f6-a8f5-7710c0412c03</uuid>
  <usage type='ceph'>
    <name>client.libvirt secret</name>
  </usage>
</secret>

[root@one-sandbox ceph]# ll /etc/ceph
total 16
-rw-r--r-- 1 qemu libvirt  71 May 21 16:19 ceph.client.admin.keyring
-rw-r--r-- 1 qemu libvirt  73 May 21 16:10 ceph.client.libvirt.keyring
-rw-r--r-- 1 qemu libvirt 574 May 21 16:07 ceph.conf

I don’t undestand what’s wrong. Opennebula uses libvirtd, if I’m correct. Libvirt is OK.
Are there any special parameters missing in Opennebula config? Please, help.

Dmitry Yermak

2

it looks like libvirt is allowed to use all necessary ceph-commands, but the oneadmin user is not allowed to do the same.
You probably need to add oneadmin to a ceph group ? (havent played with ceph yet…)

3

thank you,

the wrong oneadmin rights was the first thing I suspected to be the culprit. It looks like I fixed the problem now. There was only one server with correct libvirt and qemu-img in all the hosts listed in BRIDGE_LIST. That’s one where I run libvirt successfully :smiley:

After I reinstalled libvirt & qemu-img with rbd support enabled everywhere the problem has gone.