Ceph error monitoring datastore

Community Support
1

Hi! I have ceph cluster and want use it with opennebula but every time have problem with ceph + opennebula.

Opennebula don’t see ceph storage. In log I have:

 Command execution fail: /var/lib/one/remotes/datastore/ceph/monitor PERTX0RSSVZFUl9BQ1RJT05fREFUQT48REFUQVNUT1JFPjxJRD4xMDA8L0lEPjxVSUQ+MDwvVUlEPjxHSUQ+MD
wvR0lEPjxVTkFNRT5vbmVhZG1pbjwvVU5BTUU+PEdOQU1FPm9uZWFkbWluPC9HTkFNRT48TkFNRT5jZXBoZHM8L05BTUU+PFBFUk1JU1NJT05TPjxPV05FUl9VPjE8L09XTkVSX1U+PE9XTkVSX00+MTwvT1dORVJfTT48T1dORVJfQT4wPC9PV05FUl9BPjx
HUk9VUF9VPjE8L0dST1VQX1U+PEdST1VQX00+MDwvR1JPVVBfTT48R1JPVVBfQT4wPC9HUk9VUF9BPjxPVEhFUl9VPjA8L09USEVSX1U+PE9USEVSX00+MDwvT1RIRVJfTT48T1RIRVJfQT4wPC9PVEhFUl9BPjwvUEVSTUlTU0lPTlM+PERTX01BRD48IVtD
REFUQVtjZXBoXV0+PC9EU19NQUQ+PFRNX01BRD48IVtDREFUQVtjZXBoXV0+PC9UTV9NQUQ+PEJBU0VfUEFUSD48IVtDREFUQVsvdmFyL2xpYi9vbmUvL2RhdGFzdG9yZXMvMTAwXV0+PC9CQVNFX1BBVEg+PFRZUEU+MDwvVFlQRT48RElTS19UWVBFPjM8L
0RJU0tfVFlQRT48U1RBVEU+MDwvU1RBVEU+PENMVVNURVJTPjxJRD4wPC9JRD48L0NMVVNURVJTPjxUT1RBTF9NQj4wPC9UT1RBTF9NQj48RlJFRV9NQj4wPC9GUkVFX01CPjxVU0VEX01CPjA8L1VTRURfTUI+PElNQUdFUz48L0lNQUdFUz48VEVNUExBVE
U+PEJSSURHRV9MSVNUPjwhW0NEQVRBWzE3Mi4xNi4xMC4yMTEgMTcyLjE2LjEwLjIxMl1dPjwvQlJJREdFX0xJU1Q+PENFUEhfSE9TVD48IVtDREFUQVsxNzIuMTYuMTAuMjExOjY3ODkgMTcyLjE2LjEwLjIxMjo2Nzg5XV0+PC9DRVBIX0hPU1Q+PENFUEh
fU0VDUkVUPjwhW0NEQVRBW2UwNzAwZDUwLWIzMzgtNDc1OC1hNTk3LTg5NWU0MTY3Y2I2Yl1dPjwvQ0VQSF9TRUNSRVQ+PENFUEhfVVNFUj48IVtDREFUQVtvbmVhZG1pbl1dPjwvQ0VQSF9VU0VSPjxDTE9ORV9UQVJHRVQ+PCFbQ0RBVEFbU0VMRl1dPjwv
Q0xPTkVfVEFSR0VUPjxESVNLX1RZUEU+PCFbQ0RBVEFbUkJEXV0+PC9ESVNLX1RZUEU+PERTX01BRD48IVtDREFUQVtjZXBoXV0+PC9EU19NQUQ+PExOX1RBUkdFVD48IVtDREFUQVtOT05FXV0+PC9MTl9UQVJHRVQ+PFBPT0xfTkFNRT48IVtDREFUQVtvb
mVdXT48L1BPT0xfTkFNRT48VE1fTUFEPjwhW0NEQVRBW2NlcGhdXT48L1RNX01BRD48L1RFTVBMQVRFPjwvREFUQVNUT1JFPjwvRFNfRFJJVkVSX0FDVElPTl9EQVRBPg== 100
Thu May 26 13:34:05 2016 [Z0][ImM][I]: ExitCode: 255
Thu May 26 13:34:05 2016 [Z0][ImM][E]: Error monitoring datastore 100: -
Thu May 26 13:34:05 2016 [Z0][ImM][I]: Command execution fail: /var/lib/one/remotes/datastore/ceph/monitor PERTX0RSSVZFUl9BQ1RJT05fREFUQT48REFUQVNUT1JFPjxJRD4xMDE8L0lEPjxVSUQ+MDwvVUlEPjxHSUQ+MD
wvR0lEPjxVTkFNRT5vbmVhZG1pbjwvVU5BTUU+PEdOQU1FPm9uZWFkbWluPC9HTkFNRT48TkFNRT5jZXBoZHMxPC9OQU1FPjxQRVJNSVNTSU9OUz48T1dORVJfVT4xPC9PV05FUl9VPjxPV05FUl9NPjE8L09XTkVSX00+PE9XTkVSX0E+MDwvT1dORVJfQT4
8R1JPVVBfVT4xPC9HUk9VUF9VPjxHUk9VUF9NPjA8L0dST1VQX00+PEdST1VQX0E+MDwvR1JPVVBfQT48T1RIRVJfVT4wPC9PVEhFUl9VPjxPVEhFUl9NPjA8L09USEVSX00+PE9USEVSX0E+MDwvT1RIRVJfQT48L1BFUk1JU1NJT05TPjxEU19NQUQ+PCFb
Q0RBVEFbY2VwaF1dPjwvRFNfTUFEPjxUTV9NQUQ+PCFbQ0RBVEFbY2VwaF1dPjwvVE1fTUFEPjxCQVNFX1BBVEg+PCFbQ0RBVEFbL3Zhci9saWIvb25lLy9kYXRhc3RvcmVzLzEwMV1dPjwvQkFTRV9QQVRIPjxUWVBFPjA8L1RZUEU+PERJU0tfVFlQRT4zP
C9ESVNLX1RZUEU+PFNUQVRFPjA8L1NUQVRFPjxDTFVTVEVSUz48SUQ+MDwvSUQ+PC9DTFVTVEVSUz48VE9UQUxfTUI+MDwvVE9UQUxfTUI+PEZSRUVfTUI+MDwvRlJFRV9NQj48VVNFRF9NQj4wPC9VU0VEX01CPjxJTUFHRVM+PC9JTUFHRVM+PFRFTVBMQV
RFPjxCUklER0VfTElTVD48IVtDREFUQVsxNzIuMTYuMTAuMjExIDE3Mi4xNi4xMC4yMTJdXT48L0JSSURHRV9MSVNUPjxDRVBIX0hPU1Q+PCFbQ0RBVEFbMTcyLjE2LjEwLjIxMTo2Nzg5IDE3Mi4xNi4xMC4yMTI6Njc4OV1dPjwvQ0VQSF9IT1NUPjxDRVB
IX1NFQ1JFVD48IVtDREFUQVtlMDcwMGQ1MC1iMzM4LTQ3NTgtYTU5Ny04OTVlNDE2N2NiNmJdXT48L0NFUEhfU0VDUkVUPjxDRVBIX1VTRVI+PCFbQ0RBVEFbbGlidmlydF1dPjwvQ0VQSF9VU0VSPjxDTE9ORV9UQVJHRVQ+PCFbQ0RBVEFbU0VMRl1dPjwv
Q0xPTkVfVEFSR0VUPjxESVNLX1RZUEU+PCFbQ0RBVEFbUkJEXV0+PC9ESVNLX1RZUEU+PERTX01BRD48IVtDREFUQVtjZXBoXV0+PC9EU19NQUQ+PExOX1RBUkdFVD48IVtDREFUQVtOT05FXV0+PC9MTl9UQVJHRVQ+PFBPT0xfTkFNRT48IVtDREFUQVtvb
mVdXT48L1BPT0xfTkFNRT48VE1fTUFEPjwhW0NEQVRBW2NlcGhdXT48L1RNX01BRD48VFlQRT48IVtDREFUQVtJTUFHRV9EU11dPjwvVFlQRT48L1RFTVBMQVRFPjwvREFUQVNUT1JFPjwvRFNfRFJJVkVSX0FDVElPTl9EQVRBPg== 101
Thu May 26 13:34:05 2016 [Z0][ImM][I]: ExitCode: 255
Thu May 26 13:34:05 2016 [Z0][ImM][E]: Error monitoring datastore 101: -

[root@node01 one]# onedatastore list
ID NAME SIZE AVAIL CLUSTERS IMAGES TYPE DS TM STAT
0 system 50G 95% 0 0 sys - shared on
1 default 50G 95% 0 0 img fs shared on
2 files 50G 95% 0 0 fil fs ssh on
100 cephds 0M - 0 0 img ceph ceph on
101 cephds1 0M - 0 0 img ceph ceph on

What I do wrong? Use manual from http://docs.opennebula.org/5.0/deployment/open_cloud_storage_setup/ceph_ds.html

2

I try fresh install 3 times. Used ver. 4.14.2 and 4.90 but every time I got error:

Sat May 28 23:48:25 2016 [Z0][ImM][I]: /var/lib/one/remotes/tm/ceph/…/…/datastore/libfs.sh: line 252: RANDOM % 0: division by 0 (error token is “0”)
Sat May 28 23:48:25 2016 [Z0][ImM][E]: Datastore template missing ‘BRIDGE_LIST’ attribute.
Sat May 28 23:48:25 2016 [Z0][ImM][I]: ExitCode: 255
Sat May 28 23:48:25 2016 [Z0][ImM][E]: Error monitoring datastore 100: Datastore template missing ‘BRIDGE_LIST’ attribute.

But I have BRIDGE_LIST!
Please help me.
Thanks!

3

Can you send the output of:

$ onedatastore show 100
4

root@node01:/var/log/one# onedatastore show 103
DATASTORE 103 INFORMATION
ID : 103
NAME : cephds
USER : oneadmin
GROUP : oneadmin
CLUSTERS : 0
TYPE : IMAGE
DS_MAD : ceph
TM_MAD : ceph
BASE PATH : /var/lib/one//datastores/103
DISK_TYPE : RBD
STATE : READY

DATASTORE CAPACITY
TOTAL: : 0M
FREE: : 0M
USED: : 0M
LIMIT: : -

PERMISSIONS
OWNER : um-
GROUP : u–
OTHER : —

DATASTORE TEMPLATE
BRIDGE_LIST=“172.16.10.211 172.16.10.212”
CEPH_HOST=“172.16.10.211 172.16.10.212”
CEPH_SECRET=“c1e33b0a-860a-4230-b19b-368eebbb87f4”
CEPH_USER=“libvirt”
CLONE_TARGET=“SELF”
DISK_TYPE=“RBD”
DS_MAD=“ceph”
LN_TARGET=“NONE”
POOL_NAME=“one”
RESTRICTED_DIRS=“/”
SAFE_DIRS=“/var/tmp”
TM_MAD=“ceph”

IMAGES

5

Below are my comments based on your setup / configuration:

  1. I dont recommend using Ceph’s MON server for bridge list also. In my setup, i just use the few of the ONE’s node
  2. Does the ceph authentication has been setup properly? Please verify you can run the following command:

rbd --id libvirt ls -l -p one

You may refer my guide also and see if it help resolve the issue.

6

Hi! I read you guide. Fresh install - and all ok!! ))) Thanks!
But I can’t understand how use ceph as systemDS. ImageDS work good.
Thank you very much!

7

Great to hear the problem is solved. For System DS, you may need to use NFS or CephFS. But if your ONE is 5.0 beta, you may try System DS on ceph directly - this i have not tested