Deploy ceph for lxd node to use in nebula

Community Support Storage
1

hello guys, I was trying to add ceph storage in the frontend node with onedatastore , but unfortunately , finally, datastore didn’t get a size, now I wanna be trying to configure new storage.
ceph version 12.2.13 luminous
I add lxd node as a ceph client and access to ceph cluster,and after that create pool by lxd init
and now I wanna to be add vm in frontend check storage .
Please share with me if you have any experience ?
Do you think this is the right way to go?

2

Hello

Don’t use the LXD cli to setup a ceph storage pool at the LXD level. The way the LXD driver works doesn’t require that. The driver allows the container to use a rados block device from the datastore without any setup on the LXD side. Please follow the Ceph Datastore guide and the only thing specific for LXD is this. You should be able to run containers on ceph with that.

3

Hello @dclavijo thanks for replay
actually, I decided to setup ceph by onedatastore but unfortunately No matter how hard I try, I still can’t succeed .
I have done the following steps
in ceph node

$ceph -s
  cluster:
    id:     afdf7077-b549-4aa6-865b-7a9e99da0811
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum cephmon1,cephmon2,cephmon3
    mgr: cephadm(active), standbys: cephosd1
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   3 pools, 224 pgs
    objects: 371 objects, 1.20GiB
    usage:   2.70GiB used, 2.33TiB / 2.33TiB avail
    pgs:     224 active+clean

$ceph osd lspools
0 rbd,1 one,2 lxd,

$ ceph auth list

client.libvirt
	key: AQBL16deEWnmDhAAzr8H5KTO+ngs0+OjyDVJXQ==
	caps: [mon] profile rbd
	caps: [osd] profile rbd pool=one

in frontend node and lxdnode by oneadmin user
generate UUID for cephuser in this node and put on the sercret.xml

$ cat secret.xml
cat secret.xml 
<secret ephemeral='no' private='no'>
  <uuid>e392d7c8-c74e-4282-b826-afc466263f1c</uuid>
  <usage type='ceph'>
          <name>client.libvirt AQBL16deEWnmDhAAzr8H5KTO+ngs0+OjyDVJXQ==</name>
  </usage>
</secret>

$virsh -c qemu:///system secret-define secret.xml
$virsh -c qemu:///system secret-set-value --secret $UUID --base64 $(cat client.libvirt.key

$cat cephds.one
cat ceph.one 
NAME    = ceph_system
TM_MAD  = ceph
TYPE    = SYSTEM_DS
POOL_NAME   = one
CEPH_HOST   = "192.168.1.35:6789 192.168.1.36:6789 192.168.1.37:6789"
CEPH_USER   = libvirt
CEPH_SECRET = "e392d7c8-c74e-4282-b826-afc466263f1c"
BRIDGE_LIST = cephmon

unfortunately after create datastore can not get size

$onedatastore show 166
DATASTORE 166 INFORMATION                                                       
ID             : 166                 
NAME           : ceph_system         
USER           : oneadmin            
GROUP          : oneadmin            
CLUSTERS       : 0                   
TYPE           : SYSTEM              
DS_MAD         : -                   
TM_MAD         : ceph                
BASE PATH      : /var/lib/one//datastores/166
DISK_TYPE      : FILE                
STATE          : READY               

DATASTORE CAPACITY                                                              
TOTAL:         : 0M                  
FREE:          : 0M                  
USED:          : 0M                  
LIMIT:         : -                   

PERMISSIONS                                                                     
OWNER          : um-                 
GROUP          : u--                 
OTHER          : ---                 

DATASTORE TEMPLATE                                                              
ALLOW_ORPHANS="mixed"
BRIDGE_LIST="192.168.1.35"
CEPH_HOST="192.168.1.35:6789 192.168.1.36:6789 192.168.1.37:6789"
CEPH_SECRET="e392d7c8-c74e-4282-b826-afc466263f1c"
CEPH_USER="libvirt"
DISK_TYPE="FILE"
DS_MIGRATE="NO"
POOL_NAME="one"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
SHARED="YES"
TM_MAD="ceph"
TYPE="SYSTEM_DS"

I don’t know where the problem is and also can not find logs for this .

Thanks.