Lxc and Ceph Datastores

Community Support Storage
1

Hi ONE community.
In my current multi-node deployment I have a combination of KVM and LXC hypervisor nodes and I am interested in providing them with storage via the ceph solution (image and system datastores). All the servers of my cluster have Debian 11 operating system installed.

By reading the documentation of how to configure the ceph storage (Ceph Datastore — OpenNebula 6.2.1 documentation) I noticed that during the installation of the ceph secret in the hypervisor nodes the tool named virsh of the libvirt library is used. Although in the installation of the KVM hypervisor node the libvirt comes with the package “opennebula-node-kvm”, the same doesn’t happen in terms of the “opennebula-node-lxc” package.

So I would like to know how I can configure ceph storage in LXC hypervisor nodes.

  1. Should I install libvirt manually or even libvirt-daemon-driver-lxc package too?

  2. In the above case should I also replace the URI “qemu:///system” with the URI “lxc:///system” when I use virsh command?

2

In the LXC node the requirement for the driver to operate with Ceph is to have the rbd-nbd utility installed. This utility depends on libvirt, so you should be good to go with having the node as a Ceph client, as mentioned in the datastore documentation you linked.

However, the opennebula-node-lxc package should have had that dependency referenced like the opennebula-node-lxd package.

root@minione-6:~# apt show opennebula-node-lxd
Package: opennebula-node-lxd
Version: 6.2.1-1
Priority: Priority
Section: extra
Source: opennebula
Maintainer: OpenNebula Team <contact@opennebula.io>
Installed-Size: 220 kB
Pre-Depends: snapd
Depends: opennebula-common (= 6.2.1-1), kpartx, libvncserver1, e2fsprogs, xfsprogs, qemu-utils, adduser, ruby, vlan, ipset, pciutils, rsync, tar, cron, ruby-sqlite3
Suggests: rbd-nbd
Conflicts: lxd, lxd-client, opennebula-lxd-snap, opennebula-node-firecracker, opennebula-node-kvm, opennebula-node-lxc
Replaces: lxd, lxd-client, opennebula-lxd-snap
Homepage: http://opennebula.org/
Download-Size: 23.8 kB
APT-Sources: https://enterprise.opennebula.io/repo/6.2/Ubuntu/20.04 stable/opennebula amd64 Packages
Description: Services for OpenNebula LXD node (Enterprise Edition)

Note the Suggests: rbd-nbd, its missing on the lxc node package.

Thanks for the feedback, we will update the packages and the documentation to explicitly point this dependency out.

3

The rbd-nbd package doesn’t have the libvirt as depedency (neither libvirt-client nor libvirt-daemon-system packages are listed). So how are they related?

root@node1:~$ apt show rbd-nbd
Package: rbd-nbd
Version: 16.2.7-1~bpo11+1
Priority: optional
Section: admin
Source: ceph
Maintainer: Ceph Maintainers <ceph-maintainers@lists.ceph.com>
Installed-Size: 513 kB
Depends: libc6 (>= 2.28), libgcc-s1 (>= 3.0), libnl-3-200 (>= 3.2.7), libnl-genl-3-200 (>= 3.2.21), librados2, librbd1, libstdc++6 (>= 9)
Homepage: http://ceph.com/
Download-Size: 153 kB
APT-Sources: https://download.ceph.com/debian-pacific bullseye/main amd64 Packages
Description: NBD-based rbd client for the Ceph distributed file system
 Ceph is a massively scalable, open-source, distributed
 storage system that runs on commodity hardware and delivers object,
 block and file system storage.  This is a
 NBD-based client that allows one to map Ceph rbd images as local
 block device.
 .
 NBD base client that allows one to map Ceph rbd images as local
 block device.

N: There is 1 additional record. Please use the '-a' switch to see it```
4

It will install it as an indirect dependency. Please take a look at

root@minione-6:~# apt install rbd-nbd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  adwaita-icon-theme at-spi2-core cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-x gtk-update-icon-cache hicolor-icon-theme humanity-icon-theme
  ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavc1394-0 libbrlapi0.7 libcaca0 libcacard0 libcairo-gobject2
  libcdparanoia0 libcolord2 libdv4 libepoxy0 libfdt1 libgbm1 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgstreamer-plugins-base1.0-0
  libgstreamer-plugins-good1.0-0 libgtk-3-0 libgtk-3-bin libgtk-3-common libiec61883-0 libjack-jackd2-0 libjbig0 liblcms2-2 libmp3lame0 libmpg123-0 libnspr4 libnss3 libopus0
  liborc-0.4-0 libpciaccess0 libpcsclite1 libpmem1 libraw1394-11 librest-0.7-0 librsvg2-2 librsvg2-common libsamplerate0 libshout3 libslirp0 libsoup-gnome2.4-1 libspeex1
  libspice-server1 libtag1v5 libtag1v5-vanilla libtheora0 libtiff5 libtwolame0 libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvirt-clients libvirt-daemon
  libvirt-daemon-driver-qemu libvirt-daemon-driver-storage-rbd libvirt-daemon-system libvirt-daemon-system-systemd libvirt0 libvisual-0.4-0 libvpx6 libvte-2.91-0
  libvte-2.91-common libwavpack1 libwayland-client0 libwayland-cursor0 libwayland-egl1 libwayland-server0 libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxi6 libxinerama1
  libxkbcommon0 libxml2-utils libxrandr2 libxtst6 libxv1 libyajl2 msr-tools ovmf qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 seabios
  ubuntu-mono x11-common
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  libnl-genl-3-200
The following NEW packages will be installed:
  libnl-genl-3-200 rbd-nbd
0 upgraded, 2 newly installed, 0 to remove and 103 not upgraded.
Need to get 146 kB of archives.
After this operation, 553 kB of additional disk space will be used.
Do you want to continue? [Y/n]
5

I see that the libvirt packages are listed in the ‘remove’ list. What package did you remove and they are no longer needed?

6

I am sending the result of the proposed installation steps.
The lbvirt packages are not delivered.

root@node1:~$ apt-get install rbd-nbd
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
rbd-nbd is already the newest version (14.2.21-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@node1:~$ apt-get install opennebula-node-lxc
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
opennebula-node-lxc is already the newest version (6.2.0-1.ce).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@node1:~$ virsh -v
bash: virsh: command not found