Ceph key rollover

Product Support Operations
1

Hello,

TL;DR: what is the recommended way of rotating keys/secrets for Ceph datastore?

Maybe you have noticed the new urgent Ceph release, which fixes some critical security bugs and for security reasons introduces a new type of Ceph secrets (aes256k), deprecating the original key type (aes):

I wonder how to replace a Ceph user key in a running OpenNebula cluster with many VMs without disrupting the service. On ceph-users mailing list, they recommend that each node should have its own Ceph user and secret. But I think in ONe there is only a global datastore parameter with a Ceph secret and Ceph username.

So how would you do a key rollover for a Ceph datastore in ONe?

Thanks for any recommendations,

-Yenya

2

I’ll try to add a little sketch based on the previous discussion on the ceph-users ML and my experience with opennebula:

  1. create a new libvirtd secret on nodes, let’s say the new username is libvirt-2026, the old one is libvirt. Let’s say the new secret UUID is …2 the old one is …1.

  2. Change the datastore in opennebula to refer to ceph secret …2 and use libvirt-2026. Running VMs are not affected, new VMs will now use the new key.

3.a) live migration

If opennebula on live migration updates the ceph secret + username, live migration to another host should “just fix it” - I doubt that opennebula does that and I assume that prior XML with the old username + old ceph secret reference are staying the same on live migration.

3.b) manual migration

I fear that a manual migration of existing VMs, including a possible downtime is required. This would mean:

  • edit the XML stored in the database to change the username + secret reference

  • then shutdown the VM (undeploy?)

  • redeploy the VM which probably uses the modified XML from the database (but does not actually re-read it from the datastore definition)

3.c) manual+live

  • Maybe (untested, unverified) the manual XML change in the database is taken into account on live migration and thus “only” the DB change is required and a live migration could fix the issue without downtime.

Both 3a, 3b, 3c need to be verified and are only sketches top of my head.

3

Hello,

We are aware of this issue and are currently working on identifying and testing an action plan that minimizes the service impact.

As soon as we have validated the recommended procedure, we will get back to you with the details.

Best regards,

4

Hi,
We have completed validation of the CephX key rollover procedure for OpenNebula.
The recommended approach is to rotate the existing client.libvirt credential in place, while keeping the existing libvirt secret UUID referenced by CEPH_SECRET.
Before starting, upgrade the Ceph cluster to a release containing the fix for CVE-2025-30156 (Squid 19.2.6+ or Tentacle 20.2.4+).
It is also necessary to upgrade the Ceph client packages on all OpenNebula nodes that authenticate to the Ceph cluster, including all KVM hypervisors. Older Ceph client versions do not support authentication using the new aes256k key type and will fail to authenticate after the credential is rotated.
The OpenNebula-specific rollover is:

  1. Rotate the existing credential:
ceph auth rotate --key-type=aes256k client.libvirt
  1. Distribute the updated client.libvirt keyring to all relevant OpenNebula nodes.
  2. On every KVM host, update the value of the existing libvirt secret UUID with the new key:
virsh secret-set-value \
  --secret <EXISTING_CEPH_SECRET_UUID> \
  --base64 "$(ceph auth get-key client.libvirt)"

The UUID itself should not be changed. Existing VM XML continues to reference the same UUID, so no bulk modification of VM definitions is required.
4. Verify RBD access with the new credential and live-migrate an existing VM between KVM hosts. A successful live migration confirms that the destination QEMU process can establish a new RBD connection using the rotated credential.
Running VMs do not need to be stopped during the rollover. However, VMs that were already running when the credential was rotated may continue using Ceph authentication state established with the previous credential.
As a conservative operational measure, we recommend live-migrating or restarting these VMs within 72 hours after the rotation. This recommendation is based on the default Ceph Monitor authentication ticket lifetime and does not mean that a VM is expected to lose access exactly after 72 hours.
After the OpenNebula credential has been validated, the remaining CephX client credentials should be migrated according to the upstream Ceph procedure. Legacy aes authentication should only be disabled after the complete CephX migration and the transition of the existing VMs have been completed.
Ceph upstream procedure: CephX Config Reference — Ceph Documentation

Best regards,