The performance of Ceph when used in VMs

[Khang_Nguyen_Phuc]

Hello everyone,
I use OpenNebula with Ceph datastore for disk; however, when creating a VM (or attach disk) with a disk stored on Ceph Datastore, the performance is less than 1/3 compared to a disk on the local datastore. I have tested directly mounting an RBD disk on the KVM host, and the performance is equivalent to the VM’s disk on the local datastore. Could OpenNebula be imposing some limitations on Ceph when run with VMs?
Below are the results when I mount a Ceph RBD image onto the KVM host:

Here are the results when I create a VM and benchmark the VM’s disk (using the same Ceph pool as the image above):

Thank you,
Note: Opennebula 6.6.0 and kvm host Ubuntu 22.04

[jorel (Jan Orel)]

Well yes, there is a performance penalty when using rbd via libvirt layer.

What only comes to my mind is experimenting with cache options.

It would be also interesting if you could do some experiments with kvm/qemu num-queues but this perhaps needs to be done directly in libvirt xml.

And maybe also changing the I/O scheduler on the VM.

[Khang_Nguyen_Phuc]

I adjusted the virtio-scsi queues to match the number of CPUs and VCPUs, and added a disk with the write-back cache. However, the results are only slightly faster than before. I know that using Ceph through the libvirt layer can reduce performance, but is a 1/3 decrease considered significant?(Or did I miss configuring something?)
Result:

Thank you,