Host with LXD and KVM support

Community Support
1

Hi,

I’m testing a standalone OpenNebula setup at the moment, one hypervisor node, one high-end ScaleFlux SSD. I’ve got one large ext4 filesystem as the datastore here.

I’d like to use be able to use the same physical box with both LXD and KVM.
That’s mostly because it makes sense to have the choice, but also because I want to be able to compare the IO performance of virtual and non-virtual in a nice automated way.

But I’m a bit confused at the moment how to proceed…

Should I add the server under two names with two different drivers?
Do I need to think about reservations?

I know I could run a VM for the LXD containers, but that would take away a lot of the benefit in terms of performance & footprint.

Versions of the related components and OS (frontend, hypervisors, VMs):
CentOS 8 / ONE 5.10.1

Steps to reproduce:

Current results:
LXD support much improved but question like this can be found only unanswered in forum search

Expected results:
advice how to proceed :slight_smile:

2

Maybe you can find some hints here:

https://opennebula.io/using-lxd-and-kvm-on-the-same-host/

3

Thank you! I think that’ll be enough to get started.

Not a straight-forward thing it seems.
The post lists quite a lot of little problems with the coexistence in ONE that could all be solved but currently create the typical drag (toil in Google’s terms) that makes it unpleasant to use. I hope I’ll not hit all of them.
Also it’s not encouraging to see the status of the issue mentioned in the blog post. Do I extrapolate this is not gonna disappear any time soon or should I keep hope?

4

Hi @darkfader, if you sort the issues by emojis you’ll see there is some interest on the issue, however doing it requries significant changes in a lot of OpenNebula’s internals. We will implement it if we see a lot of interest on the issue.

5

Reminds me of the Xen strategy.
It does not work and we know what it does to a product.

6

tl;dr: it seems there is safe no way to combine KVM and LXD on one host at the moment.

edit; it should work if you just put had placement policies for all LXD and all KVM templates.
edit2: you can also try to use clusters to imply a 50% (given in kb) memory reservation. I ended up with a host wih 700TB RAM instead of 64GB RAM.

If you attempt to try this. only do it if you have the time to set up cgroups shielding the base OS from all the containers since you will hit OOM issues and random crap. Containers that are out of memory sometimes won’t shut down etc.
Honestly, just don’t try this at all.

I got a test setup up and running, can schedule KVM and LXD instances, have qcow2 support etc.
the mis-scheduling does in fact happen.
i was missing a clear info that this causes the instances to be unbootable, they’ll have an invalid disk attachment.

so what do we learn:

1) on CentOS8, LXD is a no-go for OpenNebula at the moment, since it relies on the ubuntu snap packages which require the users home to be under /home (/var/lib/one is not there). I’m sure it could be adjusted with some amount of effort, but by default the two default assumptions are incompatible.

2) LXD and KVM co-existence is not possible, since the resulting system does not just have cosmetic or scalability errors, but in fact is not reliable.

3) as a user I would request that it is made more clear in such blog posts if they have resulted in something that just has rough edges and cosmetic errors, or is unreliable at it’s core, and the post itself is more exploratory, i.e. to gauge user interest.

*4) this is a common problem with the ONE blog. i would suggest that maybe two people work on these posts, one makes the content and a second one tests stuff and writes the summary. that way the reader can check if they can use 5 hours to make it work, or 20 hours to do an exercise left for the reader, or if they should actually just spend 0 hours and simply take note that it doens’t work yet. *

in the end that’ll mean people will end up with more time to invest in contribs.

7

I don’t think OpenNebula supports the just Released LXD v4.0 LTS yet but fyi:

LXD v3.19 and LXD v4.0 LTS introduced a major new feature where LXD now supports both LXC containers and Virtual Machines without using KVM, VirtualBox etc.

This is another writeup:

brian

1 Like
8

Hello @bmullan, thanks for the update!! :nerd_face: --> A new Issue has been opened.

1 Like