tl;dr: it seems there is safe no way to combine KVM and LXD on one host at the moment.
edit; it should work if you just put had placement policies for all LXD and all KVM templates.
edit2: you can also try to use clusters to imply a 50% (given in kb) memory reservation. I ended up with a host wih 700TB RAM instead of 64GB RAM.
If you attempt to try this. only do it if you have the time to set up cgroups shielding the base OS from all the containers since you will hit OOM issues and random crap. Containers that are out of memory sometimes won’t shut down etc.
Honestly, just don’t try this at all.
I got a test setup up and running, can schedule KVM and LXD instances, have qcow2 support etc.
the mis-scheduling does in fact happen.
i was missing a clear info that this causes the instances to be unbootable, they’ll have an invalid disk attachment.
so what do we learn:
1) on CentOS8, LXD is a no-go for OpenNebula at the moment, since it relies on the ubuntu snap packages which require the users home to be under /home (/var/lib/one is not there). I’m sure it could be adjusted with some amount of effort, but by default the two default assumptions are incompatible.
2) LXD and KVM co-existence is not possible, since the resulting system does not just have cosmetic or scalability errors, but in fact is not reliable.
3) as a user I would request that it is made more clear in such blog posts if they have resulted in something that just has rough edges and cosmetic errors, or is unreliable at it’s core, and the post itself is more exploratory, i.e. to gauge user interest.
*4) this is a common problem with the ONE blog. i would suggest that maybe two people work on these posts, one makes the content and a second one tests stuff and writes the summary. that way the reader can check if they can use 5 hours to make it work, or 20 hours to do an exercise left for the reader, or if they should actually just spend 0 hours and simply take note that it doens’t work yet. *
in the end that’ll mean people will end up with more time to invest in contribs.