thank you for write up! I think that it is useful for other users!
Can you please clarify more this part for me:
If you plan to create a
LoadBalancer , it should have an IP for being able to interact with it, this way the community could write controllers (like the openstack one).
If I am understanding correctly the loadbalancer in your case is kube-api running inside the kubernetes appliance directly. The idea is to have generic loadbalancer in VNF appliance.
Your deployment could be automated by teaching VNF loadbalancing with the support for onegate (there is already keepalived for vrrp which could do loadbalancing as Kristian proposed) and deploy it together with Kubernetes appliance as a service template (as kubernetes now can be). The keepalived/loadbalancer config could even be dynamic enough (by polling the state of onegate) as is done now similarly for SDNAT.
When all this is in place (loadbalancer in VNF, support for onegate and deployment of Kubernetes and VNF as a service) then we still need to implement at least a stub of cloud-provider which will be able to feed onegate. The kube-api might still be needed as the complement to the VNF.
IMHO a simple hacking inside one or two appliances is not enough there must be some support directly inside the OpenNebula itself - to provide API - config in onegate.
Static loadbalancing as Kristian wanted can still be supported but it would be useless for this use-case.
If I misunderstood something then let me know.
EDIT: Actually when SDNAT is already there then that dynamic part is probably already taken care off… What is missing is the preconfigured pool of ip addresses and loadbalancer.