OneKE 1.33: FALLBACK_GW/DNS not available in master or worker VMs

Integration Support Marketplace Appliances
1

Please, describe the problem here and provide additional information below (if applicable) …

When I import “Service OneKE 1.33” from Apps without any changes to it and deploy it, virtual router + master/worker VMs works without issues. When I change something in the service template eg. increase deploy of storage role to “1” or something else and than try to deploy it I end up without internet connection for master/worker/storage VMs because I am dependent to FALLBACK_GW variable to connect to running vnr from same service for these VMs. I tried to set FALLBACK_GW=$ONEAPP_VROUTER_ETH1_VIP0 in VM master template, and than set it when deploying but that resulted in “Invalid IPv4/port for VNF/HAPROXY/0, aborting..” in master vm logs.

Versions of the related components and OS (frontend, hypervisors, VMs):

FE - opennebula 7.0.1-1 amd64 OpenNebula Server and Scheduler (Community Edition)
KVM server - opennebula-node-kvm 7.0.1-1 all Services for OpenNebula KVM node (Community Edition)
Multi-master Kubernetes 1.33 cluster for KVM hosts, orchestrated by OneFlow

Steps to reproduce:
1. Import “Service OneKE 1.33” from Apps
2. deploy service without adding/changing values in all inputs but enable all K8S services like cilium, multus etc. but I don’t think it’s this cuz even without changing anything it’s the same result…
3. wait for vnr, master, worker to run and configure - everything OK cuz connection to oneflow and the internet is working
4. change something in the service template, assign virtual network to the Public/Private or change deploy number for role, anything
5. deploy “new” and “updated” service same as in 2. and now there is no connection for master/worker VMs

Current results:
master/worker VM context - “FALLBACK_GW”: “”,

Expected results:
master/worker VM context - “FALLBACK_GW”: “10.21.0.2”,**
**
It’s driving me crazy, I would be glad to just use some workaround, I wanna deploooooooooy without manually adding gws to the master/worker nodes.

2

Hello @exoman,

Apologies for the late reply and welcome to this forum.

This happens because FALLBACK_GW / FALLBACK_DNS are only auto-populated when the OneKE service is deployed exactly as originally defined.

If you modify the service structure (change role counts, networks, etc.), OneFlow may no longer resolve variables like:

$ONEAPP_VROUTER_ETH1_VIP0

As a result, FALLBACK_GW ends up empty in master/worker VMs.

So the suggestion would be that instead of relying on automatic injection, define them explicitly as service inputs:

INPUTS = [
  FALLBACK_GW = "",
  FALLBACK_DNS = ""
]

Then reference them in the role templates:

CONTEXT = [
  NETWORK = "YES",
  FALLBACK_GW = "$FALLBACK_GW",
  FALLBACK_DNS = "$FALLBACK_DNS"
]

This makes the configuration independent of role ordering or service layout changes.

You can find more info here:

Hope it helps.

Cheers,

3

Hello @FrancJP

thank you for the reply.

We are just spinning up VMs and using something else to set them up in the end. But for the hell of it let’s say OK, I add both inputs to the service template, and then use them in VM templates of the master/worker roles. How do I set them to “eth1” network interface IP of vnr ? Because vnr gets IP on “eth1” from nebula from local IP pool, that’s why I tried $ONEAPP_VROUTER_ETH1_VIP0 but that ended up with error I mentioned on my first post. Is there a way to get VM parameters from oneflow and set variables to them before spinning up rest of the roles ? just like it is when is template imported ?

There are really 2 options, for me atleast :slight_smile: -

  1. set “eth1” interface manually, then I can set to this variable other variables I need so something like - ETH1_IF_IP → FALLBACK_GW=ETH1_IF_IP etc. I was hoping that $ONEAPP_VROUTER_ETH1_VIP0 will work but it seems that is used somewhere else and it breaks something when I set it up
  2. get IP from running vnr from oneflow before creating other roles, that way I don’t need to set anything manually - vnr up →get eth1 IP → set FALLBACK_GW to it, like in clean imported service
4

Hello @exoman,

My colleagues pointed out to this issue on the repo:
Sunstone - Updating Service Template.

So it looks like you basically hit a bug, that it’s under review. My suggestion is that you also comment there so engineering can review it.

Regards,