I’m starting to read the files inside /etc/one-appliance/service.d/appliance/ which made me question the ability for OpenNebula to install rancher in an air gap environment.
Does OpenNebula is supposed to be table to install Rancher in an air gap environment ?
If so, how ? I haven’t found any clue online.
Thanks!
Versions of the related components and OS (frontend, hypervisors, VMs):
OpenNebula 6.2
OneKE 1.27
Steps to reproduce:
Install OneKE 1.27 in an airgap environnement.
Does OpenNebula is supposed to be table to install Rancher in an air gap environment ?
Yes, but with the caveat that the latest image OneKE’s Features / Changelog — OpenNebula 6.8.0 documentation does not include calico and canal CNI plugins, if you want to use them they have to be downloaded. Could you share your OneFlow config maybe so I could give you some better advice?
I’m not sure about what you mean, do you mean the oneflow-server.conf file ?
Here is the service status at least but I’m afraid it won’t help.
root@one-ss2:~# oneflow show 17
SERVICE 17 INFORMATION
ID : 17
NAME : Service OneKE 1.27
USER : oneadmin
GROUP : oneadmin
STRATEGY : straight
SERVICE STATE : DEPLOYING
START TIME : 12/12 12:46:43
PERMISSIONS
OWNER : um-
GROUP : ---
OTHER : ---
ROLE vnf
ROLE STATE : RUNNING
VM TEMPLATE : 328
CARDINALITY : 1
MIN VMS : 1
NODES INFORMATION
VM_ID NAME USER GROUP
592 vnf_0_(service_17) oneadmin oneadmin
ROLE master
ROLE STATE : DEPLOYING
PARENTS : vnf
VM TEMPLATE : 329
CARDINALITY : 1
MIN VMS : 1
NODES INFORMATION
VM_ID NAME USER GROUP
593 master_0_(service_17) oneadmin oneadmin
ROLE worker
ROLE STATE : DEPLOYING
PARENTS : vnf
VM TEMPLATE : 329
CARDINALITY : 1
NODES INFORMATION
VM_ID NAME USER GROUP
594 worker_0_(service_17) oneadmin oneadmin
ROLE storage
ROLE STATE : RUNNING
PARENTS : vnf
VM TEMPLATE : 330
CARDINALITY : 0
NODES INFORMATION
VM_ID NAME USER GROUP
LOG MESSAGES
12/12/23 12:46 [I] New state: DEPLOYING
From the config you’ve provided I can deduce you’re using the same VNET as both “private” and “public” and you still do NAT there on top of it, that can’t really work… (as it doesn’t make any sense)
So either use 2 VNETs as described in the documentation or there’s an undocumented/unsupported way to use a single VNET which in your case would look like:
In that case, please try connecting to the master with SSH and take a look at /var/log/one-appliance/ONE_configure.log. You could also check /etc/haproxy/haproxy.cfg on the VNF node to see if HAProxy has actual backends configured.
(By “master” I assume you mean the VNF VM.)
The ONE_configure.log looks clean to me, unfortunately I can’t upload it as a new user so I will copy/paste it at the end of this message.
About the haproxy’s configuration file, it seems to be configured properly as far as i can see:
backend app
balance roundrobin
server app1 127.0.0.1:5001 check
server app2 127.0.0.1:5002 check
server app3 127.0.0.1:5003 check
server app4 127.0.0.1:5004 check
Please note there is nothing listening on those ports.
Here is the log :
[Wed Dec 13 16:30:33 UTC 2023] => INFO: =============================
[Wed Dec 13 16:30:33 UTC 2023] => INFO: === CONFIGURATION STARTED ===
[Wed Dec 13 16:30:33 UTC 2023] => INFO: =============================
[Wed Dec 13 16:30:33 UTC 2023] => INFO: Create empty context file: /etc/one-appliance/context.json
[Wed Dec 13 16:30:33 UTC 2023] => INFO: Try to load original vrouter's parameters if used
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VROUTER_ETH0_MANAGEMENT =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VROUTER_ETH0_VIP =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH0_IP as ONEAPP_VROUTER_ETH0_IP = 10.10.10.222
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH0_MASK as ONEAPP_VROUTER_ETH0_MASK = 255.255.255.0
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH0_MAC as ONEAPP_VROUTER_ETH0_MAC = 02:00:0a:3b:0a:de
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH0_DNS as ONEAPP_VROUTER_ETH0_DNS = 10.10.10.200 10.10.10.201
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH0_GATEWAY as ONEAPP_VROUTER_ETH0_GATEWAY = 10.10.10.254
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH0_MTU as ONEAPP_VROUTER_ETH0_MTU =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VROUTER_ETH1_MANAGEMENT =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VROUTER_ETH1_VIP =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH1_IP as ONEAPP_VROUTER_ETH1_IP = 172.16.32.1
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH1_MASK as ONEAPP_VROUTER_ETH1_MASK =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH1_MAC as ONEAPP_VROUTER_ETH1_MAC = 02:00:ac:10:20:01
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH1_DNS as ONEAPP_VROUTER_ETH1_DNS =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH1_GATEWAY as ONEAPP_VROUTER_ETH1_GATEWAY =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: SAVED: ETH1_MTU as ONEAPP_VROUTER_ETH1_MTU =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: Unify the separators for multivalue parameters
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_DNS_INTERFACES =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_DNS_INTERFACES_DISABLED =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_DHCP4_INTERFACES =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_DHCP4_INTERFACES_DISABLED =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_ROUTER4_INTERFACES = ETH0 ETH1
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_ROUTER4_INTERFACES_DISABLED =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_NAT4_INTERFACES_OUT = ETH0
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_NAT4_INTERFACES_OUT_DISABLED =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_SDNAT4_INTERFACES =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_SDNAT4_INTERFACES_DISABLED =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_LB_INTERFACES =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_LB_INTERFACES_DISABLED =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_HAPROXY_INTERFACES = ETH0
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_HAPROXY_INTERFACES_DISABLED =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_KEEPALIVED_INTERFACES =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: INJECTED: ONEAPP_VNF_KEEPALIVED_INTERFACES_DISABLED =
[Wed Dec 13 16:30:33 UTC 2023] => INFO: Sort out VNFs: ENABLED/DISABLED
[Wed Dec 13 16:30:33 UTC 2023] => INFO: VNF DHCP4 will be: DISABLED
[Wed Dec 13 16:30:33 UTC 2023] => INFO: VNF ROUTER4 will be: ENABLED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF ROUTER4 is modified - it will be: RELOADED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF DNS will be: DISABLED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF KEEPALIVED will be: ENABLED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF KEEPALIVED is modified - it will be: RELOADED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF NAT4 will be: DISABLED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF SDNAT4 will be: DISABLED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF LB will be: DISABLED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF HAPROXY will be: ENABLED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF HAPROXY is modified - it will be: RELOADED
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF ROUTER4: configure IPv4 forwarding
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF KEEPALIVED: write Keepalived configuration: /etc/keepalived//keepalived.conf
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF DHCP4 was not enabled or changed - skipping (re)configuration
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF DNS was not enabled or changed - skipping (re)configuration
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF NAT4 was not enabled or changed - skipping (re)configuration
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF SDNAT4 was not enabled or changed - skipping (re)configuration
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF LB was not enabled or changed - skipping (re)configuration
[Wed Dec 13 16:30:34 UTC 2023] => INFO: VNF HAPROXY: Create HAPROXY section in the configuration file: /opt/one-appliance/etc/one-vnf-config.js
[Wed Dec 13 16:30:34 UTC 2023] => INFO: Store current context in the file: /etc/one-appliance/context.json
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Toggle VNF services (Start/Stop)
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Keepalived will take care of starting and stopping of VNFs
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Stop and disable all VNFs except keepalived
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Stop and disable DHCP4 VNF
* rc-update: service `kea-dhcp4' is not in the runlevel `boot'
* rc-update: service `kea-dhcp4' is not in the runlevel `boot'
* rc-update: service `kea-dhcp4' is not in the runlevel `default'
* WARNING: kea-dhcp4 is already stopped
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Stop and disable ROUTER4 VNF
[Wed Dec 13 16:30:35 UTC 2023] => INFO: VNF ROUTER4: is about to be disabled
renamed '/etc/sysctl.d/01-one-router4.conf' -> '/etc/sysctl.d/01-one-router4.conf-disabled'
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.eth0.forwarding = 0
net.ipv4.conf.eth1.forwarding = 0
net.ipv4.conf.lo.forwarding = 0
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Stop and disable DNS VNF
* rc-update: service `one-unbound' is not in the runlevel `boot'
* rc-update: service `one-unbound' is not in the runlevel `boot'
* rc-update: service `one-unbound' is not in the runlevel `default'
* WARNING: one-unbound is already stopped
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Stop and disable NAT4 VNF
* rc-update: service `one-nat4' is not in the runlevel `boot'
* rc-update: service `one-nat4' is not in the runlevel `boot'
* rc-update: service `one-nat4' is not in the runlevel `default'
* WARNING: one-nat4 is already stopped
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Waiting for NAT4 rules to be cleared...
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Stop and disable SDNAT4 VNF
[Wed Dec 13 16:30:35 UTC 2023] => INFO: Stop and disable LB VNF
[Wed Dec 13 16:30:36 UTC 2023] => INFO: Stop and disable HAPROXY VNF
[Wed Dec 13 16:30:36 UTC 2023] => INFO: Enable KEEPALIVED VNF
* service keepalived added to runlevel boot
[Wed Dec 13 16:30:36 UTC 2023] => INFO: Start KEEPALIVED VNF
* Starting keepalived ... [ ok ]
[Wed Dec 13 16:30:36 UTC 2023] => INFO: Waiting for Keepalived to start (pidfile: /run/keepalived.pid)...
[Wed Dec 13 16:30:39 UTC 2023] => INFO: Save context/config variables as a report in: /etc/one-appliance/config
[Wed Dec 13 16:30:39 UTC 2023] => INFO: --- CONFIGURATION FINISHED ---
I think you’re using much older version Please re-download latest version Service OneKE 1.27. It uses new VNF implementation, you can find all the source code here as we made everything public and open source. We are actively working on both VRouter and OneKE, please report any issues you see, this will greatly help us improve both appliances. You can find latest documentation in the wiki here.
.