oneks create cluster or with sunstone creates a seed vm. Seed VM is created only with public nic, private nic is missing. Cluster provisioning fails with timeout:
oneks show cluster 43
ONEKS CLUSTER 43 INFORMATION
ID : 43
NAME : oneks-dev
USER : oneadmin
GROUP : oneadmin
STATE : PROVISIONING
KUBERNETES VERSION : v1.34.2
ENDPOINT : N/A
REGISTRATION TIME : 09/10 14:19:19
CONTROL PLANE
ID : 44
NAME : controlplane-general-standalone-9e7bfa92ac83
FAMILY : general
FLAVOUR : standalone
STATE : BOOTSTRAPPING
VM IDS : –
onevm show 664
VIRTUAL MACHINE 664 INFORMATION
ID : 664
NAME : controlplane-general-standalone-9e7bfa92ac83-seed
USER : oneadmin
GROUP : oneadmin
STATE : ACTIVE
LCM_STATE : RUNNING
LOCK : None
RESCHED : No
HOST : somehost
CLUSTER ID : 0
CLUSTER : default
START TIME : 09/10 14:19:19
END TIME : -
DEPLOY ID : 40875664-139b-4e08-9d73-1c0122e130d4
[E][EVT]: Dependency SeedVM failed for group 44: EVENT API one.vrouter.allocate subscriber timed out after 500 seconds
[E][EVT]: Dependency ClusterRouter failed for group 44: EVENT API one.vrouter.allocate subscriber timed out after 500 seconds
[I][GRP]: ControlPlane (ID=44) changed state from BOOTSTRAPPING to BOOTSTRAPPING_FAILURE
[I][CLS]: Cluster 43 changed state from PROVISIONING to PROVISIONING_FAILURE
The Seed VM in OneKS is only attached to the public network. The virtual router (public + private) and the control plane VMs (private) are created later during bootstrap. Since this failed during the bootstrap phase, there likely wouldn’t be anything on the private network yet.
This may be related to a known OneKS bug: OneKS forces default image datastore · Issue #8081 · OpenNebula/one · GitHub . Are you using an image datastore other than ID 1, or has that datastore been removed or disabled? There are a couple of places that default to ID 1 when creating images for OneKS. That causes image creation, and therefore bootstrapping, to fail and prevents the cluster from deploying. It appears you’re using image-ssh datastore, was that renamed from default or is it a separate datastore?
Can you check whether the router and node images were created?
oneimage list | grep oneks-general
The current workaround is to create the images manually in the datastore you want them in. If failed images with these names already exist, delete them first. The current image URLs are in the files under /var/lib/one/oneks (`grep -rn qcow2 /var/lib/one/oneks/`), so use those if they differ from the ones below:
Create them as the same user that deploys the clusters so ownership and permissions match(appears to be oneadmin from your output). Wait until both show READY in `oneimage list`, then delete the failed cluster and create a new one.
Check oned.log for calls coming from the Seed VM during creation to verify that it is actually contacting the API at all.
If you see no calls from the Seed VM, it can’t reach the API. OneKS reaches oned through the transparent proxy, and that needs a tproxy entry for 2633 in addition to the 5030 one used for OneGate, in /var/lib/one/remotes/etc/vnm/OpenNebulaNetwork.conf:
Run onehost sync -f after editing. If the :networks: key is set on those entries, the public vnet ID needs to be in the list. You can verify from inside the Seed VM with curl -v ``http://169.254.16.9:2633/RPC2. any HTTP response means it’s reachable.
If you do see calls and they’re failing, the error text should point at what it’s trying to allocate.
The Seed VM should still be up during the 500s window (it’s only destroyed once the group reaches RUNNING). Get the IP from onevm show <seed_vm_id>, then SSH in and check the cluster logs:
kubectl get pods -A
kubectl -n capone-system logs deploy/capone-controller-manager --tail=200
kubectl get cluster,onecluster,machine -A
kubectl describe onecluster -A
Hello Neal,
thank you for your anser.
tproxy entry is defined, but http://169.254.16.9:2633/RPC2 is not reachable from the seed_vm, port 2633 is open on sunstone host. How can i enable 169.254.16.9?
Can you try updating your /etc/one/oned.conf and define ONEGATE_ENDPOINT = "http://169.254.16.9:5030" ? Perhaps this is still set to your frontend’s IP but when using Transparent Proxy this should be set to the tproxy address. The network config may not be generating properly for the VM if that value is pointed to the frontend’s address rather than the tproxy address.
If that doesn’t work you can try updating the OneKS templates and add a start script to the contect section: START_SCRIPT = "ip route replace 169.254.16.9 dev eth0" ?
Hello Neal,
ONEGATE_ENDPOINT = “http://169.254.16.9:5030” helped, route is now ok.
But the cluster provisioning still times out.
When i try a kubectl command in the seed vm I’ll get:
3457 memcache.go:265] “Unhandled Error” err=“couldn’t get current server API group list: Get “https://127.0.0.1:43953/api?timeout=32s\”: dial tcp 127.0.0.1:43953: connect: connection refused”
I assume the provisioning supposed to be done with podman, but no containers are running.
I’m only seeing a container exited 5 months ago:
podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
48a3027b8d74 docker.io/kindest/node@sha256:18fbefc20a7113353c7b75b5c869d7145a6abd6269154825872dc59c1329912e 5 months ago Exited (137) 5 months ago 127.0.0.1:43953->6443/tcp kind-control-plane