Issues with connecting SSH from front-end to node (followed the tutorial more than 10 times now)

Hi all!

I am new here and also new to OpenNebula. I am trying out OpenNebula to see if it’s something we can use.
However I am experiencing issues with setting up a node. The main issue I cannot seem to connect to the node from the front-end as oneadmin. No matter what I try or do…

I followed both the documentation (https://docs.opennebula.org/5.4/deployment/node_installation/kvm_node_installation.html) as this tutorial (https://computingforgeeks.com/opennebula-kvm-node-installation-centos-7/) many times. Even redid the installation several times. And I am 100% confident that I followed the documentation / tutorial to the letter. Also tried several other things.

But for some reason, I cannot connect to the node at all as user “oneadmin”. It is giving me a headache after 2 days. Setting up the front-end was easy compared to setting up a node. And the only problem is, is connecting by SSH to it.

Before I forget. SSH is working for the node; as I can connect to it by “root” without issues at all. And if I use “ssh-copy-id @” I can even login as root without password from the front-end. So I am 100% sure SSH is working and accepting connections.

I am really stumped by this. Maybe I am a complete idiot, but I have no clue what I am doing wrong, especially since I followed the documentation/tutorial to the letter. So I think there is missing a step somewhere. I have cleaned out “/var/lib/one/”-folder (on the node) several times and redid all the steps once again. But, again, with the same nasty result. No SSH connection possible/allowed/whatever as “oneadmin”. Sigh…

While I was typing this, I redid everything on the node once more (obviously with exactly the same result).

What I did (removed “/var/lib/one/”-folder before installing ofcourse) after installing “opennebula-node-kvm” and modifying /etc/libvirt/libvirtd.conf and restarting it, was the following:

Open my SSH connection to the front-end:

  • su - oneadmin
    Result:
    -bash-4.2$

  • ssh-keyscan ip-front-end ip-node >> /var/lib/one/.ssh/known_hosts
    Result:

  • ip-front-end:22 SSH-2.0-OpenSSH_7.4
  • ip-front-end:22 SSH-2.0-OpenSSH_7.4
  • ip-front-end:22 SSH-2.0-OpenSSH_7.4
  • ip-node:22 SSH-2.0-OpenSSH_7.4
  • ip-node:22 SSH-2.0-OpenSSH_7.4
  • ip-node:22 SSH-2.0-OpenSSH_7.4

I am guessing that I now type “exit” and return as “root” on the front-end?
(if this is not correct, rest assured I tried the same from the bash as well)

  • scp -rp /var/lib/one/.ssh ip-node:/var/lib/one/
    Result:
    id_rsa 100% 1679 78.5KB/s 00:00
    id_rsa.pub 100% 406 20.7KB/s 00:00
    authorized_keys 100% 406 22.2KB/s 00:00
    known_hosts 100% 25KB 739.3KB/s 00:00

So far so good, I guess?

Now there is a minor different between the documentation and the tutorial, so I will use the example given in the documentation. According to the documentation:

  • I opened a new SSH connection (just to be sure) to the front-end and logged in as “oneadmin” with my created password.
    Result:
    -bash-4.2$

So next up is: ssh

  • ssh ip-node
    Result:
    oneadmin@ip-node’s password:

So here it already goes wrong; instead of giving me access without password, it asks for a password. Sigh.
What am I doing wrong?! Where do I have to look?! What do I have to change to make this work?! Going crazy here…

Furthermore, even if I enter the password, which I use for “oneadmin” it’s not being accepted.
Which isn’t a big surprise as there is not “oneadmin” user in /etc/passwd. So I doubt this will ever work.
Created the “oneadmin”-user manually on the node and retried the above. It still asks for a password, but when I enter it, I can login (but not without password).

So next up; trying to login from the node to the front-end (I am guessing this should be possible). I logged in as user “oneadmin” and tried to ssh to the front-end. The result:

The authenticity of host ‘ip-front-end (ip-front-end)’ can’t be established.
ECDSA key fingerprint is SHA256:uJtfvOGkCnPRsItvT4UmHaGlXXXXXX-REMOVED-XXXXX.
ECDSA key fingerprint is MD5:29:fe:76:21:84:e8:de:1b:a9:XX:XX:XX:XX:XX:XX:XX.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘ip-front-end’ (ECDSA) to the list of known hosts.
oneadmin@ip-front-end’s password:

Also the other way around it asks for a password…

So, after redoing everything for 2 days in a row now and not getting anywhere, I am getting pretty annoyed by this. I was expecting this to be much, much easier. However it seems setting up a node is very, very difficult. Not only that, but I also have a headache because of this. I really have no clue what I am doing wrong, as I followed the documentation and tutorial perfectly (and redid it many times now). But all without any kind of success.

I searched the forums about similar issues in regards to SSH connecting without password and I also searched the internet with Google, but also without any result, let alone a working solution. Even tried chown, chmod and what else. To be honest; I tried so many things that I have no clue what I tried or did trying to make it work…

…yes, maybe I am an idiot and I am making a mistake somewhere, but when I follow the documentation / tutorial to the letter, it should simply work, right?

So I am hoping someone experienced can provide some insight, solution or anything that can help me in getting SSH access to the node correctly and without password.

And for your information; front-end and node were completely setup from scratch with CentOS 7.x and updated everything (as mentioned in the documentation). I even tried different versions op OpenNebula from 5.4 (yesterday) and 5.6 (today). All without any form of succes. Well that’s not completely true; the front-end appears to work, but without the possibility to connect nodes to it, it’s not much use…

Anyways, I hope I can get some help here, or at least provide me in the right direction. Thank you!!

Your SSH login is added differently when you log in the first time using IP, hostname or FQDN… oneadmin is not a regular user account, and it does take some work to get all the nodes and the host to agree on who is who… For my install I used Ubuntu and MAAS, which installs a clean OS with the default account called ubuntu. This account, just like the oneadmin account, is password-less. I used that to open up SSH and create the necessary credentials.

There is also the issue of the host file - your hostname works without the domain part - both oneadmin@hostname and oneadmin@hostname.domain.xyz resolves to the same IP but you do not want to allow that… you should prefer to use oneadmin@hostname.domain.xyz from the start and never use the oneadmin@hostname. Especially if you are going to use LDAP or other directory and such… its also just neater.

I spent days until I started to pay attention and made the effort to stay 100% consistent.
NOTE: on ubuntu it is also strongly advised to disable NetworkManager right away as it does complicate things, at least to me it did.

Hello @Restafval, it is all about setup ssh access using keys. So I will try to do this:

  • su - oneadmin
  • rm -f ~/.ssh/known_hosts
  • rm -f ~/.ssh/authorized_keys
  • cat ~/.ssh/id_rsa.pub > .ssh/authorized_keys
  • ssh-keyscan ip-frontend ip-backend hostname-frontend hostname-backend > ~/.ssh/known_hosts
  • chown 700 ~/.ssh
  • chown 600 ~/.ssh/authorized_keys
  • rm -rf ~/.ssh/* on node
  • rsync -avh .ssh/ node:~/.ssh/

Them if you have still problems try to login with verbose option. ssh -vvv oneadmin@node