Invalid username or password when trying to login to web interface

Community Support
1

This problem started about a few weeks ago. We’d get that error on login and for expediency we’d just reboot the box and everything would be ok.

It used to only happen once in a while…but now its gotten progressively worse. It went from once every day to once a day, to twice a day…to 2 times an hour. To now it will not start the opennebula daemon (I can see it fail on start up sequence).

I “believe” we are on ver. 5.4.15 (I cant find the command to pull it up and currently we are “locked out”). This was installed on Ubuntu 16.04.

The following is output from the sunstone.log file of some entries to note:

Tue Jul 03 12:12:28 2018 [E]: [OneFlow] Error connecting to server (Failed to open TCP connection to localhost:2474 (Connection refused - connect(2) for “localhost” port 2474)).
Server: localhost:2474

&

Tue Jul 03 13:18:35 2018 [E]: Failed to open TCP connection to localhost:2633 (Connection refused - connect(2) for “localhost” port 2633)

I have poked around the forums and seen varying degrees of this issue but nothing exactly like what we are experiencing.

2

Hello @Aech,

These registry entries are due to OpenNebula being turned off or not started correctly.

Can you check oned.log?

BR.

Abel.

3

I don’t see any errors in here. It looks like the service starts up and then starts with no error/stop codes:

Thu Jul 5 07:53:22 2018 [Z0][ONE][I]: Log level:3 [0=ERROR,1=WARNING,2=INFO,3=DEBUG]
Thu Jul 5 07:53:22 2018 [Z0][ONE][I]: Support for xmlrpc-c > 1.31: yes
Thu Jul 5 07:53:22 2018 [Z0][ONE][I]: Checking database version.
Thu Jul 5 07:53:22 2018 [Z0][ONE][I]: oned is using version 4.90.0 for local_db_versioning
Thu Jul 5 07:53:22 2018 [Z0][ONE][I]: oned is using version 5.2.0 for db_versioning
Thu Jul 5 07:53:22 2018 [Z0][ACL][I]: Starting ACL Manager…
Thu Jul 5 07:53:22 2018 [Z0][ACL][I]: ACL Manager started.
Thu Jul 5 07:53:22 2018 [Z0][VMM][I]: Starting Virtual Machine Manager…
Thu Jul 5 07:53:22 2018 [Z0][LCM][I]: Starting Life-cycle Manager…
Thu Jul 5 07:53:22 2018 [Z0][VMM][I]: Virtual Machine Manager started.
Thu Jul 5 07:53:22 2018 [Z0][InM][I]: Starting Information Manager…
Thu Jul 5 07:53:22 2018 [Z0][LCM][I]: Life-cycle Manager started.
Thu Jul 5 07:53:22 2018 [Z0][InM][I]: Information Manager started.
Thu Jul 5 07:53:22 2018 [Z0][TrM][I]: Starting Transfer Manager…
Thu Jul 5 07:53:22 2018 [Z0][DiM][I]: Starting Dispatch Manager…
Thu Jul 5 07:53:22 2018 [Z0][TrM][I]: Transfer Manager started.
Thu Jul 5 07:53:22 2018 [Z0][DiM][I]: Dispatch Manager started.
Thu Jul 5 07:53:22 2018 [Z0][HKM][I]: Starting Hook Manager…
Thu Jul 5 07:53:22 2018 [Z0][HKM][I]: Hook Manager started.
Thu Jul 5 07:53:22 2018 [Z0][AuM][I]: Starting Auth Manager…
Thu Jul 5 07:53:22 2018 [Z0][AuM][I]: Authorization Manager started.
Thu Jul 5 07:53:22 2018 [Z0][ImM][I]: Starting Image Manager…
Thu Jul 5 07:53:22 2018 [Z0][ImM][I]: Starting Marketplace Manager…
Thu Jul 5 07:53:22 2018 [Z0][ImM][I]: Image Manager started.
Thu Jul 5 07:53:22 2018 [Z0][MKP][I]: Marketplace Manager started.
Thu Jul 5 07:53:22 2018 [Z0][IPM][I]: Starting IPAM Manager…
Thu Jul 5 07:53:22 2018 [Z0][MKP][I]: Marketplace Manager started.
Thu Jul 5 07:53:22 2018 [Z0][IPM][I]: Starting IPAM Manager…
Thu Jul 5 07:53:22 2018 [Z0][IPM][I]: IPAM Manager started.
Thu Jul 5 07:53:23 2018 [Z0][VMM][I]: Loading Virtual Machine Manager drivers.
Thu Jul 5 07:53:23 2018 [Z0][VMM][I]: Loading driver: kvm (KVM)
Thu Jul 5 07:53:24 2018 [Z0][VMM][I]: Driver kvm loaded.
Thu Jul 5 07:53:24 2018 [Z0][VMM][I]: Loading driver: vcenter (XML)
Thu Jul 5 07:53:24 2018 [Z0][VMM][I]: Driver vcenter loaded.
Thu Jul 5 07:53:24 2018 [Z0][InM][I]: Loading Information Manager drivers.
Thu Jul 5 07:53:24 2018 [Z0][InM][I]: Loading driver: collectd
Thu Jul 5 07:53:24 2018 [Z0][InM][I]: Driver collectd loaded
Thu Jul 5 07:53:24 2018 [Z0][InM][I]: Loading driver: kvm
Thu Jul 5 07:53:24 2018 [Z0][InM][I]: Driver kvm loaded
Thu Jul 5 07:53:24 2018 [Z0][InM][I]: Loading driver: vcenter
Thu Jul 5 07:53:24 2018 [Z0][InM][I]: Driver vcenter loaded
Thu Jul 5 07:53:24 2018 [Z0][TM][I]: Loading Transfer Manager driver.
Thu Jul 5 07:53:24 2018 [Z0][TM][I]: Transfer manager driver loaded
Thu Jul 5 07:53:24 2018 [Z0][HKM][I]: Loading Hook Manager driver.
Thu Jul 5 07:53:25 2018 [Z0][HKM][I]: Hook Manager loaded
Thu Jul 5 07:53:25 2018 [Z0][ImM][I]: Loading Image Manager driver.
Thu Jul 5 07:53:25 2018 [Z0][ImM][I]: Image Manager loaded
Thu Jul 5 07:53:25 2018 [Z0][MKP][I]: Loading Marketplace Manager driver.
Thu Jul 5 07:53:25 2018 [Z0][MKP][I]: Marketplace Manager loaded
Thu Jul 5 07:53:25 2018 [Z0][IPM][I]: Loading IPAM Manager driver.
Thu Jul 5 07:53:25 2018 [Z0][IPM][I]: IPAM Manager loaded
Thu Jul 5 07:53:25 2018 [Z0][AuM][I]: Loading Auth. Manager driver.
Thu Jul 5 07:53:25 2018 [Z0][AuM][I]: Auth Manager loaded
Thu Jul 5 07:53:25 2018 [Z0][ReM][I]: Starting Request Manager…
Thu Jul 5 07:53:25 2018 [Z0][ReM][I]: Starting XML-RPC server, port 2633 …
Thu Jul 5 07:53:25 2018 [Z0][ReM][I]: Request Manager started.
Thu Jul 5 07:53:25 2018 [Z0][ACL][I]: ACL Manager stopped.
Thu Jul 5 07:53:25 2018 [Z0][MKP][I]: Stopping Marketplace Manager…
Thu Jul 5 07:53:25 2018 [Z0][TrM][I]: Stopping Transfer Manager…
Thu Jul 5 07:53:25 2018 [Z0][DiM][I]: Stopping Dispatch Manager…
Thu Jul 5 07:53:25 2018 [Z0][DiM][I]: Dispatch Manager stopped.
Thu Jul 5 07:53:25 2018 [Z0][InM][I]: Stopping Information Manager…
Thu Jul 5 07:53:25 2018 [Z0][ReM][I]: Stopping Request Manager…
Thu Jul 5 07:53:25 2018 [Z0][HKM][I]: Stopping Hook Manager…
Thu Jul 5 07:53:25 2018 [Z0][ImM][I]: Stopping Image Manager…
Thu Jul 5 07:53:25 2018 [Z0][IPM][I]: Stopping IPAM Manager…
Thu Jul 5 07:53:25 2018 [Z0][VMM][I]: Stopping Virtual Machine Manager…
Thu Jul 5 07:53:25 2018 [Z0][LCM][I]: Stopping Life-cycle Manager…
Thu Jul 5 07:53:25 2018 [Z0][LCM][I]: Life-cycle Manager stopped.
Thu Jul 5 07:53:25 2018 [Z0][ReM][I]: XML-RPC server stopped.
Thu Jul 5 07:53:25 2018 [Z0][ReM][I]: Request Manager stopped.
Thu Jul 5 07:53:26 2018 [Z0][MKP][I]: Marketplace Manager stopped.
Thu Jul 5 07:53:26 2018 [Z0][ImM][I]: Image Manager stopped.
u Jul 5 07:53:26 2018 [Z0][HKM][I]: Hook Manager stopped.
Thu Jul 5 07:53:26 2018 [Z0][TrM][I]: Transfer Manager stopped.
Thu Jul 5 07:53:26 2018 [Z0][IPM][I]: IPAM Manager stopped.
Thu Jul 5 07:53:27 2018 [Z0][VMM][I]: Virtual Machine Manager stopped.
Thu Jul 5 07:53:28 2018 [Z0][InM][I]: Information Manager stopped.
Thu Jul 5 07:53:28 2018 [Z0][ONE][I]: All modules finalized, exiting.

4

Top half of the file:

Thu Jul 5 07:53:22 2018 [Z0][ONE][I]: Starting OpenNebula 5.2.1

 OpenNebula Configuration File

AUTH_MAD=AUTHN=ssh,x509,ldap,server_cipher,server_x509,EXECUTABLE=one_auth_mad
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=core,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=public,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=ssh,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=x509,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=YES,MAX_TOKEN_TIME=86400,NAME=ldap,PASSWORD_CHANGE=YES
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=server_cipher,PASSWORD_CHANGE=NO
AUTH_MAD_CONF=DRIVER_MANAGED_GROUPS=NO,MAX_TOKEN_TIME=-1,NAME=server_x509,PASSWORD_CHANGE=NO
DATASTORE_CAPACITY_CHECK=yes
DATASTORE_LOCATION=/var/lib/one//datastores
DATASTORE_MAD=ARGUMENTS=-t 15 -d dummy,fs,lvm,ceph,dev,iscsi_libvirt,vcenter -s shared,ssh,ceph,fs_lvm,qcow2,EXECUTABLE=one_datastore
DB=BACKEND=sqlite
DEFAULT_AUTH=ldap
DEFAULT_CDROM_DEVICE_PREFIX=hd
DEFAULT_COST=CPU_COST=0,DISK_COST=0,MEMORY_COST=0
DEFAULT_DEVICE_PREFIX=hd
DEFAULT_IMAGE_TYPE=OS
DEFAULT_UMASK=177
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=ceph,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=DISK_TYPE,BRIDGE_LIST
DS_MAD_CONF=NAME=dev,PERSISTENT_ONLY=YES,REQUIRED_ATTRS=DISK_TYPE
DS_MAD_CONF=NAME=iscsi_libvirt,PERSISTENT_ONLY=YES,REQUIRED_ATTRS=DISK_TYPE,ISCSI_HOST
DS_MAD_CONF=NAME=dummy,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=fs,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=lvm,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=DISK_TYPE,BRIDGE_LIST
DS_MAD_CONF=MARKETPLACE_ACTIONS=export,NAME=vcenter,PERSISTENT_ONLY=YES,REQUIRED_ATTRS=VCENTER_CLUSTER
DS_MAD_CONF=NAME=shared,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=ssh,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=
DS_MAD_CONF=NAME=vmfs,PERSISTENT_ONLY=NO,REQUIRED_ATTRS=BRIDGE_LIST
ENABLE_OTHER_PERMISSIONS=YES
FEDERATION=MASTER_ONED=,MODE=STANDALONE,ZONE_ID=0
HM_MAD=EXECUTABLE=one_hm
HOST_MONITORING_EXPIRATION_TIME=43200
HOST_PER_INTERVAL=15
IMAGE_RESTRICTED_ATTR=SOURCE
IM_MAD=ARGUMENTS=-p 4124 -f 5 -t 50 -i 20,EXECUTABLE=collectd,NAME=collectd
IM_MAD=ARGUMENTS=-r 3 -t 15 kvm,EXECUTABLE=one_im_ssh,NAME=kvm,SUNSTONE_NAME=KVM
IM_MAD=ARGUMENTS=-c -t 15 -r 0 vcenter,EXECUTABLE=one_im_sh,NAME=vcenter,SUNSTONE_NAME=VMWare vCenter
INHERIT_DATASTORE_ATTR=CEPH_HOST
INHERIT_DATASTORE_ATTR=CEPH_SECRET
INHERIT_DATASTORE_ATTR=CEPH_USER
INHERIT_DATASTORE_ATTR=CEPH_CONF
INHERIT_DATASTORE_ATTR=POOL_NAME
INHERIT_DATASTORE_ATTR=ISCSI_USER
INHERIT_DATASTORE_ATTR=POOL_NAME
INHERIT_DATASTORE_ATTR=ISCSI_USER
INHERIT_DATASTORE_ATTR=ISCSI_USAGE
INHERIT_DATASTORE_ATTR=ISCSI_HOST
INHERIT_DATASTORE_ATTR=GLUSTER_HOST
INHERIT_DATASTORE_ATTR=GLUSTER_VOLUME
INHERIT_DATASTORE_ATTR=DISK_TYPE
INHERIT_DATASTORE_ATTR=ADAPTER_TYPE
INHERIT_IMAGE_ATTR=ISCSI_USER
INHERIT_IMAGE_ATTR=ISCSI_USAGE
INHERIT_IMAGE_ATTR=ISCSI_HOST
INHERIT_IMAGE_ATTR=ISCSI_IQN
INHERIT_IMAGE_ATTR=DISK_TYPE
INHERIT_IMAGE_ATTR=ADAPTER_TYPE
INHERIT_VNET_ATTR=VLAN_TAGGED_ID
INHERIT_VNET_ATTR=FILTER_IP_SPOOFING
INHERIT_VNET_ATTR=FILTER_MAC_SPOOFING
INHERIT_VNET_ATTR=MTU
INHERIT_VNET_ATTR=INBOUND_AVG_BW
INHERIT_VNET_ATTR=INBOUND_PEAK_BW
INHERIT_VNET_ATTR=INBOUND_PEAK_KB
INHERIT_VNET_ATTR=OUTBOUND_AVG_BW
INHERIT_VNET_ATTR=OUTBOUND_PEAK_BW
INHERIT_VNET_ATTR=OUTBOUND_PEAK_KB
IPAM_MAD=ARGUMENTS=-t 1 -i dummy,EXECUTABLE=one_ipam
KEEPALIVE_MAX_CONN=30
KEEPALIVE_TIMEOUT=15
LISTEN_ADDRESS=0.0.0.0
LOG=DEBUG_LEVEL=3,SYSTEM=file
LOG_CALL_FORMAT=Req:%i UID:%u %m invoked %l
MAC_PREFIX=02:00
MANAGER_TIMER=15
MARKET_MAD=ARGUMENTS=-t 15 -m http,s3,one,EXECUTABLE=one_market
MARKET_MAD_CONF=APP_ACTIONS=monitor,NAME=one,PUBLIC=yes,REQUIRED_ATTRS=,SUNSTONE_NAME=OpenNebula.org Marketplace
MARKET_MAD_CONF=APP_ACTIONS=create, delete, monitor,NAME=http,REQUIRED_ATTRS=BASE_URL,PUBLIC_DIR,SUNSTONE_NAME=HTTP server
MARKET_MAD_CONF=APP_ACTIONS=create, delete, monitor,NAME=s3,REQUIRED_ATTRS=ACCESS_KEY_ID,SECRET_ACCESS_KEY,REGION,BUCKET,SUNSTONE_NAME=Amazon S3
MAX_CONN=15
MAX_CONN_BACKLOG=15
MESSAGE_SIZE=1073741824
MONITORING_INTERVAL=60
MONITORING_THREADS=50
NETWORK_SIZE=254
PCI_PASSTHROUGH_BUS=0x01
PORT=2633
RPC_LOG=NO
SCRIPTS_REMOTE_DIR=/var/tmp/one
SESSION_EXPIRATION_TIME=900
SCRIPTS_REMOTE_DIR=/var/tmp/one
SESSION_EXPIRATION_TIME=900
TIMEOUT=15
TM_MAD=ARGUMENTS=-t 15 -d dummy,lvm,shared,fs_lvm,qcow2,ssh,ceph,dev,vcenter,iscsi_libvirt,EXECUTABLE=one_tm
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=NONE,NAME=dummy,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SELF,LN_TARGET=NONE,NAME=lvm,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=NONE,NAME=shared,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=SYSTEM,NAME=fs_lvm,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,LN_TARGET=NONE,NAME=qcow2,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=YES,LN_TARGET=SYSTEM,NAME=ssh,SHARED=NO
TM_MAD_CONF=CLONE_TARGET=SELF,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=ceph,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SELF,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=iscsi_libvirt,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=NONE,LN_TARGET=NONE,NAME=dev,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=NONE,LN_TARGET=NONE,NAME=vcenter,SHARED=YES
TM_MAD_CONF=CLONE_TARGET=SYSTEM,DS_MIGRATE=NO,LN_TARGET=NONE,NAME=vmfs,SHARED=YES
VLAN_IDS=RESERVED=0, 1, 4095,START=2
VM_INDIVIDUAL_MONITORING=no
VM_MAD=ARGUMENTS=-t 15 -r 0 kvm,DEFAULT=vmm_exec/vmm_exec_kvm.conf,EXECUTABLE=one_vmm_exec,IMPORTED_VMS_ACTIONS=terminate, terminate-hard, hold, release, suspend,
resume, delete, reboot, reboot-hard, resched, unresched, disk-attach,
disk-detach, nic-attach, nic-detach, snap-create, snap-delete,KEEP_SNAPSHOTS=no,NAME=kvm,SUNSTONE_NAME=KVM,TYPE=kvm
VM_MAD=ARGUMENTS=-p -t 15 -r 0 vcenter -s sh,DEFAULT=vmm_exec/vmm_exec_vcenter.conf,EXECUTABLE=one_vmm_sh,IMPORTED_VMS_ACTIONS=terminate, terminate-hard, hold, release, suspend,
resume, delete, reboot, reboot-hard, resched, unresched, poweroff,
poweroff-hard, disk-attach, disk-detach, nic-attach, nic-detach,
snap-create, snap-delete,KEEP_SNAPSHOTS=yes,NAME=vcenter,SUNSTONE_NAME=VMWare vCenter,TYPE=xml
VM_MONITORING_EXPIRATION_TIME=14400
VM_PER_INTERVAL=5
VM_RESTRICTED_ATTR=CONTEXT/FILES
VM_RESTRICTED_ATTR=NIC/MAC
VM_RESTRICTED_ATTR=NIC/VLAN_ID
VM_RESTRICTED_ATTR=NIC/BRIDGE
VM_RESTRICTED_ATTR=NIC/INBOUND_AVG_BW
VM_RESTRICTED_ATTR=NIC/INBOUND_PEAK_BW
VM_RESTRICTED_ATTR=NIC/INBOUND_PEAK_KB
VM_RESTRICTED_ATTR=NIC/OUTBOUND_AVG_BW
VM_RESTRICTED_ATTR=NIC/OUTBOUND_PEAK_BW
VM_RESTRICTED_ATTR=NIC/OUTBOUND_PEAK_KB
VM_RESTRICTED_ATTR=NIC_DEFAULT/MAC
VM_RESTRICTED_ATTR=NIC_DEFAULT/VLAN_ID
VM_RESTRICTED_ATTR=NIC_DEFAULT/BRIDGE
VM_RESTRICTED_ATTR=DISK/TOTAL_BYTES_SEC
VM_RESTRICTED_ATTR=DISK/READ_BYTES_SEC
VM_RESTRICTED_ATTR=DISK/WRITE_BYTES_SEC
VM_RESTRICTED_ATTR=DISK/TOTAL_IOPS_SEC
VM_RESTRICTED_ATTR=DISK/READ_IOPS_SEC
VM_RESTRICTED_ATTR=DISK/WRITE_IOPS_SEC
VM_RESTRICTED_ATTR=DISK/ORIGINAL_SIZE
VM_RESTRICTED_ATTR=CPU_COST
VM_RESTRICTED_ATTR=DISK/ORIGINAL_SIZE
VM_RESTRICTED_ATTR=CPU_COST
VM_RESTRICTED_ATTR=MEMORY_COST
VM_RESTRICTED_ATTR=DISK_COST
VM_RESTRICTED_ATTR=PCI
VM_RESTRICTED_ATTR=EMULATOR
VM_RESTRICTED_ATTR=USER_INPUTS/CPU
VM_RESTRICTED_ATTR=USER_INPUTS/MEMORY
VM_RESTRICTED_ATTR=USER_INPUTS/VCPU
VM_SUBMIT_ON_HOLD=NO
VNC_PORTS=START=5900
VNET_RESTRICTED_ATTR=VN_MAD
VNET_RESTRICTED_ATTR=PHYDEV
VNET_RESTRICTED_ATTR=VLAN_ID
VNET_RESTRICTED_ATTR=BRIDGE
VNET_RESTRICTED_ATTR=AR/VN_MAD
VNET_RESTRICTED_ATTR=AR/PHYDEV
VNET_RESTRICTED_ATTR=AR/VLAN_ID
VNET_RESTRICTED_ATTR=AR/BRIDGE
VXLAN_IDS=START=2

5

It’s seems that your opennebula is not on ver. 5.4.15.

OpenNebula 5.2.1 is out of date and we don’t give support anymore. It would be better if you upgrade to the latest stable version 5.4.13.

Regards,

Abel.

6

Looking into the best way to do that now.

7

I’m getting the same error just trying to do the premigration steps for vcenter:

“Error contacting OpenNebula Failed to open TCP connection to localhost:2633 (Connection refused - connect(2) for “localhost” port 2633)”

I know you arent supporting the version we are on but this TCP error seems to be preventing me from doing a graceful upgrade

8

To debug this, execute a ‘onevm list’

9

Same error:

Failed to open TCP connection to localhost:2633 (Connection refused - connect(2) for “localhost” port 2633)

Firewall is disabled.

Keep in mind this was working fine up until a few weeks ago and no one changed anything.

10

Poking around online a few things to note. oned.log last entry was from yesterday in the morning (And i have done things with this server that should’ve written to that log file). Following some instructions on one of the topics addressing this for another sysadmin using opennebula I get the following results:

netstat -ntlp
Output:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:46635 0.0.0.0:* LISTEN 11775/sunstone
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 985/memcached
tcp 0 0 0.0.0.0:29876 0.0.0.0:* LISTEN 1256/python2
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 999/sshd
tcp6 0 0 :::80 :::* LISTEN 1165/apache2
tcp6 0 0 :::22 :::* LISTEN 999/sshd
tcp6 0 0 :::443 :::* LISTEN 1165/apache2

systemctl status opennebula
Output:
● opennebula.service - OpenNebula Cloud Controller Daemon
Loaded: loaded (/lib/systemd/system/opennebula.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2018-07-06 08:19:59 EDT; 5min ago
Process: 9299 ExecStartPre=/usr/sbin/logrotate -s /tmp/logrotate.state -f /etc/logrotate.d/opennebula (code=exited, status=1/FAILURE)
Process: 9294 ExecStartPre=/bin/chown oneadmin:oneadmin /var/log/one (code=exited, status=0/SUCCESS)
Process: 9290 ExecStartPre=/bin/mkdir -p /var/log/one (code=exited, status=0/SUCCESS)

Jul 06 08:19:59 oc02 systemd[1]: Starting OpenNebula Cloud Controller Daemon…
Jul 06 08:19:59 oc02 logrotate[9299]: error: error setting owner of /var/log/one/oned.log-20180706-1530879599 to uid 0 and gid 0: Operation not permitted
Jul 06 08:19:59 oc02 logrotate[9299]: error: error setting owner of /var/log/one/sched.log-20180706-1530879599 to uid 0 and gid 0: Operation not permitted
Jul 06 08:19:59 oc02 systemd[1]: opennebula.service: Control process exited, code=exited status=1
Jul 06 08:19:59 oc02 systemd[1]: Failed to start OpenNebula Cloud Controller Daemon.
Jul 06 08:19:59 oc02 systemd[1]: opennebula.service: Unit entered failed state.
Jul 06 08:19:59 oc02 systemd[1]: opennebula.service: Failed with result ‘exit-code’.

systemctl restart opennebula
Output:
Job for opennebula.service failed because the control process exited with error code. See “systemctl status opennebula.service” and “journalctl -xe” for details.

systemctl status opennebula
Output:
Same as previous
opennebula.service - OpenNebula Cloud Controller Daemon
Loaded: loaded (/lib/systemd/system/opennebula.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2018-07-06 08:26:47 EDT; 19min ago
Process: 11101 ExecStartPre=/usr/sbin/logrotate -s /tmp/logrotate.state -f /etc/logrotate.d/opennebula (code=exited, status=1/FAILURE)
Process: 11095 ExecStartPre=/bin/chown oneadmin:oneadmin /var/log/one (code=exited, status=0/SUCCESS)
Process: 11092 ExecStartPre=/bin/mkdir -p /var/log/one (code=exited, status=0/SUCCESS)

Jul 06 08:26:47 oc02 systemd[1]: Stopped OpenNebula Cloud Controller Daemon.
Jul 06 08:26:47 oc02 systemd[1]: Starting OpenNebula Cloud Controller Daemon…
Jul 06 08:26:47 oc02 logrotate[11101]: error: error setting owner of /var/log/one/oned.log-20180706-1530880007 to uid 0 and gid 0: Operation not permitted
Jul 06 08:26:47 oc02 logrotate[11101]: error: error setting owner of /var/log/one/sched.log-20180706-1530880007 to uid 0 and gid 0: Operation not permitted
Jul 06 08:26:47 oc02 systemd[1]: opennebula.service: Control process exited, code=exited status=1
Jul 06 08:26:47 oc02 systemd[1]: Failed to start OpenNebula Cloud Controller Daemon.
Jul 06 08:26:47 oc02 systemd[1]: opennebula.service: Unit entered failed state.
Jul 06 08:26:47 oc02 systemd[1]: opennebula.service: Failed with result ‘exit-code’.

netstat -ntlp
Same as before
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:46635 0.0.0.0:* LISTEN 11775/sunstone
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 985/memcached
tcp 0 0 0.0.0.0:29876 0.0.0.0:* LISTEN 1256/python2
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 999/sshd
tcp6 0 0 :::80 :::* LISTEN 1165/apache2
tcp6 0 0 :::22 :::* LISTEN 999/sshd
tcp6 0 0 :::443 :::* LISTEN 1165/apache2

None of these commands or their output shows up in the oned.log

11

I should also mention we have a newer instance of this product running and we only need this running long enough for our customers to transfer their data to the new cloud. SO the effort expended kinda sucks since we are arent keeping this around but we do need to be able to let people to login.

12

If /var/log/one is owned by oneadmin, this shouldn’t appear. Seems /var/log/one is owned by root or another user? chown -R oneadmin:oneadmin /var/log/one would fix this error?

13

I thought as well but I did chown the folder a attempts ago:

drwxr-xr-x 2 oneadmin oneadmin 80K Jul 9 08:45 one

14

oned.log is owned by root…changing it now.

15

Did that solve your issue? I’m looking at a similar oddity:

root@one-a:~# netstat -antp |grep one
tcp        1      0 192.0.2.94:42230       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        1      0 192.0.2.94:42236       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        1      0 192.0.2.94:42242       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        1      0 192.0.2.94:42226       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        1      0 192.0.2.94:42232       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        1      0 192.0.2.94:42238       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        0      0 192.0.2.94:42222       192.0.2.195:3306       ESTABLISHED 1418/oned       
tcp        1      0 192.0.2.94:42228       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        1      0 192.0.2.94:42240       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        1      0 192.0.2.94:42234       192.0.2.195:3306       CLOSE_WAIT  1418/oned       
tcp        1      0 192.0.2.94:42224       192.0.2.195:3306       CLOSE_WAIT  1418/oned

So, oned is running — but not completely, the all-mighty port 2633 hasn’t been opened, hence OpenNebula is basically dead:

root@one-a:~# onevm list
Failed to open TCP connection to localhost:2633 (Connection refused - connect(2) for "localhost" port 2633)

No [E]rror logged either:

root@one-a:~# grep '\[E\]' /var/log/one/oned.log
root@one-a:~# grep '\[E\]' /var/log/one/oned.log-20181004-15386*
root@one-a:~# grep '\[I\]' /var/log/one/oned.log
root@one-a:~# grep '\[I\]' /var/log/one/oned.log-20181004-15386*
/var/log/one/oned.log-20181004-1538661941:Thu Oct  4 16:05:35 2018 [Z0][ONE][I]: Starting OpenNebula 5.2.1
/var/log/one/oned.log-20181004-1538661941:Thu Oct  4 16:05:35 2018 [Z0][ONE][I]: Log level:3 [0=ERROR,1=WARNING,2=INFO,3=DEBUG]
/var/log/one/oned.log-20181004-1538661941:Thu Oct  4 16:05:35 2018 [Z0][ONE][I]: Support for xmlrpc-c > 1.31: yes

This install was running for months nicely (yes, 5.2.1; upgrade was planned for a later date but priority is to restore functionality first, obviously).

Any hints on where to look for unlogged obstacles or possible issues are greatly appreciated! (No, it’s not covered in other “Failed connect to localhost:2633”-style threads; I did a search …) Note that oned has stated, but did not open port 2633 but didn’t complain either :frowning:

16

FTR: Issue was caused by using a Galera Cluster where the master was waiting for a disk full condition to clear. Somehow oned’s initial CREATE DATABASE IF NOT EXISTS got queued – oned could benefit from a timeout here –, hence no Checking database version. output in the log. Port 2633 get’s opened much later …