OpenNebula keeps trying to bootstrap database

OpenNebula keeps trying to bootstrap the database on top of the already existing database. Today our OpenNebula installation begin presenting authentication errors, I recreated the passwords for oneadmin and serveradmin but the problem persisted. After many hours of trial and error, now even the opennebula.service is refusing to start with the following error. I would like to add that recreating the database is not an option, as I will not start this installation over, I would rather migrate to another solution.

 Password file /var/lib/one//.one/sunstone_auth already exists but OpenNebula is boostraping the database. Check your database configuration in oned.conf.

OpenNebula Version:

ii  opennebula                             6.0.0-1.ce                            amd64        OpenNebula Server and Scheduler (Community Edition)
ii  opennebula-common                      6.0.0-1.ce                            all          Common OpenNebula package shared by various components (Community Edition)
ii  opennebula-common-onecfg               6.0.0-1.ce                            all          Helpers for OpenNebula onecfg (Community Edition)
ii  opennebula-fireedge                    6.0.0-1.ce                            amd64        OpenNebula web interface FireEdge (Community Edition)
ii  opennebula-flow                        6.0.0-1.ce                            all          OpenNebula Flow server (Community Edition)
ii  opennebula-gate                        6.0.0-1.ce                            all          OpenNebula Gate server (Community Edition)
ii  opennebula-guacd                       6.0.0-1.ce                            amd64        Provides Guacamole server for Fireedge to be used in Sunstone (Community Edition)
ii  opennebula-libs                        6.0.0-1.ce                            all          OpenNebula libraries (Community Edition)
ii  opennebula-node-kvm                    6.0.0-1.ce                            all          Services for OpenNebula KVM node (Community Edition)
ii  opennebula-provision                   6.0.0-1.ce                            all          OpenNebula infrastructure provisioning (Community Edition)
ii  opennebula-provision-data              6.0.0-1.ce                            all          OpenNebula infrastructure provisioning data (Community Edition)
ii  opennebula-rubygems                    6.0.0-1.ce                            all          Ruby dependencies for OpenNebula (Community Edition)
ii  opennebula-sunstone                    6.0.0-1.ce                            all          OpenNebula web interface Sunstone (Community Edition)
ii  opennebula-tools                       6.0.0-1.ce                            all          OpenNebula command line tools (Community Edition)

Oned.conf:

#*******************************************************************************
#                       OpenNebula Configuration file
#*******************************************************************************

#*******************************************************************************
# Daemon configuration attributes
#-------------------------------------------------------------------------------
#  MANAGER_TIMER: Time in seconds the core uses to evaluate periodical functions.
#  MONITORING_INTERVALS cannot have a smaller value than MANAGER_TIMER.
#
#  MONITORING_INTERVAL_MARKET: Time in seconds between market monitorization.
#  MONITORING_INTERVAL_DATASTORE: Time in seconds between image monitorization.
#  MONITORING_INTERVAL_DB_UPDATE: Time in seconds between DB writes of VM
#  monitoring information. -1 to disable DB updating and 0 to write every update
#
#  DS_MONITOR_VM_DISK: Number of MONIROTING_INTERVAL_DATASTORE intervals to monitor
#  VM disks. 0 to disable. Only applies to fs and fs_lvm datastores
#
#  SCRIPTS_REMOTE_DIR: Remote path to store the monitoring and VM management
#  scripts.
#
#  DB: Configuration attributes for the database backend
#   backend : can be sqlite or mysql (default is sqlite)
#   server  : (mysql) host name or an IP address for the MySQL server
#   port    : (mysql) port for the connection to the server.
#                     If set to 0, the default port is used.
#   user    : (mysql) user's MySQL login ID
#   passwd  : (mysql) the password for user
#   db_name : (mysql) the database name
#   connections: (mysql) number of max. connections to mysql server
#   compare_binary: (mysql) compare strings using BINARY clause
#                   makes name searches case sensitive.
#   encoding: charset to use for the db connections
#   timeout : (sqlite) timeout in ms for acquiring lock to DB,
#             should be at least 100 ms
#   errors_limit : number of consecutive DB errors to stop oned node in HA
#                  default 25, use -1 to disable this feature
#
#  VNC_PORTS: VNC port pool for automatic VNC port assignment, if possible the
#  port will be set to ``START`` + ``VMID``
#   start   : first port to assign
#   reserved: comma separated list of ports or ranges. Two numbers separated by
#   a colon indicate a range.
#
#  LOG: Configuration for the logging system
#   system: defines the logging system:
#      file      to log in the oned.log file
#      syslog    to use the syslog facilities
#      std       to use the default log stream (stderr) to use with systemd
#   debug_level: 0 = ERROR, 1 = WARNING, 2 = INFO, 3 = DEBUG
#
#*******************************************************************************

LOG = [
  SYSTEM      = "file",
  DEBUG_LEVEL = 3
]

#MANAGER_TIMER = 15

MONITORING_INTERVAL_DATASTORE = 300
MONITORING_INTERVAL_MARKET    = 600
MONITORING_INTERVAL_DB_UPDATE = 0

#DS_MONITOR_VM_DISK = 10

SCRIPTS_REMOTE_DIR=/var/tmp/one

# DB = [ BACKEND = "sqlite",
#       TIMEOUT = 2500 ]

# Sample configuration for MySQL
DB = [ BACKEND = "mysql",
       SERVER  = "<HOST>",
       PORT    = <PORT>,
       USER    = "<ONEADMIN>",
       PASSWD  = "<PASSWD>",
       DB_NAME = "<DBNAME>",
       CONNECTIONS = 25,
       COMPARE_BINARY = "no" ]

VNC_PORTS = [
    START    = 5900,
    RESERVED = "32768:65536"
    # RESERVED = "6800, 6801, 6810:6820, 9869"
]


HM_MAD = [
    EXECUTABLE = "one_hm",
    ARGUMENTS = "-p 2101 -l 2102 -b 127.0.0.1"]

AUTH_MAD = [
    EXECUTABLE = "one_auth_mad",
    AUTHN = "ssh,x509,ldap,server_cipher,server_x509"
]

#DEFAULT_AUTH = "default"

SESSION_EXPIRATION_TIME = 900

#ENABLE_OTHER_PERMISSIONS = "YES"

DEFAULT_UMASK = 177

Database credentials removed for obvious reasons

Steps to reproduce:

systemctl restart opennebula* 

Current results:

 Password file /var/lib/one//.one/sunstone_auth already exists but OpenNebula is boostraping the database. Check your database configuration in oned.conf.

Expected results:

OpenNebula Service Running

This is a security check not to overwrite an existing DB by error. It seems that at some point the bootstrap process did not complete or an existing DB was not properly cleaned. Simply delete the file it complains about (it’ll be regenerated). You can remove all the auth files but the oneadmin password.

Cheers!

I did that, it just recreates all the files and keeps giving me the same error.