One master node not starting after 5.12 upgrade to 6 Matra

Community Support General
1

After upgrading from 5.12.3 to 6.0 Mutara the nodes start and connect ok but the master is failing wth the following error.

oneadmin@one-mast-01:~$ onehost list
ID NAME CLUSTER TVM ALLOCATED_CPU ALLOCATED_MEM STAT
3 one-node-03 default 0 0 / 800 (0%) 0K / 31G (0%) on
2 one-node-02 default 0 0 / 800 (0%) 0K / 31G (0%) on
1 one-node-01 default 0 0 / 1600 (0%) 0K / 31G (0%) on
0 one-mast-01 default 0 0 / 1600 (0%) 0K / 30.9G (0%) err

monitor log
Tue Apr 20 18:05:52 2021 [Z0][HMM][D]: Monitoring host one-mast-01(0)
Tue Apr 20 18:05:54 2021 [Z0][HMM][E]: Unable to monitor host id: 0
Tue Apr 20 18:05:54 2021 [Z0][MDP][W]: Error parsing start message for host 0: <MONITOR_MESSAGES><SYSTEM_HOST>Error executing machines_models.rb: </SYSTEM_HOST>
Tue Apr 20 18:05:54 2021 [Z0][HMM][D]: Updated Host 0, state ERROR
Tue Apr 20 18:05:56 2021 [Z0][HMM][I]: Successfully monitored VM: 40
Sun Apr 18 00:28:25 2021 [Z0][HMM][I]: Successfully monitored VM: 40

2

There are a couple of things you may try:

  • Be sure that nodes has also been updated to the latest opennebula-node packages
  • You can try to execute the probe directly, log into the host and execute /var/tmp/one/im/kvm-probes.d/host/system/machines_models.rb
  • Also, you can try to put the host in offline, wipeout the /var/tmp/one in the host and then enable it.
3

Hi Ruben, Thanks for your response, your suggestions forced me to look at the opennebula-node package and found I had not installed opennebula-node-kvm on the master in my home lab configuration.
I carried out your suggestions and also installed opennebula-node-kvm on the master and purged opennebula-node from all nodes including the master. After restarting opennebula all is good.

All opennebula nodes installed on ubuntu 20.04

# Check packages installed on node
dpkg -l | grep opennebula
ii  opennebula-common                    6.0.0-1.ce                            all          Common OpenNebula package shared by various components (Community Edition)
ii  opennebula-common-onecfg             6.0.0-1.ce                            all          Helpers for OpenNebula onecfg (Community Edition)
rc  opennebula-node                      5.12.0.3-1.ce                         all          Services for OpenNebula KVM node (Community Edition)
ii  opennebula-node-kvm                  6.0.0-1.ce                            all          Services for OpenNebula KVM node (Community Edition)

# Fix Node issues:
# Remove opennebula-node 5.12.0.3-1.ce does not remove source code
sudo apt remove opennebula-node
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'opennebula-node' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

# Purge opennebula-node 5.12.0.3-1.ce source code
sudo dpkg --purge opennebula-node
(Reading database ... 125166 files and directories currently installed.)
Purging configuration files for opennebula-node (5.12.0.3-1.ce) ...

# Check packages installed on node
dpkg -l | grep opennebula
ii  opennebula-common                    6.0.0-1.ce                            all          Common OpenNebula package shared by various components (Community Edition)
ii  opennebula-common-onecfg             6.0.0-1.ce                            all          Helpers for OpenNebula onecfg (Community Edition)
ii  opennebula-node-kvm                  6.0.0-1.ce                            all          Services for OpenNebula KVM node (Community Edition)

# Fix Master issues:
# Check packages installed on master
dpkg -l | grep opennebula
ii  opennebula                           6.0.0-1.ce                        amd64        OpenNebula Server and Scheduler (Community Edition)
ii  opennebula-common                    6.0.0-1.ce                        all          Common OpenNebula package shared by various components (Community Edition)
ii  opennebula-common-onecfg             6.0.0-1.ce                        all          Helpers for OpenNebula onecfg (Community Edition)
ii  opennebula-flow                      6.0.0-1.ce                        all          OpenNebula Flow server (Community Edition)
ii  opennebula-gate                      6.0.0-1.ce                        all          OpenNebula Gate server (Community Edition)
ii  opennebula-libs                      6.0.0-1.ce                        all          OpenNebula libraries (Community Edition)
ii  opennebula-migration-community       6.0.0-2.ee                        all          Migration tools for OpenNebula Community Edition
rc  opennebula-node                      5.12.0.3-1.ce                     all          Services for OpenNebula KVM node (Community Edition)
ii  opennebula-rubygems                  6.0.0-1.ce                        all          Ruby dependencies for OpenNebula (Community Edition)
ii  opennebula-sunstone                  6.0.0-1.ce                        all          OpenNebula web interface Sunstone (Community Edition)
ii  opennebula-tools                     6.0.0-1.ce                        all          OpenNebula command line tools (Community Edition)

# Install missing opennebula-node-kvm package
sudo apt install opennebula-node-kvm
...

# Check packages installed on master
dpkg -l | grep opennebula
ii  opennebula                           6.0.0-1.ce                            amd64        OpenNebula Server and Scheduler (Community Edition)
ii  opennebula-common                    6.0.0-1.ce                            all          Common OpenNebula package shared by various components (Community Edition)
ii  opennebula-common-onecfg             6.0.0-1.ce                            all          Helpers for OpenNebula onecfg (Community Edition)
ii  opennebula-flow                      6.0.0-1.ce                            all          OpenNebula Flow server (Community Edition)
ii  opennebula-gate                      6.0.0-1.ce                            all          OpenNebula Gate server (Community Edition)
ii  opennebula-libs                      6.0.0-1.ce                            all          OpenNebula libraries (Community Edition)
ii  opennebula-migration-community       6.0.0-2.ee                            all          Migration tools for OpenNebula Community Edition
rc  opennebula-node                      5.12.0.3-1.ce                         all          Services for OpenNebula KVM node (Community Edition)
ii  opennebula-node-kvm                  6.0.0-1.ce                            all          Services for OpenNebula KVM node (Community Edition)
ii  opennebula-rubygems                  6.0.0-1.ce                            all          Ruby dependencies for OpenNebula (Community Edition)
ii  opennebula-sunstone                  6.0.0-1.ce                            all          OpenNebula web interface Sunstone (Community Edition)
ii  opennebula-tools                     6.0.0-1.ce                            all          OpenNebula command line tools (Community Edition)

# Purge opennebula-node 5.12.0.3-1.ce source code
sudo dpkg --purge opennebula-node
(Reading database ... 187198 files and directories currently installed.)
Purging configuration files for opennebula-node (5.12.0.3-1.ce) ...

dpkg -l | grep opennebula
ii  opennebula                           6.0.0-1.ce                            amd64        OpenNebula Server and Scheduler (Community Edition)
ii  opennebula-common                    6.0.0-1.ce                            all          Common OpenNebula package shared by various components (Community Edition)
ii  opennebula-common-onecfg             6.0.0-1.ce                            all          Helpers for OpenNebula onecfg (Community Edition)
ii  opennebula-flow                      6.0.0-1.ce                            all          OpenNebula Flow server (Community Edition)
ii  opennebula-gate                      6.0.0-1.ce                            all          OpenNebula Gate server (Community Edition)
ii  opennebula-libs                      6.0.0-1.ce                            all          OpenNebula libraries (Community Edition)
ii  opennebula-migration-community       6.0.0-2.ee                            all          Migration tools for OpenNebula Community Edition
ii  opennebula-node-kvm                  6.0.0-1.ce                            all          Services for OpenNebula KVM node (Community Edition)
ii  opennebula-rubygems                  6.0.0-1.ce                            all          Ruby dependencies for OpenNebula (Community Edition)
ii  opennebula-sunstone                  6.0.0-1.ce                            all          OpenNebula web interface Sunstone (Community Edition)
ii  opennebula-tools                     6.0.0-1.ce                            all          OpenNebula command line tools (Community Edition)

# Start opennebula and list nodes
onehost list
  ID NAME                                                                                                             CLUSTER    TVM      ALLOCATED_CPU      ALLOCATED_MEM STAT
   3 one-node-03                                                                                                      default      0       0 / 800 (0%)      0K / 31G (0%) on
   2 one-node-02                                                                                                      default      0       0 / 800 (0%)      0K / 31G (0%) on
   1 one-node-01                                                                                                      default      0      0 / 1600 (0%)      0K / 31G (0%) on
   0 one-mast-01                                                                                                      default      0      0 / 3200 (0%)    0K / 61.8G (0%) on

~