Error parsing VM_STATE: in /var/log/one/oned.log

Community Support
1

I have a 5.12 test system that’s been working fine for months. I keep seeing this message over and over again on oned.log:

Fri Nov 6 19:07:31 2020 [Z0][InM][E]: Error parsing VM_STATE: syntax error, unexpected VARIABLE, expecting EQUAL or EQUAL_EMPTY at line 0, columns 8:11

I deleted all the VMs and it persists.

2

This is an error coming from the monitoring system. Try restarting opennebula and see if it keeps happening. If you don’t have any VM it shouldn’t keep reporting it. The VM monitoring data is stored on a sqlite db on the hypervisor nodes, you can manually delete it there so it won’t feed monitord

sqlite3 /var/tmp/one/im/status_lxd_0.db 
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> .tables
states
sqlite> select * from states;
one-0|0|one-0|one-0|1604939093|0|RUNNING|lxd

Make sure you don’t have any instances left

sudo lxc list
To start your first container, try: lxc launch ubuntu:18.04

+-------+---------+------------------------+------+------------+-----------+
| NAME  |  STATE  |          IPV4          | IPV6 |    TYPE    | SNAPSHOTS |
+-------+---------+------------------------+------+------------+-----------+
| one-0 | RUNNING | 192.168.150.100 (eth0) |      | PERSISTENT | 0         |
+-------+---------+------------------------+------+------------+-----------+
3

Hi Daniel,

We’re running MariaDB. I tried restarting the opennebula service a couple of times, and verified no VMs or containers running

[oneadmin@admin one]$ onevm list
  ID USER     GROUP    NAME                                                                                                              STAT  CPU     MEM HOST                                                                              TIM
oneadmin@t6:~$ lxc list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
oneadmin@t6:~$

Is there an equivalent database command to the one you mention?

I backed up the database and did a onedb purge-done

MariaDB [opennebula]> select * from vm_pool;
Empty set (0.000 sec)

I’m still seeing the errors on oned.log- thanks for your help.

4

Hi

OpenNebula runs on a main database which is the one you manage with onedb. The new monitoring system creates an sqlite database on each hypervisor node where it stores the VM info I pasted before. Those commands are run on the hypervisor node. Can you make sure the sqlite database exists there and check if it still holds any VM info ?

5

Interesting! I didn’t know about the sqlite DB on the hypervisor nodes.

I get this on my LXD node:

oneadmin@hpc-onet6:~$ sqlite3 /var/tmp/one/im/status_lxd_11.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .tables
states
sqlite> select * from states;
sqlite>

My two other KVM nodes don’t seem to have the sqlite3 executable installed but I see this:

[oneadmin@hpc-onet-admin ~]$ ssh hpc-onet1 ls /var/tmp/one/im/status*
/var/tmp/one/im/status_kvm_6.db
[oneadmin@hpc-onet-admin ~]$ ssh hpc-onet2 ls /var/tmp/one/im/status*
/var/tmp/one/im/status_kvm_7.db

$ ssh hpc-onet1 sqlite3
bash: sqlite3: command not found
...

I wonder if it’s some residue from the 5.10 -> 5.12 upgrade I did a couple of weeks ago?

6

install the package in that provides the utility to open the sqlite databases and inspect it. And, if you think you are having a mixt(5.10 + 5.12) of im drivers just delete /var/tmp/one in the virtualization nodes and run onehost sync in the frontend in order to perform a clean clean cleanup (yes) and make sure you only have the synced drivers from 5.12.

7

Thanks a lot for your help with this Daniel. What worked in the end was cleaning out /var/tmp/one as you suggested and then deleting and adding the hosts back into the cluster. That finally got rid of the spurious messages.