for rollback sake I prepared brand new ONE FE 6.0, restored production DB into it, and granularly upgraded it to the 6.4 without problems. After that I am no longer able to see VM logs - they are empty and monitor.log is full of:
[Z0][MDP][W]: Failed to monitor VM state for host X: Error executing state.rb: database is locked.
All nodes (KVM) updated to 6.4, all host force synced. FE in Stand-alone mode. OS Ubuntu 20.04.
“virsh …” can be called by oneadmin.
Any ideas welcomed, thank you.
There is an sqlite database on the virtualization node. The monitoring information is cached on there. There might be an issue with db file itself. It should look like
oneadmin@ubuntu2004-kvm-qcow2-6-4-rpZvf-2:~$ sqlite3 /var/tmp/one/im/status_kvm_1.db 'select * from states'
having 1 row per VM. You can delete such database on the host and the monitoring service should recreate it on its own.
Thank you Daniel for your quick reply.
I can confirm, that there is such a database, it can be read, contains one VM per row, and when I delete it, monitoring re-creates it immediately.
The result stays the same unfortunately, still the same error message with one change - right after deleting the db file, I got:
Failed to monitor VM state for host X: Error executing state.rb: attempt to write a readonly database
for a few moments, then locked database again.
There were an overlooked monitoring processes accessing the same DB but with bad One FE address on the hosts (leftover from old FE HA cluster). After killing all process groups error messages stopped coming.