Please, describe the problem here and provide additional information below (if applicable) …
3 Opennebula hosts running Sunstone Frontend cluster across the three as well as Mysql DB on three.
Our Opennebula cloud had a stuck VM image that we had to remove manually by going into the master Node MYSQL DB and removing it. After that, we noticed that the entry was still present in the other ‘followers’ Database and we had to delete the same entry on the other two nodes.
What is the best way to update the database from the Leader and replicate the changes into the followers?
Versions of the related components and OS (frontend, hypervisors, VMs):
Opennebula 5.12
Steps to reproduce:
Make a change in one node DB using Mysql client and notice the change is not propagated to the other nodes.
With recent versions of ONe (I think starting with 5.4 ) the replication is handeled by the core. From my understanding this means that if you change something in the underlying database, those changes are not replicated, as the core is not aware of them in any means.
To replicate such a manual change on the lower layer, you would probably need to do a full resync of the tables, or, as you did, execute the update on every node on its own. As it works around the replication-mechanism, it might be not safe to do so.
As for your question: Check out the ondb CLI-Tool. This would be my first to go tool to do such changes. It does not allow to edit everything in the database, but by now I was able to fix everything that I broke with it. With that tool changes are done via the core and therefor replicated to followers. As the manpage states serveral times, its always sane to have a backup before using it
In my setup the “EDITOR”-ENV wasnt set so for me a command like
EDITOR=vim onedb update-body image --id 3956
allowed me to open an editor and “rescue” an image from a stalled locked-state. Afterwards I was able to delete it normaly. (But I double-checked the datastore etc. that everything went ok )
Hello, there are no rolling updates supported when you updating to a new major version resp. when there are changes in DB structure.
You should follow docs, where is everything about updating HA cluster. You have to take cluster down. Would be fine to test procedure in staging environment.
) the replication is handeled by the core. From my understanding this means that if you change something in the underlying database, those changes are not replicated, as the core is not aware of them in any means.