Database crash after bad power down

Hi.

Our opennebula database crashed after a powerfail…

After some repair at our mysql all seems to be fine… opennebula boot and daemon starts, but i found that its not possible to check hosts, and without hosts it seems no to possible deploy mv.

the command onevm list work fine, but onehost crash…

[oneadmin@cloud ~]$ onehost list ID NAME CLUSTER RVM ALLOCATED_CPU ALLOCATED_MEM STAT /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:57:informat_pool’: undefined method []' for nil:NilClass (NoMethodError) from /usr/lib/one/ruby/cli/cli_helper.rb:293:incall’
from /usr/lib/one/ruby/cli/cli_helper.rb:293:in data_array' from /usr/lib/one/ruby/cli/cli_helper.rb:292:incollect’
from /usr/lib/one/ruby/cli/cli_helper.rb:292:in data_array' from /usr/lib/one/ruby/cli/cli_helper.rb:291:incollect’
from /usr/lib/one/ruby/cli/cli_helper.rb:291:in data_array' from /usr/lib/one/ruby/cli/cli_helper.rb:260:inprint_data’
from /usr/lib/one/ruby/cli/cli_helper.rb:255:in print_table' from /usr/lib/one/ruby/cli/cli_helper.rb:219:inshow’
from /usr/lib/one/ruby/cli/one_helper.rb:485:in list_pool' from /usr/bin/onehost:202 from /usr/lib/one/ruby/cli/command_parser.rb:449:incall’
from /usr/lib/one/ruby/cli/command_parser.rb:449:in run' from /usr/lib/one/ruby/cli/command_parser.rb:76:ininitialize’
from /usr/bin/onehost:36:in new' from /usr/bin/onehost:36
Does anybody can help to solve that? or seems that the only solution is restoring the previus database/table???

Thanks in advance.

Hi @alfeijoo

Have you tried the onedb fsck command as oneadmin user?

It is necessary to stop the opennebula service while it is executed this command.

Hi @esfreire

yep! onedb and fix ussing mysql tool too… but bad lucky now… :frowning:

Today found an error at one host described in host_pool

but seems there are others problems similar:

MySQL dump stored in /var/lib/one/mysql_localhost_opennebula.sql_2016-2-3_10:47:34.bck
Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file


Datastore 104 is in Cluster 103 datastore id list, but it should not
Host 32 is in Cluster 100 host id list, but it should not
Host 46 is in Cluster 100 host id list, but it should not
Host 90 is in Cluster 100 host id list, but it should not
Host 92 is in Cluster 100 host id list, but it should not
Host 93 is in Cluster 100 host id list, but it should not
VNet 0 is in Cluster 100 vnet id list, but it should not
VNet 1 is in Cluster 100 vnet id list, but it should not
VNet 2 is in Cluster 100 vnet id list, but it should not
VNet 3 is in Cluster 100 vnet id list, but it should not
VNet 4 is in Cluster 100 vnet id list, but it should not
VNet 5 is in Cluster 100 vnet id list, but it should not
VNet 8 is in Cluster 100 vnet id list, but it should not
VNet 10 is in Cluster 100 vnet id list, but it should not
VNet 11 is in Cluster 100 vnet id list, but it should not
VNet 13 is in Cluster 100 vnet id list, but it should not
VNet 17 is in Cluster 105 vnet id list, but it should not

#<REXML::ParseException: Missing end tag for 'DISK_TYPE' (got "DISK_TYPTYPE")
Line: 
Position: 
Last 80 unconsumed characters:
<![CDATA[CDROM]]></TYPE></CONTEXT><CPU><![CDATA[1]]></CPU><DISK><BUS><![CDATA[vir>
/usr/lib/ruby/1.8/rexml/parsers/baseparser.rb:330:in `pull'
/usr/lib/ruby/1.8/rexml/parsers/treeparser.rb:22:in `parse'
/usr/lib/ruby/1.8/rexml/document.rb:245:in `build'
/usr/lib/ruby/1.8/rexml/document.rb:43:in `initialize'
/usr/lib/one/ruby/onedb/fsck.rb:639:in `new'
/usr/lib/one/ruby/onedb/fsck.rb:639:in `fsck'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:151:in `each'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:312:in `fetch_rows'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:367:in `yield_rows'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:312:in `fetch_rows'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:180:in `_execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:in `execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/connecting.rb:229:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/connection_pool/threaded.rb:105:in `hold'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/connecting.rb:229:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:in `execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:762:in `execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:351:in `execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:296:in `fetch_rows'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:151:in `each'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/dataset.rb:46:in `fetch'
/usr/lib/one/ruby/onedb/fsck.rb:638:in `fsck'
/usr/lib/one/ruby/onedb/onedb.rb:276:in `fsck'
/usr/bin/onedb:317
/usr/lib/one/ruby/cli/command_parser.rb:449:in `call'
/usr/lib/one/ruby/cli/command_parser.rb:449:in `run'
/usr/lib/one/ruby/cli/command_parser.rb:76:in `initialize'
/usr/bin/onedb:210:in `new'
/usr/bin/onedb:210
...
Missing end tag for 'DISK_TYPE' (got "DISK_TYPTYPE")
Line: 
Position: 
Last 80 unconsumed characters:
<![CDATA[CDROM]]></TYPE></CONTEXT><CPU><![CDATA[1]]></CPU><DISK><BUS><![CDATA[vir
Line: 
Position: 
Last 80 unconsumed characters:
<![CDATA[CDROM]]></TYPE></CONTEXT><CPU><![CDATA[1]]></CPU><DISK><BUS><![CDATA[vir
/usr/lib/ruby/1.8/rexml/parsers/treeparser.rb:92:in `parse'
/usr/lib/ruby/1.8/rexml/document.rb:245:in `build'
/usr/lib/ruby/1.8/rexml/document.rb:43:in `initialize'
/usr/lib/one/ruby/onedb/fsck.rb:639:in `new'
/usr/lib/one/ruby/onedb/fsck.rb:639:in `fsck'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:151:in `each'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:312:in `fetch_rows'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:367:in `yield_rows'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:312:in `fetch_rows'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:180:in `_execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:in `execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/connecting.rb:229:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/connection_pool/threaded.rb:105:in `hold'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/connecting.rb:229:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:in `execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:762:in `execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:351:in `execute'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:296:in `fetch_rows'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:151:in `each'
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/dataset.rb:46:in `fetch'
/usr/lib/one/ruby/onedb/fsck.rb:638:in `fsck'
/usr/lib/one/ruby/onedb/onedb.rb:276:in `fsck'
/usr/bin/onedb:317
/usr/lib/one/ruby/cli/command_parser.rb:449:in `call'
/usr/lib/one/ruby/cli/command_parser.rb:449:in `run'
/usr/lib/one/ruby/cli/command_parser.rb:76:in `initialize'
/usr/bin/onedb:210:in `new'
/usr/bin/onedb:210

Error running fsck version 4.11.80
The database will be restored
MySQL DB opennebula at localhost restored.

anybody know if is possible to know the table that make onedb crash?

I found some errors at DB and solve (in some case deleting the entry)

now opennebula boot, but onevm list show that error:

[oneadmin@cloud ~]$ onevm list
Unexpected error executing code for particular method, detected by Xmlrpc-c method registry code.  Method did not fail; rather, it did not complete at all.  615270-byte supposed UTF-8 string is not valid UTF-8.  UTF-8 multibyte sequence contains character 0x3c, which does not indicate continuation.

This seems also to be a a problem with an corrupeted entry, that cannot be
parsed. This case is a VM… yo can try do a loop with a onevm show and
see if you have luck…

1 Like

At the end I did that, make a bucle from our las 7000 mv and just wait the onevm show $ID that crash.

founded an extra (")

now ITs work :slightly_smiling: with a more recovered tables (like network_pool) that i lost but i had a backup from january 2015 but that table was not updated :smiley:

Thanks for your idea @ruben

any idea where to find that?

**undefined method content=' for nil:NilClass** /usr/lib/one/ruby/onedb/fsck.rb:1480:infsck’
/usr/lib/one/ruby/onedb/fsck.rb:1269:in each' /usr/lib/one/ruby/onedb/fsck.rb:1269:infsck’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:151:in each' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:312:infetch_rows’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:367:in yield_rows' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:312:infetch_rows’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:180:in _execute' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:inexecute’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/connecting.rb:229:in synchronize' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/connection_pool/threaded.rb:92:inhold’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/connecting.rb:229:in synchronize' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:inexecute’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:762:in execute' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:351:inexecute’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/adapters/mysql.rb:296:in fetch_rows' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/dataset/actions.rb:151:ineach’
/usr/lib/one/ruby/onedb/fsck.rb:1260:in fsck' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/query.rb:300:in_transaction’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/query.rb:263:in transaction' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/connecting.rb:229:insynchronize’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/connection_pool/threaded.rb:105:in hold' /usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/connecting.rb:229:insynchronize’
/usr/lib/ruby/gems/1.8/gems/sequel-3.39.0/lib/sequel/database/query.rb:261:in transaction' /usr/lib/one/ruby/onedb/fsck.rb:1259:infsck’
/usr/lib/one/ruby/onedb/onedb.rb:276:in fsck' /usr/bin/onedb:317 /usr/lib/one/ruby/cli/command_parser.rb:449:incall’
/usr/lib/one/ruby/cli/command_parser.rb:449:in run' /usr/lib/one/ruby/cli/command_parser.rb:76:ininitialize’
/usr/bin/onedb:210:in `new’
/usr/bin/onedb:210