[SOLVED] Database update not working from 5.4.12 to 5.6 RC

angel_ramiro · July 3, 2018, 2:18pm

Please, describe the problem here and provide additional information below (if applicable) …

We are trying to test OpenNebula upgrade from v5.4.12 to v5.6 Release Candidate (5.5.90-1) on our testing environment, and all the upgrade proccess seems to run smoothly until we try to upgrade the database. We get this output:

[root@opennebula ~]# onedb upgrade -v -S opennebula -u oneadmin -d opennebula
_MySQL Password: _
Version read:
Shared tables 5.4.1 : OpenNebula 5.4.1 daemon bootstrap
Local tables 5.4.1 : OpenNebula 5.4.1 daemon bootstrap

MySQL dump stored in /var/lib/one/mysql_opennebula_opennebula_2018-7-3_16:10:59.sql
Use ‘onedb restore’ or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file

>>> Running migrators for shared tables
_ > Running migrator /usr/lib/one/ruby/onedb/shared/5.4.1_to_5.5.80.rb_
_ > Done in 0.00s_

Database migrated from 5.4.1 to 5.5.80 (OpenNebula 5.5.80) by onedb command.

>>> Running migrators for local tables
_ > Running migrator /usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb_
_ > Old Az file not found, skipping Az host migration_

Mysql2::Error: Table ‘old_vm_pool’ already exists
/usr/local/share/gems/gems/mysql2-0.5.1/lib/mysql2/client.rb:131:in _query'_ _/usr/local/share/gems/gems/mysql2-0.5.1/lib/mysql2/client.rb:131:inblock in query’
/usr/local/share/gems/gems/mysql2-0.5.1/lib/mysql2/client.rb:130:in handle_interrupt'_ _/usr/local/share/gems/gems/mysql2-0.5.1/lib/mysql2/client.rb:130:inquery’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:137:in block in _execute'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/logging.rb:38:inlog_connection_yield’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:132:in _execute'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:inblock in execute’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in block in synchronize'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/connection_pool/threaded.rb:91:inhold’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in synchronize'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:inexecute’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:68:in execute_dui'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/query.rb:43:inexecute_ddl’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/query.rb:76:in run'_ _/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:199:inbug_2189’
/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:40:in up'_ _/usr/lib/one/ruby/onedb/onedb.rb:232:inapply_migrators’
/usr/lib/one/ruby/onedb/onedb.rb:179:in upgrade'_ _/usr/bin/onedb:389:inblock (2 levels) in ’
/usr/lib/one/ruby/cli/command_parser.rb:457:in call'_ _/usr/lib/one/ruby/cli/command_parser.rb:457:inrun’
/usr/lib/one/ruby/cli/command_parser.rb:84:in initialize'_ _/usr/bin/onedb:294:innew’
/usr/bin/onedb:294:in `’

The database will be restored
MySQL DB opennebula at opennebula restored.

Versions of the related components and OS (frontend, hypervisors, VMs):
Opennebula 5.5.90-1 + mariadb-5.5.56-2.el7.x86_64 on same VMware VM on a CentOS 7.4
Steps to reproduce:
Migrate from a working OpenNebula v5.4.12 environment to OpenNebula v5.5.90-1 (release candidate)
Current results:
Database update not working
Expected results:

atodorov_storpool · July 3, 2018, 2:25pm

It looks like there are artifacts from previus (unsuccessful?) upgrade? You should try clean all old_* tables from the opennebula database and try again.

BR,
Anton

angel_ramiro · July 3, 2018, 2:32pm

Thanks for your answer!

This error refers to a second attempt.

The first error was:

**undefined method add_child' for nil:NilClass_** _/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:212:inblock (2 levels) in bug_2189’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:151:in block in each'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:inblock (2 levels) in fetch_rows’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:in each'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:inblock in fetch_rows’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:152:in _execute'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:inblock in execute’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in block in synchronize'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/connection_pool/threaded.rb:87:inhold’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in synchronize'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:inexecute’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:1082:in execute'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:273:inexecute’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:236:in fetch_rows'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:151:ineach’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/dataset.rb:51:in fetch'_ _/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:203:inblock in bug_2189’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/transactions.rb:204:in _transaction'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/transactions.rb:179:inblock in transaction’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in block in synchronize'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/connection_pool/threaded.rb:91:inhold’
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in synchronize'_ _/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/transactions.rb:145:intransaction’
/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:202:in bug_2189'_ _/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:40:inup’
/usr/lib/one/ruby/onedb/onedb.rb:232:in apply_migrators'_ _/usr/lib/one/ruby/onedb/onedb.rb:179:inupgrade’
/usr/bin/onedb:389:in block (2 levels) in <main>'_ _/usr/lib/one/ruby/cli/command_parser.rb:457:incall’
/usr/lib/one/ruby/cli/command_parser.rb:457:in run'_ _/usr/lib/one/ruby/cli/command_parser.rb:84:ininitialize’
/usr/bin/onedb:294:in new'_ _/usr/bin/onedb:294:in’

Thanks in advance.

Best regards.

mschau · July 3, 2018, 2:49pm

Same here!
First attempt:
[root@LXLI03-T11 opennebula-5.5.90-1]# onedb upgrade -v -S localhost -u oneadmin -p XXXXXXXX -d opennebula
Version read:
Shared tables 5.4.1 : OpenNebula 5.4.6 daemon bootstrap
Local tables 5.4.1 : OpenNebula 5.4.6 daemon bootstrap

MySQL dump stored in /var/lib/one/mysql_localhost_opennebula_2018-7-3_16:38:48.sql
Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file


>>> Running migrators for shared tables
  > Running migrator /usr/lib/one/ruby/onedb/shared/5.4.1_to_5.5.80.rb
  > Done in 0.00s

Database migrated from 5.4.1 to 5.5.80 (OpenNebula 5.5.80) by onedb command.

>>> Running migrators for local tables
  > Running migrator /usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb
  > Old Az file not found, skipping Az host migration

undefined method `add_child' for nil:NilClass
/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:212:in `block (2 levels) in bug_2189'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:151:in `block in each'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:in `block (2 levels) in fetch_rows'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:in `each'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:in `block in fetch_rows'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:152:in `_execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:in `block in execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in `block in synchronize'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/connection_pool/threaded.rb:87:in `hold'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in `synchronize'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:in `execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:1082:in `execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:273:in `execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:236:in `fetch_rows'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:151:in `each'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/dataset.rb:51:in `fetch'
/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:203:in `block in bug_2189'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/transactions.rb:204:in `_transaction'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/transactions.rb:179:in `block in transaction'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in `block in synchronize'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/connection_pool/threaded.rb:91:in `hold'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in `synchronize'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/transactions.rb:145:in `transaction'
/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:202:in `bug_2189'
/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:40:in `up'
/usr/lib/one/ruby/onedb/onedb.rb:232:in `apply_migrators'
/usr/lib/one/ruby/onedb/onedb.rb:179:in `upgrade'
/usr/bin/onedb:389:in `block (2 levels) in <main>'
/usr/lib/one/ruby/cli/command_parser.rb:457:in `call'
/usr/lib/one/ruby/cli/command_parser.rb:457:in `run'
/usr/lib/one/ruby/cli/command_parser.rb:84:in `initialize'
/usr/bin/onedb:294:in `new'
/usr/bin/onedb:294:in `<main>'


The database will be restored
MySQL DB opennebula at localhost restored.

Second attempt:
[root@LXLI03-T11 opennebula-5.5.90-1]# onedb upgrade -v -S localhost -u oneadmin -p XXXXXXX -d opennebula
Version read:
Shared tables 5.4.1 : OpenNebula 5.4.6 daemon bootstrap
Local tables 5.4.1 : OpenNebula 5.4.6 daemon bootstrap

MySQL dump stored in /var/lib/one/mysql_localhost_opennebula_2018-7-3_16:39:40.sql
Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file


>>> Running migrators for shared tables
  > Running migrator /usr/lib/one/ruby/onedb/shared/5.4.1_to_5.5.80.rb
  > Done in 0.00s

Database migrated from 5.4.1 to 5.5.80 (OpenNebula 5.5.80) by onedb command.

>>> Running migrators for local tables
  > Running migrator /usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb
  > Old Az file not found, skipping Az host migration

Mysql2::Error: Table 'old_vm_pool' already exists
/usr/local/share/gems/gems/mysql2-0.5.1/lib/mysql2/client.rb:131:in `_query'
/usr/local/share/gems/gems/mysql2-0.5.1/lib/mysql2/client.rb:131:in `block in query'
/usr/local/share/gems/gems/mysql2-0.5.1/lib/mysql2/client.rb:130:in `handle_interrupt'
/usr/local/share/gems/gems/mysql2-0.5.1/lib/mysql2/client.rb:130:in `query'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:137:in `block in _execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/logging.rb:38:in `log_connection_yield'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:132:in `_execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:in `block in execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in `block in synchronize'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/connection_pool/threaded.rb:91:in `hold'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in `synchronize'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:in `execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:68:in `execute_dui'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/query.rb:43:in `execute_ddl'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/query.rb:76:in `run'
/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:199:in `bug_2189'
/usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb:40:in `up'
/usr/lib/one/ruby/onedb/onedb.rb:232:in `apply_migrators'
/usr/lib/one/ruby/onedb/onedb.rb:179:in `upgrade'
/usr/bin/onedb:389:in `block (2 levels) in <main>'
/usr/lib/one/ruby/cli/command_parser.rb:457:in `call'
/usr/lib/one/ruby/cli/command_parser.rb:457:in `run'
/usr/lib/one/ruby/cli/command_parser.rb:84:in `initialize'
/usr/bin/onedb:294:in `new'
/usr/bin/onedb:294:in `<main>'


The database will be restored
MySQL DB opennebula at localhost restored.

[root@LXLI03-T11 opennebula-5.5.90-1]# onedb fsck -S localhost -u oneadmin -p XXXXXXX -d opennebula
Version mismatch: fsck file is for version
Shared: 5.5.80, Local: 5.5.80

Current database is version
Shared: 5.4.1, Local: 5.4.1

atodorov_storpool · July 3, 2018, 3:14pm

It is a long shot, but could you try editing the migrator /usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb line 212:

                doc.xpath("//SNAPSHOTS").first.add_child(doc.create_element("NEXT_SNAPSHOT")).content = next_snapshot

With (note that theyou should use <tab> to align the second line):

                if ( !doc.xpath("//SNAPSHOTS").first.nil? )
                    doc.xpath("//SNAPSHOTS").first.add_child(doc.create_element("NEXT_SNAPSHOT")).content = next_snapshot

It looks like there is no //SNAPSHOTS element in the VM’s metadata…

And then retry (dropping old_vm_pool manually first)?

BR,
Anton

mschau · July 3, 2018, 3:44pm

Hi!
Thanks for the reply - I changed the line and tried again:

[root@LXLI03-T11 ~]# onedb upgrade -v -S localhost -u root -p test1234 -d opennebula
Version read:
Shared tables 5.4.1 : OpenNebula 5.4.6 daemon bootstrap
Local tables  5.4.1 : OpenNebula 5.4.6 daemon bootstrap

MySQL dump stored in /var/lib/one/mysql_localhost_opennebula_2018-7-3_18:19:52.sql
Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file


>>> Running migrators for shared tables
  > Running migrator /usr/lib/one/ruby/onedb/shared/5.4.1_to_5.5.80.rb
  > Done in 0.00s

Database migrated from 5.4.1 to 5.5.80 (OpenNebula 5.5.80) by onedb command.

>>> Running migrators for local tables
  > Running migrator /usr/lib/one/ruby/onedb/local/5.4.1_to_5.5.80.rb
  > Old Az file not found, skipping Az host migration
  > Done in 0.16s

Database migrated from 5.4.1 to 5.5.80 (OpenNebula 5.5.80) by onedb command.

Total time: 0.17s



[root@LXLI03-T11 ~]# onedb fsck -S localhost -u root -d opennebula
MySQL Password:
MySQL dump stored in /var/lib/one/mysql_localhost_opennebula_2018-7-3_18:22:48.sql
Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file
incompatible character encodings: UTF-8 and ISO-8859-1
/usr/lib/one/ruby/onedb/fsck/host.rb:51:in `block in check_host_cluster'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:151:in `block in each'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:in `block (2 levels) in fetch_rows'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:in `each'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:238:in `block in fetch_rows'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:152:in `_execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:in `block in execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in `block in synchronize'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/connection_pool/threaded.rb:91:in `hold'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/connecting.rb:264:in `synchronize'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/utils/mysql_mysql2.rb:38:in `execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:1082:in `execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:273:in `execute'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/adapters/mysql2.rb:236:in `fetch_rows'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/dataset/actions.rb:151:in `each'
/usr/local/share/gems/gems/sequel-5.7.1/lib/sequel/database/dataset.rb:51:in `fetch'
/usr/lib/one/ruby/onedb/fsck/host.rb:19:in `check_host_cluster'
/usr/lib/one/ruby/onedb/fsck.rb:288:in `fsck'
/usr/lib/one/ruby/onedb/onedb.rb:280:in `fsck'
/usr/bin/onedb:405:in `block (2 levels) in <main>'
/usr/lib/one/ruby/cli/command_parser.rb:457:in `call'
/usr/lib/one/ruby/cli/command_parser.rb:457:in `run'
/usr/lib/one/ruby/cli/command_parser.rb:84:in `initialize'
/usr/bin/onedb:294:in `new'
/usr/bin/onedb:294:in `<main>'

Error running fsck version 5.5.80
The database will be restored
MySQL DB opennebula at localhost restored.

mschau · July 4, 2018, 9:59am

Still no glue - any hint on this?

Thanks!

ruben · July 4, 2018, 10:24am

Thanks guys for the feedback.

There are two issues here:

missing snapshots, fixed with @atodorov_storpool patch. THANKS!!!
Failing fsck becasue of UTF-8 and ISO-8859-1 (Something we tried to fix for 5.6, but obviously we failed).

I’m going to take care of patch for upgrade code right away. And we’ll look into the other issue.

@mschau, I’d try to procced with the testing as usually there should not be any inconsistencies in the DB.

Cheers

ruben · July 4, 2018, 10:27am

BTW @mschau, Do you have any non-ascii chars in the host/cluster names?.

We are looking to fsck issue here:

https://github.com/OpenNebula/one/issues/1557

mschau · July 4, 2018, 10:45am

Thanks for the quick replys - just proceeding did the trick
At least it seems to work for now.

angel_ramiro · July 4, 2018, 1:19pm

Thank you Anton!

We made the change and it worked perfectly, including the fsck, Ruben.

Regards.

Topic		Replies	Views
Onedb upgrade 5.6 to 5.8 upgrades database to 5.10 Upgrade	2	1096	July 12, 2019
Failed Upgrade from 4.12 to 5.2 Community Support	3	737	November 18, 2016
Onedb upgrade fails with table already exists Community Support	0	783	November 7, 2016
Opennebula Upgrade to 5.0 Community Support	2	801	June 15, 2016
Upgrade on debian 10 from 6.4 to 6.6 -- database upgrade not working Upgrade	3	343	October 10, 2024

[SOLVED] Database update not working from 5.4.12 to 5.6 RC

Related topics