Oned SQL errors

[tobx]

Hey,

since yesterday I am experiencing strange SQL errors in oned.log

Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'
Wed Mar 28 10:47:18 2018 [Z0][VMM][D]: VM 461 successfully monitored: STATE=a CPU=30.27 MEMORY=4373244 NETRX=746980356 NETTX=30071531 DISKRDBYTES=7024960000 DISKWRBYTES=12642642944 DISKRDIOPS=170554 DISKWRIOPS=694745
Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'
Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'
Wed Mar 28 10:47:18 2018 [Z0][VMM][D]: VM 464 successfully monitored: STATE=a CPU=4.04 MEMORY=4301668 NETRX=2178896375 NETTX=275122974 DISKRDBYTES=14115959808 DISKWRBYTES=17456934912 DISKRDIOPS=605371 DISKWRIOPS=710688
Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'
Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'

Has anyone seen something before and has a fix for it?

Especially the duplicate key issue seems odd: Duplicate entry ‘PPGTEMP01-disk-0-28’ for key ‘name’

I have already run a fsck but this did not fix the problem.

I am running 5.4.9 with RAFT.

[GabrielDias (Roman)]

Did you do onedb upgrade?

UPD: The same error

[Solved] Duplicate entry 'xx' for key 'name'

fsck alone did not solve it as somehow there was a log entry that did throw it back in instantly after at least two nodes were up. Solved it now by first running fsck then truncating logdb backing up the db again and restoring it on all nodes. Some more details: OpenNebula 5.4.1 running on Arch, 3 node cluster via raft Now everything is green again

[tobx]

I could finally solve it by doing the steps in the attached topic. But really strange that it happened. I could not reproduce the problem but somehow there was a stuck sql statement in the log table of one of the nodes. This lead to a complete malfunctioning cluster.