Oned SQL errors

Hey,

since yesterday I am experiencing strange SQL errors in oned.log

Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'
Wed Mar 28 10:47:18 2018 [Z0][VMM][D]: VM 461 successfully monitored: STATE=a CPU=30.27 MEMORY=4373244 NETRX=746980356 NETTX=30071531 DISKRDBYTES=7024960000 DISKWRBYTES=12642642944 DISKRDIOPS=170554 DISKWRIOPS=694745
Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'
Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'
Wed Mar 28 10:47:18 2018 [Z0][VMM][D]: VM 464 successfully monitored: STATE=a CPU=4.04 MEMORY=4301668 NETRX=2178896375 NETTX=275122974 DISKRDBYTES=14115959808 DISKWRBYTES=17456934912 DISKRDIOPS=605371 DISKWRIOPS=710688
Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'
Wed Mar 28 10:47:18 2018 [Z0][ONE][E]: SQL command was: INSERT INTO image_pool (oid, name, body, uid, gid, owner_u, group_u, other_u) VALUES (145,'PPGTEMP01-disk-0','<IMAGE><ID>145</ID><UID>28</UID><GID>121</GID><UNAME>svcon-admin</UNAME><GNAME>svcon</GNAME><NAME>PPGTEMP01-disk-0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>0</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><TYPE>0</TYPE><DISK_TYPE>3</DISK_TYPE><PERSISTENT>0</PERSISTENT><REGTIME>1522168727</REGTIME><SOURCE><![CDATA[]]></SOURCE><PATH><![CDATA[one/one-136]]></PATH><FSTYPE><![CDATA[raw]]></FSTYPE><SIZE>25600</SIZE><STATE>4</STATE><RUNNING_VMS>0</RUNNING_VMS><CLONING_OPS>0</CLONING_OPS><CLONING_ID>136</CLONING_ID><TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT><DATASTORE_ID>102</DATASTORE_ID><DATASTORE>HDD-IMAGE</DATASTORE><VMS></VMS><CLONES></CLONES><APP_CLONES></APP_CLONES><TEMPLATE><DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX><DRIVER><![CDATA[raw]]></DRIVER><FSTYPE><![CDATA[raw]]></FSTYPE></TEMPLATE><SNAPSHOTS><ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS></SNAPSHOTS></IMAGE>',28,121,1,0,0), error 1062 : Duplicate entry 'PPGTEMP01-disk-0-28' for key 'name'

Has anyone seen something before and has a fix for it?

Especially the duplicate key issue seems odd: Duplicate entry ‘PPGTEMP01-disk-0-28’ for key ‘name’

I have already run a fsck but this did not fix the problem.

I am running 5.4.9 with RAFT.

Did you do onedb upgrade?

UPD: The same error

I could finally solve it by doing the steps in the attached topic. But really strange that it happened. I could not reproduce the problem but somehow there was a stuck sql statement in the log table of one of the nodes. This lead to a complete malfunctioning cluster.