Hello all Community,
after upgrade from 6.2 to 6.4 my fronted nodes can’t select a leader. I don’t think it’s due new packages. More likely reason it that I rebooted all 3 nodes. I googled all day and have no idea what’s wrong and how repair it.
My nodes status
HA & FEDERATION SYNC STATUS
ID NAME STATE TERM INDEX COMMIT VOTE FED_INDEX
4 master-1 candidate 55964 9753534 9753534 -1 -1
5 master-2 leader 55964 9754097 9754097 4 -1
6 master-3 follower 55964 8819719 8819719 4 -1
|
v
HA & FEDERATION SYNC STATUS
ID NAME STATE TERM INDEX COMMIT VOTE FED_INDEX
4 master-1 candidate 55965 9753534 9753534 -1 -1
5 master-2 follower 55965 9754097 9754097 -1 -1
6 master-3 follower 55965 8819719 8819719 -1 -1
|
v
HA & FEDERATION SYNC STATUS
ID NAME STATE TERM INDEX COMMIT VOTE FED_INDEX
4 master-1 candidate 55972 9753534 9753534 -1 -1
5 master-2 leader 55972 9754099 9754099 4 -1
6 master-3 leader 55972 8819719 8819719 4 -1
oned.log
root@srv1:/usr/share/one# tail -n 10 /var/log/one/oned.log
Fri Jun 3 15:45:11 2022 [Z0][RCM][I]: Vote not granted from follower 6: [one.zone.voterequest] Candidate's log is outdated
Fri Jun 3 15:45:11 2022 [Z0][RCM][I]: No leader found, starting new election in 1715ms
Fri Jun 3 15:45:13 2022 [Z0][RCM][I]: Vote not granted from follower 5: [one.zone.voterequest] Candidate's log is outdated
Fri Jun 3 15:45:13 2022 [Z0][RCM][I]: Vote not granted from follower 6: [one.zone.voterequest] Candidate's log is outdated
Fri Jun 3 15:45:13 2022 [Z0][RCM][I]: No leader found, starting new election in 6362ms
Fri Jun 3 15:45:18 2022 [Z0][ReM][D]: Req:7856 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked
Fri Jun 3 15:45:18 2022 [Z0][ReM][D]: Req:7856 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>4</..."
Fri Jun 3 15:45:20 2022 [Z0][RCM][I]: Vote not granted from follower 5: [one.zone.voterequest] Candidate's log is outdated
Fri Jun 3 15:45:20 2022 [Z0][RCM][I]: Vote not granted from follower 6: [one.zone.voterequest] Candidate's log is outdated
Fri Jun 3 15:45:20 2022 [Z0][RCM][I]: No leader found, starting new election in 5338ms
===============
root@srv2:/etc/one# tail -n 10 /var/log/one/oned.log
Fri Jun 3 15:45:20 2022 [Z0][ReM][D]: Req:8912 UID:0 one.hookpool.info result SUCCESS, "<HOOK_POOL><HOOK><ID..."
Fri Jun 3 15:45:21 2022 [Z0][RRM][E]: Failed to get heartbeat from leader. Starting election proccess
Fri Jun 3 15:45:21 2022 [Z0][ReM][D]: Req:512 UID:0 IP:10.44.0.52 one.zone.voterequest invoked , 56015, 4, 9754113, 56000
Fri Jun 3 15:45:21 2022 [Z0][ReM][D]: Req:512 UID:0 one.zone.voterequest result SUCCESS, 56015
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: Got vote from follower 5. Total votes: 1
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: Got majority of votes
Fri Jun 3 15:45:21 2022 [Z0][ReM][D]: Req:1328 UID:0 IP:10.44.0.53 one.zone.voterequest invoked , 56015, 4, 8819720, 55972
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: Becoming leader of the zone. Last log record: 9754113 last applied record: 9754113
Fri Jun 3 15:45:21 2022 [Z0][ReM][E]: Req:1328 UID:0 one.zone.voterequest result FAILURE [one.zone.voterequest] Candidate's log is outdated
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: oned is now the leader of the zone
==============
root@srv3:/usr/share/one# tail -n 10 /var/log/one/oned.log
Fri Jun 3 15:45:20 2022 [Z0][RCM][I]: oned is set to follower mode
Fri Jun 3 15:45:20 2022 [Z0][ReM][E]: Req:6528 UID:0 one.zone.voterequest result FAILURE [one.zone.voterequest] Candidate's log is outdated
Fri Jun 3 15:45:20 2022 [Z0][RRM][E]: Failed to get heartbeat from leader. Starting election proccess
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: Vote not granted from follower 5: [one.zone.voterequest] Candidate's log is outdated
Fri Jun 3 15:45:21 2022 [Z0][ReM][D]: Req:1360 UID:0 IP:127.0.0.1 one.zone.voterequest invoked , 56015, 4, 8819720, 55972
Fri Jun 3 15:45:21 2022 [Z0][ReM][D]: Req:1360 UID:0 one.zone.voterequest result SUCCESS, 56015
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: Got vote from follower 6. Total votes: 1
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: Got majority of votes
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: Becoming leader of the zone. Last log record: 8819720 last applied record: 8819720
Fri Jun 3 15:45:21 2022 [Z0][RCM][I]: oned is now the leader of the zone