Frontend HA, leader changed due to timeout

Hello, today I find out that RAFT leader was changed yesterday on our cluster. When I am looking into logs, I see a few warnings about slow DB queries. Can this cause heartbeat timeouts?

Original leader logs:

Wed Feb 19 08:12:16 2020 [Z0][SQL][W]: Slow query (2.19s) detected: SELECT c.log_index, c.term, c.sqlcmd, c.timestamp, c.fed_index, p.log_index, p.term FROM logdb c, logdb p WHERE c.log_index = 51273615 AND p.log_index = 51273614
Wed Feb 19 08:12:21 2020 [Z0][MKP][I]: Recovering MarketPlace drivers
Wed Feb 19 08:12:21 2020 [Z0][RCM][I]: Follower 1 term (73) is higher than current (72)
Wed Feb 19 08:12:21 2020 [Z0][RCM][I]: Follower 2 term (73) is higher than current (72)
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: New term (73) discovered from leader 2
Wed Feb 19 08:12:21 2020 [Z0][RCM][I]: Follower 1 term (73) is higher than current (72)
Wed Feb 19 08:12:21 2020 [Z0][RCM][I]: Follower 2 term (73) is higher than current (72)
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: New term (73) discovered from leader 2
Wed Feb 19 08:12:21 2020 [Z0][ReM][D]: Req:288 UID:0 IP:192.168.44.23 one.zone.voterequest invoked , 73, 2, 51273617, 72
Wed Feb 19 08:12:21 2020 [Z0][ReM][D]: Req:2928 UID:0 IP:192.168.44.22 one.zone.voterequest invoked , 73, 1, 51273614, 72
Wed Feb 19 08:12:21 2020 [Z0][SQL][W]: Slow query (8.86s) detected: SELECT body FROM history WHERE vid = 123 AND seq = 286
Wed Feb 19 08:12:21 2020 [Z0][ReM][D]: Req:6608 UID:2 IP:127.0.0.1 one.vm.unlock invoked , 123
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: New term (73) discovered from leader 2
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: New term (73) discovered from leader 2
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: New term (73) discovered from leader 2
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: New term (73) discovered from leader 2
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: New term (73) discovered from leader 2
Wed Feb 19 08:12:21 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:21 2020 [Z0][DBM][E]: Not applying log record, oned is now a follower
Wed Feb 19 08:12:21 2020 [Z0][RCM][I]: Replication thread stopped
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: Replication thread stopped
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: Replication thread stopped
Wed Feb 19 08:12:22 2020 [Z0][VMM][D]: VM 509 successfully monitored: STATE=a CPU=0.0 MEMORY=30896 NETRX=6553049634 NETTX=1580615510 DISKRDBYTES=954772864 DISKWRBYTES=11630503936 DISKRDIOPS=29041 DISKWRIOPS=1028492
Wed Feb 19 08:12:22 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:22 2020 [Z0][SQL][W]: Slow query (0.78s) detected: DELETE FROM host_monitoring WHERE last_mon_time < 1582053141
Wed Feb 19 08:12:22 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: Replication thread stopped
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][ReM][I]: New term (73) discovered from candidate 2
Wed Feb 19 08:12:22 2020 [Z0][VMM][D]: VM 712 successfully monitored: STATE=a CPU=3.99 MEMORY=30936 NETRX=5318756313 NETTX=461641468 DISKRDBYTES=64371343684 DISKWRBYTES=39616807424 DISKRDIOPS=823281 DISKWRIOPS=1776222
Wed Feb 19 08:12:22 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][ReM][E]: Req:288 UID:0 one.zone.voterequest result FAILURE [one.zone.voterequest] Candidate's log is outdated
Wed Feb 19 08:12:22 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:22 2020 [Z0][VMM][D]: VM 713 successfully monitored: STATE=a CPU=3.0 MEMORY=30784 NETRX=5307735423 NETTX=497844023 DISKRDBYTES=49518509380 DISKWRBYTES=40688912896 DISKRDIOPS=666554 DISKWRIOPS=1780054
Wed Feb 19 08:12:22 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][DBM][E]: Not applying log record, oned is now a follower
Wed Feb 19 08:12:22 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:22 2020 [Z0][SQL][W]: Slow query (1.37s) detected: DELETE FROM vm_monitoring WHERE last_poll < 1582081941
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:22 2020 [Z0][VMM][D]: VM 737 successfully monitored: STATE=a CPU=100.83 MEMORY=32988 NETRX=6573981782 NETTX=1000773654 DISKRDBYTES=38982903616 DISKWRBYTES=69116060160 DISKRDIOPS=667505 DISKWRIOPS=2278992
Wed Feb 19 08:12:22 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:23 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:23 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:23 2020 [Z0][ReM][I]: New term (73) discovered from candidate 1
Wed Feb 19 08:12:23 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:23 2020 [Z0][ReM][E]: Req:2928 UID:0 one.zone.voterequest result FAILURE [one.zone.voterequest] Candidate's log is outdated
Wed Feb 19 08:12:24 2020 [Z0][VMM][D]: VM 123 successfully monitored: STATE=a CPU=7.99 MEMORY=26640 NETRX=22731067008 NETTX=1450553249 DISKRDBYTES=4715827200 DISKWRBYTES=1638705807360 DISKRDIOPS=362816 DISKWRIOPS=41302151
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][VMM][D]: VM 835 successfully monitored: STATE=a CPU=2.01 MEMORY=26932 NETRX=4705884288 NETTX=1025201123 DISKRDBYTES=19455170136 DISKWRBYTES=26109777408 DISKRDIOPS=295021 DISKWRIOPS=845177
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][VMM][D]: VM 836 successfully monitored: STATE=a CPU=3.02 MEMORY=26960 NETRX=3523963395 NETTX=357946531 DISKRDBYTES=56966093636 DISKWRBYTES=29080155136 DISKRDIOPS=730784 DISKWRIOPS=801358
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][VMM][D]: VM 843 successfully monitored: STATE=a CPU=2.01 MEMORY=26792 NETRX=3088143901 NETTX=144505247 DISKRDBYTES=4233165144 DISKWRBYTES=9834427392 DISKRDIOPS=68389 DISKWRIOPS=273784
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][VMM][D]: VM 124 successfully monitored: STATE=a CPU=0.0 MEMORY=27236 NETRX=13980820578 NETTX=521127560 DISKRDBYTES=20587627580 DISKWRBYTES=63831535616 DISKRDIOPS=471325 DISKWRIOPS=6441827
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:24 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:25 2020 [Z0][VMM][D]: VM 847 successfully monitored: STATE=a CPU=5.03 MEMORY=26372 NETRX=3168083380 NETTX=102614698 DISKRDBYTES=44023101764 DISKWRBYTES=24843573248 DISKRDIOPS=602310 DISKWRIOPS=383319
Wed Feb 19 08:12:25 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:25 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:25 2020 [Z0][VMM][D]: VM 127 successfully monitored: STATE=a CPU=12.98 MEMORY=35080 NETRX=89730720469 NETTX=193963400219 DISKRDBYTES=1181845824732 DISKWRBYTES=1792894859776 DISKRDIOPS=63558081 DISKWRIOPS=103833770
Wed Feb 19 08:12:25 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:25 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:25 2020 [Z0][VMM][D]: VM 848 successfully monitored: STATE=a CPU=6.04 MEMORY=26220 NETRX=2813854656 NETTX=147612577 DISKRDBYTES=47954683716 DISKWRBYTES=23609918976 DISKRDIOPS=653036 DISKWRIOPS=395328
Wed Feb 19 08:12:25 2020 [Z0][DBM][E]: Tried to modify DB being a follower
Wed Feb 19 08:12:25 2020 [Z0][HKM][D]: Message received: EXECUTE SUCCESS -1 RAFT_FOLLOWER_HOOK: 

Wed Feb 19 08:12:25 2020 [Z0][DBM][E]: Tried to modify DB being a follower

Original follower logs:

Wed Feb 19 08:11:44 2020 [Z0][SQL][W]: Slow query (0.58s) detected: DELETE FROM logdb WHERE applied = 1 AND log_index >= 0 AND fed_index = -1 AND log_index < 51023396 LIMIT 100000
Wed Feb 19 08:11:44 2020 [Z0][DBM][I]: Purging obsolete LogDB records: 717 records purged. Log state: 0,51273395 - 0,51273395
Wed Feb 19 08:11:44 2020 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 1
Wed Feb 19 08:11:50 2020 [Z0][ReM][D]: Req:8736 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked 
Wed Feb 19 08:11:50 2020 [Z0][ReM][D]: Req:8736 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>1</..."
Wed Feb 19 08:12:17 2020 [Z0][RRM][E]: Failed to get heartbeat from leader. Starting election proccess
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: Error requesting vote from follower 0:RPC call timed out and aborted
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: Vote not granted from follower 2: [one.zone.voterequest] Candidate's log is outdated
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: No leader found, starting new election in 565ms
Wed Feb 19 08:12:18 2020 [Z0][ReM][D]: Req:8160 UID:0 IP:192.168.44.23 one.zone.voterequest invoked , 73, 2, 51273617, 72
Wed Feb 19 08:12:18 2020 [Z0][ReM][D]: Req:8160 UID:0 one.zone.voterequest result SUCCESS, 73
Wed Feb 19 08:12:19 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:19 2020 [Z0][RCM][I]: oned is set to follower mode
Wed Feb 19 08:12:20 2020 [Z0][ReM][D]: Req:4656 UID:0 IP:127.0.0.1 one.zone.raftstatus invoked 
Wed Feb 19 08:12:20 2020 [Z0][ReM][D]: Req:4656 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>1</..."
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: Leader term (72) is outdated (73)
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: Leader term (72) is outdated (73)
Wed Feb 19 08:12:28 2020 [Z0][VMM][I]: --Mark--
Wed Feb 19 08:12:46 2020 [Z0][SQL][W]: Slow query (0.59s) detected: DELETE FROM logdb WHERE applied = 1 AND log_index >= 0 AND fed_index = -1 AND log_index < 51023618 LIMIT 100000
Wed Feb 19 08:12:46 2020 [Z0][DBM][I]: Purging obsolete LogDB records: 222 records purged. Log state: 0,51273617 - 0,51273617
Wed Feb 19 08:12:46 2020 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 1

Original follower which becomes new Leader

Wed Feb 19 08:12:08 2020 [Z0][ReM][D]: Req:1776 UID:0 one.zone.raftstatus result SUCCESS, "<RAFT><SERVER_ID>2</..."
Wed Feb 19 08:12:17 2020 [Z0][RRM][E]: Failed to get heartbeat from leader. Starting election proccess
Wed Feb 19 08:12:18 2020 [Z0][ReM][D]: Req:6880 UID:0 IP:192.168.44.22 one.zone.voterequest invoked , 73, 1, 51273614, 72
Wed Feb 19 08:12:18 2020 [Z0][ReM][E]: Req:6880 UID:0 one.zone.voterequest result FAILURE [one.zone.voterequest] Candidate's log is outdated
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: Error requesting vote from follower 0:RPC call timed out and aborted
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: Got vote from follower 1. Total votes: 1
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: Got majority of votes
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: Becoming leader of the zone. Last log record: 51273617 last applied record: 51273616
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: oned is now the leader of the zone
Wed Feb 19 08:12:18 2020 [Z0][RCM][I]: Replicating log to followers
Wed Feb 19 08:12:19 2020 [Z0][RCM][I]: Leader log replicated
Wed Feb 19 08:12:19 2020 [Z0][RCM][D]: Faild to replicate log record at index: 51273617 on follower: 0, error: Error replicating log entry 51273617 on follower 0: RPC call timed out and aborted
Wed Feb 19 08:12:21 2020 [Z0][RCM][D]: Faild to replicate log record at index: 51273617 on follower: 0, error: Error replicating log entry 51273617 on follower 0: RPC call timed out and aborted
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: Leader term (72) is outdated (73)
Wed Feb 19 08:12:21 2020 [Z0][ReM][I]: Leader term (72) is outdated (73)
Wed Feb 19 08:12:22 2020 [Z0][RCM][D]: Faild to replicate log record at index: 51273617 on follower: 0, error: Error replicating log entry 51273617 on follower 0: RPC call timed out and aborted
Wed Feb 19 08:12:25 2020 [Z0][DBM][I]: Purging obsolete LogDB records: 536 records purged. Log state: 10583085,51273617 - 10583085,51273617
Wed Feb 19 08:12:25 2020 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 1
Wed Feb 19 08:12:26 2020 [Z0][ReM][D]: Req:7440 UID:1 IP:127.0.0.1 one.documentpool.info invoked , -2, -1, -1, 100
Wed Feb 19 08:12:26 2020 [Z0][ReM][D]: Req:7440 UID:1 one.documentpool.info result SUCCESS, "<DOCUMENT_POOL></DOC..."
Wed Feb 19 08:12:29 2020 [Z0][HKM][D]: Message received: EXECUTE SUCCESS -1 RAFT_LEADER_HOOK: ARPING 185.174.169.10 from 185.174.169.10 eth0