Hi,
If I got it right, all DSs are monitored as part of script /var/lib/one/remotes/im/run_probes kvm, which calls /var/lib/one/remotes/im/kvm-probes.d/monitor_ds.sh on each host.
In case when DS is NFS shared, would this result in redundant information? For example, with 10 KVM hosts, collectd one ONE controller gets exactly the same info x10 (each host reports the same info for NFS shared DS).
Thank you.
Yes, the information is collected in all hosts, this:
it’s redundant but makes the system keep working in case a host fails
There is no much overhead as this represent just some bytes in the
message (note that there is no extra message)
The schema is the same for SSH and Shared system DS.
As there is no practical overhead we think that 3 (same logic) and 1 (HA on
host failures) worth the extra information
Cheers
Got it, thank you.
I’m troubleshooting [1] and thought the way DS is monitored could contribute to deadlocks.
Would it make sense to move DS monitoring of off hosts to ONE controller (ONED) ? ONE controller already has all DSs mounted and can collect/report in one place. In my case I have 40 hosts and 17 datastores, and since most of deadlocks reported on DS, I thought this could be a reason…
Thank you.
[1] SQL error 1213 : Deadlock found when trying to get lock; try restarting transaction
Hi Dmitri
We can easily check that the origin of the deadlocks id the probe. Go to
the probe (monitor_ds.sh) and add a check to excute it only in one host
if [ hostname = “xxxx” ] ; then
fi
Also we’ve changed a du command for a df command, please check also that
this has been updated in your setup.
Cheers
Good idea.
For a test I disabled monitor_ds.sh at all (put exit 0 as 1st line, w/o checking for hostname).
I still get error 1213 : Deadlock found when trying to update DS info SQL command was: REPLACE INTO datastore_pool in oned.log file.
Not sure what is happening here 
Tue Mar 17 14:03:31 2015 [Z0][ONE][E]: SQL command was: REPLACE INTO datastore_pool (oid, name, body, uid, gid, owner_u, group_u, other_u, cid) VALUES (144,'RHEL System vol2','<DATASTORE><ID>144</ID><UID>4</UID><GID>0</GID><UNAME>dchebota</UNAME><GNAME>oneadmin</GNAME><NAME>RHEL System vol2</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>1</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><DS_MAD><![CDATA[-]]></DS_MAD><TM_MAD><![CDATA[shared]]></TM_MAD><BASE_PATH><![CDATA[/var/lib/one//datastores/144]]></BASE_PATH><TYPE>1</TYPE><DISK_TYPE>0</DISK_TYPE><CLUSTER_ID>100</CLUSTER_ID><CLUSTER>RHEL</CLUSTER><TOTAL_MB>1048576</TOTAL_MB><FREE_MB>909650</FREE_MB><USED_MB>353651</USED_MB><IMAGES></IMAGES><TEMPLATE><SHARED><![CDATA[YES]]></SHARED><TM_MAD><![CDATA[shared]]></TM_MAD><TYPE><![CDATA[SYSTEM_DS]]></TYPE></TEMPLATE></DATASTORE>',4,0,1,1,0,100), error 1213 : Deadlock found when trying to get lock; try restarting transactionMaybe we can get info from the innodb engine, could you attach the output
of:
SHOW ENGINE INNODB STATUS
SHOW ENGINE INNODB MUTEX
Cheers
Hi, sorry for the delay. Here is output from both commands:
| InnoDB | |
=====================================
2015-03-23 15:51:14 7f22f3300700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 25 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1184998 srv_active, 0 srv_shutdown, 717369 srv_idle
srv_master_thread log flush and writes: 1902367
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 10586626
OS WAIT ARRAY INFO: signal count 9029866
Mutex spin waits 1309033708, rounds 1866925467, OS waits 9199505
RW-shared spins 1263805, rounds 30936071, OS waits 936063
RW-excl spins 353258, rounds 8540500, OS waits 149132
Spin rounds per wait: 1.43 mutex, 24.48 RW-shared, 24.18 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
2015-03-23 15:36:30 7f232944a700
*** (1) TRANSACTION:
TRANSACTION 269909658, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 5 lock struct(s), heap size 1184, 4 row lock(s), undo log entries 2
MySQL thread id 3602609, OS thread handle 0x7f22f3eba700, query id 70694744 one-1 192.168.1.5 oneadmin update
REPLACE INTO datastore_pool (oid, name, body, uid, gid, owner_u, group_u, other_u, cid) VALUES (144,'RHEL_SYS_0','<DATASTORE><ID>144</ID><UID>4</UID><GID>0</GID><UNAME>dchebota</UNAME><GNAME>oneadmin</GNAME><NAME>RHEL_SYS_0</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>1</OWNER_A><GROUP_U>1</GROUP_U><GROUP_M>1</GROUP_M><GROUP_A>1</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><DS_MAD><![CDATA[-]]></DS_MAD><TM_MAD><![CDATA[shared]]></TM_MAD><BASE_PATH><![CDATA[/var/lib/one//datastores/144]]></BASE_PATH><TYPE>1</TYPE><DISK_TYPE>0</DISK_TYPE><CLUSTER_ID>100</CLUSTER_ID><CLUSTER>RHEL</CLUSTER><TOTAL_MB>1048576</TOTAL_MB><FREE_MB>1014967</FREE_MB><USED_MB>33610</USED_MB><IMAGES></IMAGES><TEMPLATE><INFO><![CDATA[Running images for VMs in RHEL]]></INFO><SHARED><![CDATA[YES]]></SHARED><TM_MAD><![CDATA[shared]]></TM_MAD><TYPE><![CDATA[SYSTEM_DS]]></TYPE><
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 31 page no 4 n bits 88 index `name` of table `opennebula`.`datastore_pool` trx id 269909658 lock_mode X waiting
Record lock, heap no 2
*** (2) TRANSACTION:
TRANSACTION 269909662, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1184, 3 row lock(s), undo log entries 2
MySQL thread id 3602616, OS thread handle 0x7f232944a700, query id 70694748 one-1 192.168.1.5 oneadmin update
REPLACE INTO datastore_pool (oid, name, body, uid, gid, owner_u, group_u, other_u, cid) VALUES (0,'system','<DATASTORE><ID>0</ID><UID>0</UID><GID>0</GID><UNAME>oneadmin</UNAME><GNAME>oneadmin</GNAME><NAME>system</NAME><PERMISSIONS><OWNER_U>1</OWNER_U><OWNER_M>1</OWNER_M><OWNER_A>0</OWNER_A><GROUP_U>1</GROUP_U><GROUP_M>0</GROUP_M><GROUP_A>0</GROUP_A><OTHER_U>0</OTHER_U><OTHER_M>0</OTHER_M><OTHER_A>0</OTHER_A></PERMISSIONS><DS_MAD><![CDATA[-]]></DS_MAD><TM_MAD><![CDATA[shared]]></TM_MAD><BASE_PATH><![CDATA[/var/lib/one/datastores/0]]></BASE_PATH><TYPE>1</TYPE><DISK_TYPE>0</DISK_TYPE><CLUSTER_ID>112</CLUSTER_ID><CLUSTER>R900-LAB</CLUSTER><TOTAL_MB>552960</TOTAL_MB><FREE_MB>498533</FREE_MB><USED_MB>54428</USED_MB><IMAGES></IMAGES><TEMPLATE><SHARED><![CDATA[YES]]></SHARED><TM_MAD><![CDATA[shared]]></TM_MAD><TYPE><![CDATA[SYSTEM_DS]]></TYPE></TEMPLATE></DATASTORE>',0,0,1,1,0,112)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 31 page no 4 n bits 88 index `name` of table `opennebula`.`datastore_pool` trx id 269909662 lock_mode X locks rec but not gap
Record lock, heap no 2
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 31 page no 4 n bits 88 index `name` of table `opennebula`.`datastore_pool` trx id 269909662 lock_mode X waiting
Record lock, heap no 2
*** WE ROLL BACK TRANSACTION (2)
------------
TRANSACTIONS
------------
Trx id counter 269925042
Purge done for trx's n:o < 269925040 undo n:o < 0 state: running but idle
History list length 433
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 4945572, OS thread handle 0x7f22f3300700, query id 70716091 localhost root init
SHOW ENGINE INNODB STATUS
---TRANSACTION 269925035, not started
MySQL thread id 3602618, OS thread handle 0x7f2329409700, query id 70716073 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925020, not started
MySQL thread id 3602617, OS thread handle 0x7f22f32bf700, query id 70716053 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925025, not started
MySQL thread id 3602616, OS thread handle 0x7f232944a700, query id 70716058 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925028, not started
MySQL thread id 3602615, OS thread handle 0x7f22f38d3700, query id 70716069 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925032, not started
MySQL thread id 3602614, OS thread handle 0x7f22f3f7d700, query id 70716071 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925037, not started
MySQL thread id 3602613, OS thread handle 0x7f23293c8700, query id 70716074 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925033, not started
MySQL thread id 3602612, OS thread handle 0x7f22f3649700, query id 70716072 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925022, not started
MySQL thread id 3602611, OS thread handle 0x7f22f3f3c700, query id 70716054 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925030, not started
MySQL thread id 3602610, OS thread handle 0x7f22f3914700, query id 70716070 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 269925023, not started
MySQL thread id 3602609, OS thread handle 0x7f22f3eba700, query id 70716055 one-1 192.168.1.5 oneadmin cleaning up
---TRANSACTION 204812720, not started
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
99753 OS file reads, 152662026 OS file writes, 54821846 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 49.80 writes/s, 22.72 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 296, seg size 298, 2145 merges
merged operations:
insert 2713, delete mark 87806, delete 11265
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 9461399, node heap has 996 buffer(s)
34.00 hash searches/s, 30.00 non-hash searches/s
---
LOG
---
Log sequence number 646710017727
Log flushed up to 646710017727
Pages flushed up to 646710017727
Last checkpoint at 646710017727
0 pending log writes, 0 pending chkp writes
25705185 log i/o's done, 8.44 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 4395630592; in additional pool allocated 0
Dictionary memory allocated 1697547
Buffer pool size 262136
Free buffers 29828
Database pages 231312
Old database pages 85237
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 13631, not young 36384
0.00 youngs/s, 0.00 non-youngs/s
Pages read 99256, created 132063, written 117771701
0.00 reads/s, 0.00 creates/s, 36.04 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 231312, unzip_LRU len: 0
I/O sum[12520]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 32767
Free buffers 2755
Database pages 29904
Old database pages 11033
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 197, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 13509, created 16395, written 28415316
0.00 reads/s, 0.00 creates/s, 11.64 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 29904, unzip_LRU len: 0
I/O sum[1565]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 32767
Free buffers 3644
Database pages 28991
Old database pages 10681
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 12029, not young 36384
0.00 youngs/s, 0.00 non-youngs/s
Pages read 12653, created 16345, written 10741828
0.00 reads/s, 0.00 creates/s, 3.48 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 28991, unzip_LRU len: 0
I/O sum[1565]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 32767
Free buffers 3649
Database pages 28989
Old database pages 10681
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 303, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 12174, created 16815, written 15956799
0.00 reads/s, 0.00 creates/s, 3.84 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 28989, unzip_LRU len: 0
I/O sum[1565]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 32767
Free buffers 4168
Database pages 28467
Old database pages 10488
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 249, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 11696, created 16771, written 22680048
0.00 reads/s, 0.00 creates/s, 7.36 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 28467, unzip_LRU len: 0
I/O sum[1565]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 32767
Free buffers 4719
Database pages 27922
Old database pages 10287
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 315, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 11582, created 16340, written 14432151
0.00 reads/s, 0.00 creates/s, 3.56 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 27922, unzip_LRU len: 0
I/O sum[1565]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 32767
Free buffers 3352
Database pages 29295
Old database pages 10793
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 103, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 12981, created 16314, written 6347726
0.00 reads/s, 0.00 creates/s, 1.56 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 29295, unzip_LRU len: 0
I/O sum[1565]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 32767
Free buffers 4096
Database pages 28547
Old database pages 10517
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 74, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 12180, created 16367, written 8485798
0.00 reads/s, 0.00 creates/s, 1.24 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 28547, unzip_LRU len: 0
I/O sum[1565]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 32767
Free buffers 3445
Database pages 29197
Old database pages 10757
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 361, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 12481, created 16716, written 10712035
0.00 reads/s, 0.00 creates/s, 3.36 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 29197, unzip_LRU len: 0
I/O sum[1565]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Main thread process no. 18636, id 139789036242688, state: sleeping
Number of rows inserted 23435191, updated 13801725, deleted 22386626, read 19359396988
4.92 inserts/s, 4.04 updates/s, 3.72 deletes/s, 8373.51 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
mysql> SHOW ENGINE INNODB MUTEX\g
+--------+--------------------------+------------------+
| Type | Name | Status |
+--------+--------------------------+------------------+
| InnoDB | trx0trx.cc:108 | os_waits=1 |
| InnoDB | trx0purge.cc:132 | os_waits=205 |
| InnoDB | trx0rseg.cc:196 | os_waits=266 |
| InnoDB | trx0rseg.cc:196 | os_waits=314 |
| InnoDB | trx0rseg.cc:196 | os_waits=264 |
| InnoDB | trx0rseg.cc:196 | os_waits=280 |
| InnoDB | trx0rseg.cc:196 | os_waits=302 |
| InnoDB | trx0rseg.cc:196 | os_waits=272 |
| InnoDB | trx0rseg.cc:196 | os_waits=259 |
| InnoDB | trx0rseg.cc:196 | os_waits=283 |
| InnoDB | trx0rseg.cc:196 | os_waits=286 |
| InnoDB | trx0rseg.cc:196 | os_waits=296 |
| InnoDB | trx0rseg.cc:196 | os_waits=293 |
| InnoDB | trx0rseg.cc:196 | os_waits=286 |
| InnoDB | trx0rseg.cc:196 | os_waits=262 |
| InnoDB | trx0rseg.cc:196 | os_waits=305 |
| InnoDB | trx0rseg.cc:196 | os_waits=279 |
| InnoDB | trx0rseg.cc:196 | os_waits=266 |
| InnoDB | trx0rseg.cc:196 | os_waits=299 |
| InnoDB | trx0rseg.cc:196 | os_waits=284 |
| InnoDB | trx0rseg.cc:196 | os_waits=288 |
| InnoDB | trx0rseg.cc:196 | os_waits=299 |
| InnoDB | trx0rseg.cc:196 | os_waits=297 |
| InnoDB | trx0rseg.cc:196 | os_waits=290 |
| InnoDB | trx0rseg.cc:196 | os_waits=291 |
| InnoDB | trx0rseg.cc:196 | os_waits=296 |
| InnoDB | trx0rseg.cc:196 | os_waits=295 |
| InnoDB | trx0rseg.cc:196 | os_waits=281 |
| InnoDB | trx0rseg.cc:196 | os_waits=289 |
| InnoDB | trx0rseg.cc:196 | os_waits=308 |
| InnoDB | trx0rseg.cc:196 | os_waits=296 |
| InnoDB | trx0rseg.cc:196 | os_waits=293 |
| InnoDB | trx0rseg.cc:196 | os_waits=294 |
| InnoDB | trx0rseg.cc:196 | os_waits=305 |
| InnoDB | trx0rseg.cc:196 | os_waits=287 |
| InnoDB | trx0rseg.cc:196 | os_waits=248 |
| InnoDB | trx0rseg.cc:196 | os_waits=302 |
| InnoDB | trx0rseg.cc:196 | os_waits=292 |
| InnoDB | trx0rseg.cc:196 | os_waits=267 |
| InnoDB | trx0rseg.cc:196 | os_waits=295 |
| InnoDB | trx0rseg.cc:196 | os_waits=304 |
| InnoDB | trx0rseg.cc:196 | os_waits=307 |
| InnoDB | trx0rseg.cc:196 | os_waits=279 |
| InnoDB | trx0rseg.cc:196 | os_waits=292 |
| InnoDB | trx0rseg.cc:196 | os_waits=286 |
| InnoDB | trx0rseg.cc:196 | os_waits=295 |
| InnoDB | trx0rseg.cc:196 | os_waits=293 |
| InnoDB | trx0rseg.cc:196 | os_waits=322 |
| InnoDB | trx0rseg.cc:196 | os_waits=290 |
| InnoDB | trx0rseg.cc:196 | os_waits=254 |
| InnoDB | trx0rseg.cc:196 | os_waits=288 |
| InnoDB | trx0rseg.cc:196 | os_waits=308 |
| InnoDB | trx0rseg.cc:196 | os_waits=295 |
| InnoDB | trx0rseg.cc:196 | os_waits=276 |
| InnoDB | trx0rseg.cc:196 | os_waits=289 |
| InnoDB | trx0rseg.cc:196 | os_waits=290 |
| InnoDB | trx0rseg.cc:196 | os_waits=276 |
| InnoDB | trx0rseg.cc:196 | os_waits=275 |
| InnoDB | trx0rseg.cc:196 | os_waits=276 |
| InnoDB | trx0rseg.cc:196 | os_waits=277 |
| InnoDB | trx0rseg.cc:196 | os_waits=296 |
| InnoDB | trx0rseg.cc:196 | os_waits=298 |
| InnoDB | trx0rseg.cc:196 | os_waits=274 |
| InnoDB | trx0rseg.cc:196 | os_waits=302 |
| InnoDB | trx0rseg.cc:196 | os_waits=291 |
| InnoDB | trx0rseg.cc:196 | os_waits=318 |
| InnoDB | trx0rseg.cc:196 | os_waits=265 |
| InnoDB | trx0rseg.cc:196 | os_waits=291 |
| InnoDB | trx0rseg.cc:196 | os_waits=312 |
| InnoDB | trx0rseg.cc:196 | os_waits=323 |
| InnoDB | trx0rseg.cc:196 | os_waits=288 |
| InnoDB | trx0rseg.cc:196 | os_waits=280 |
| InnoDB | trx0rseg.cc:196 | os_waits=306 |
| InnoDB | trx0rseg.cc:196 | os_waits=306 |
| InnoDB | trx0rseg.cc:196 | os_waits=299 |
| InnoDB | trx0rseg.cc:196 | os_waits=288 |
| InnoDB | trx0rseg.cc:196 | os_waits=295 |
| InnoDB | trx0rseg.cc:196 | os_waits=285 |
| InnoDB | trx0rseg.cc:196 | os_waits=271 |
| InnoDB | trx0rseg.cc:196 | os_waits=281 |
| InnoDB | trx0rseg.cc:196 | os_waits=278 |
| InnoDB | trx0rseg.cc:196 | os_waits=299 |
| InnoDB | trx0rseg.cc:196 | os_waits=316 |
| InnoDB | trx0rseg.cc:196 | os_waits=287 |
| InnoDB | trx0rseg.cc:196 | os_waits=278 |
| InnoDB | trx0rseg.cc:196 | os_waits=299 |
| InnoDB | trx0rseg.cc:196 | os_waits=288 |
| InnoDB | trx0rseg.cc:196 | os_waits=280 |
| InnoDB | trx0rseg.cc:196 | os_waits=278 |
| InnoDB | trx0rseg.cc:196 | os_waits=271 |
| InnoDB | trx0rseg.cc:196 | os_waits=271 |
| InnoDB | trx0rseg.cc:196 | os_waits=271 |
| InnoDB | trx0rseg.cc:196 | os_waits=282 |
| InnoDB | trx0rseg.cc:196 | os_waits=287 |
| InnoDB | trx0rseg.cc:196 | os_waits=276 |
| InnoDB | trx0rseg.cc:196 | os_waits=269 |
| InnoDB | trx0rseg.cc:196 | os_waits=298 |
| InnoDB | trx0rseg.cc:196 | os_waits=283 |
| InnoDB | trx0rseg.cc:196 | os_waits=290 |
| InnoDB | trx0rseg.cc:196 | os_waits=291 |
| InnoDB | trx0rseg.cc:196 | os_waits=258 |
| InnoDB | trx0rseg.cc:196 | os_waits=292 |
| InnoDB | trx0rseg.cc:196 | os_waits=292 |
| InnoDB | trx0rseg.cc:196 | os_waits=302 |
| InnoDB | trx0rseg.cc:196 | os_waits=288 |
| InnoDB | trx0rseg.cc:196 | os_waits=286 |
| InnoDB | trx0rseg.cc:196 | os_waits=239 |
| InnoDB | trx0rseg.cc:196 | os_waits=321 |
| InnoDB | trx0rseg.cc:196 | os_waits=289 |
| InnoDB | trx0rseg.cc:196 | os_waits=316 |
| InnoDB | trx0rseg.cc:196 | os_waits=323 |
| InnoDB | trx0rseg.cc:196 | os_waits=277 |
| InnoDB | trx0rseg.cc:196 | os_waits=281 |
| InnoDB | trx0rseg.cc:196 | os_waits=297 |
| InnoDB | trx0rseg.cc:196 | os_waits=266 |
| InnoDB | trx0rseg.cc:196 | os_waits=300 |
| InnoDB | trx0rseg.cc:196 | os_waits=295 |
| InnoDB | trx0rseg.cc:196 | os_waits=284 |
| InnoDB | trx0rseg.cc:196 | os_waits=279 |
| InnoDB | trx0rseg.cc:196 | os_waits=288 |
| InnoDB | trx0rseg.cc:196 | os_waits=275 |
| InnoDB | trx0rseg.cc:196 | os_waits=282 |
| InnoDB | trx0rseg.cc:196 | os_waits=308 |
| InnoDB | trx0rseg.cc:196 | os_waits=283 |
| InnoDB | trx0rseg.cc:196 | os_waits=291 |
| InnoDB | trx0rseg.cc:196 | os_waits=303 |
| InnoDB | trx0rseg.cc:196 | os_waits=262 |
| InnoDB | trx0rseg.cc:196 | os_waits=277 |
| InnoDB | trx0rseg.cc:196 | os_waits=298 |
| InnoDB | trx0rseg.cc:196 | os_waits=271 |
| InnoDB | ibuf0ibuf.cc:533 | os_waits=114 |
| InnoDB | dict0dict.cc:990 | os_waits=7 |
| InnoDB | dict0dict.cc:974 | os_waits=115595 |
| InnoDB | trx0sys.cc:594 | os_waits=5587 |
| InnoDB | dict0stats_bg.cc:228 | os_waits=5 |
| InnoDB | buf0dblwr.cc:146 | os_waits=111 |
| InnoDB | lock0lock.cc:612 | os_waits=179 |
| InnoDB | lock0lock.cc:609 | os_waits=2898634 |
| InnoDB | log0log.cc:848 | os_waits=816 |
| InnoDB | log0log.cc:844 | os_waits=5993173 |
| InnoDB | buf0buf.cc:1318 | os_waits=1 |
| InnoDB | buf0buf.cc:1270 | os_waits=12940 |
| InnoDB | buf0buf.cc:1318 | os_waits=4 |
| InnoDB | buf0buf.cc:1270 | os_waits=12383 |
| InnoDB | buf0buf.cc:1318 | os_waits=1 |
| InnoDB | buf0buf.cc:1270 | os_waits=10769 |
| InnoDB | buf0buf.cc:1318 | os_waits=2 |
| InnoDB | buf0buf.cc:1270 | os_waits=11619 |
| InnoDB | buf0buf.cc:1318 | os_waits=7 |
| InnoDB | buf0buf.cc:1270 | os_waits=12943 |
| InnoDB | buf0buf.cc:1318 | os_waits=6 |
| InnoDB | buf0buf.cc:1270 | os_waits=12624 |
| InnoDB | buf0buf.cc:1318 | os_waits=4 |
| InnoDB | buf0buf.cc:1270 | os_waits=10217 |
| InnoDB | buf0buf.cc:1318 | os_waits=10 |
| InnoDB | buf0buf.cc:1270 | os_waits=13785 |
| InnoDB | fil0fil.cc:1690 | os_waits=2455 |
| InnoDB | srv0start.cc:1874 | os_waits=4 |
| InnoDB | srv0srv.cc:936 | os_waits=44471 |
| InnoDB | combined buf0buf.cc:1020 | os_waits=4018 |
| InnoDB | dict0dict.cc:2551 | os_waits=63 |
| InnoDB | dict0dict.cc:2551 | os_waits=63 |
| InnoDB | dict0dict.cc:2551 | os_waits=79 |
| InnoDB | dict0dict.cc:2551 | os_waits=221 |
| InnoDB | dict0dict.cc:2551 | os_waits=3044 |
| InnoDB | dict0dict.cc:2551 | os_waits=17 |
| InnoDB | dict0dict.cc:2551 | os_waits=1 |
| InnoDB | dict0dict.cc:2551 | os_waits=1 |
| InnoDB | dict0dict.cc:2551 | os_waits=2 |
| InnoDB | dict0dict.cc:2551 | os_waits=383 |
| InnoDB | dict0dict.cc:2551 | os_waits=116 |
| InnoDB | dict0dict.cc:2551 | os_waits=230 |
| InnoDB | dict0dict.cc:2551 | os_waits=29798 |
| InnoDB | fil0fil.cc:1321 | os_waits=1134 |
| InnoDB | dict0dict.cc:2551 | os_waits=1942 |
| InnoDB | fil0fil.cc:1321 | os_waits=30 |
| InnoDB | dict0dict.cc:2551 | os_waits=1759 |
| InnoDB | trx0purge.cc:128 | os_waits=482 |
| InnoDB | dict0dict.cc:2551 | os_waits=199 |
| InnoDB | dict0dict.cc:983 | os_waits=319011 |
| InnoDB | fil0fil.cc:1321 | os_waits=7055 |
| InnoDB | log0log.cc:907 | os_waits=529254 |
| InnoDB | btr0sea.cc:173 | os_waits=2458 |
| InnoDB | hash0hash.cc:392 | os_waits=1 |
| InnoDB | hash0hash.cc:392 | os_waits=2 |
| InnoDB | hash0hash.cc:392 | os_waits=1 |
| InnoDB | hash0hash.cc:392 | os_waits=1 |
| InnoDB | hash0hash.cc:392 | os_waits=1 |
| InnoDB | hash0hash.cc:392 | os_waits=1 |
| InnoDB | hash0hash.cc:392 | os_waits=1 |
| InnoDB | hash0hash.cc:392 | os_waits=2 |
| InnoDB | combined buf0buf.cc:1021 | os_waits=187761 |
+--------+--------------------------+------------------+
192 rows in set (0.04 sec)